# «University of Nebraska - Lincoln DigitalCommons of Nebraska - Lincoln Dissertations and Theses in Statistics Statistics, Department of 8-2010 ...»

includes covariates, but not school effects, where y ig is student i’s score in year g, g = 1, 2, …, m, and g is the year-specific mean. Time-invariant and time-varying covariates for student i are included in the vector, x ig. The random teacher effects for each year, θ g, are linked to students by φ ig, which allows fractional contributions of teachers to student i’s score during year g. The persistency parameters, gg *, model the strength of past teacher effects on current student measurements, where gg * 1 when g* = g. The term eig represents random error. Similar to the EVAAS teacher model, the variable persistence model can be extended to account for multiple cohorts of students, subjects per grade and school systems, but the model quickly grows in complexity (Lockwood, McCaffrey, Mariano, et al., 2007).

Although the variable persistence model allows teacher effects to persist into the future, it does not assume these effects persist undiminished. Instead, the persistency parameters are freely estimated, so the total teacher contribution to the variability of scores does not inherently increase over time, unlike the cross-classified and EVAAS teacher models (McCaffrey et al., 2004). However, as the persistency parameters approach zero, correlations between future scores of students who shared a teacher in the past are no longer accounted for, eliminating one of the benefits of using the layered modeling approach (Wright & Sanders, 2008). In fact, McCaffrey et al. (2004) and Lockwood, McCaffrey, Mariano, et al. (2007) found estimated teacher effect persistency parameters to be relatively small.

McCaffrey et al. (2004) showed, under certain restrictions, each of the alternative value-added modeling approaches discussed is a special case of the general, variable persistence model (Figure 2.1). Although the multivariate methods allow efficient use of all available data and are typically recommended, they are computationally intensive and require much more in the way of computing resources than either the gain score or covariate adjustment methods. While similar in many regards, each of the multivariate approaches has unique features (Figure 2.2). For instance, unlike the EVAAS teacher and omputationally intensive and require much more in the way of computing resources than either the gain score or covariate adjustment methods.

Without covariates, gain scores and the cross-classified model are special cases of the layered model with restrictions to the overall time trend and/or the distribution of residual errors. The layered model is a special case of the general model with restrictions to the αs and without covariates. The covariate adjustment and gain-score model with covariates are special cases of the general model with restrictions to the distribution of residual errors and the αs.

Note. From “Models for Value-Added Modeling of Teacher Effects,” by D. F. McCaffrey, J. R. Lockwood, D. Koretz, T.A. Louis, and L. Hamilton, 2004, Journal of Educational and Behavioral Statistics, 29(1), p.

78. Copyright 2004 by Sage Publications, Inc. Adapted with permission. McCaffrey et al. (2004) refer to the EVAAS layered model as the TVAAS layered model.

the variable persistence models, the cross-classified framework models individual growth curves for students, so its within-student covariance matrix is restricted. Also, while both the cross-classified and EVAAS teacher models assume teacher effects persist undiminished in the future, the variable persistence model allows the strength of these effects on future scores to vary. Consequently, the variable persistence model only requires scores be linearly related instead of on a single developmental scale (McCaffrey et al., 2004): that is, the cumulative distribution functions for different scales should not cross (McCaffrey et al., 2003). While the additional complexities of these modeling approaches can be beneficial, they also come with the price of additional computational burdens.

2.3 Estimating Teacher Effects Studies investigating value-added teacher effects provide evidence teachers have differing effects on student learning (Rivkin et al., 2005; Rowan et al., 2002; Wright et al., 1997) that persist over time (Sanders & Rivers, 1996), but these studies have shortcomings. Statistical and psychometric issues arise when estimating teacher effects using longitudinal student achievement data (McCaffrey et al., 2003). Different modeling variations, such as the use of layered versus non-layered design matrices and the estimation of random versus fixed teacher effects, are described, and the impact of such considerations on estimated teacher effects follows.

2.3.1 Layered Models Wright and Sanders (2008) distinguish between the layered and non-layered model in the construction of the coefficient matrix for teacher effects (Table 2.1). In the non-layered model, each student’s outcome in a given year is linked only to the current teacher. In contrast, the layered model links a student’s achievement to current and previous teachers within a given time span. Therefore, the coefficient matrix for the layered model can have several “1”s in a row, connecting past teachers with subsequent student outcomes. This approach accounts for the “correlation of future scores for students who [have] shared a past teacher” (Lockwood, McCaffrey, Mariano, et al., 2007, p. 126).

Teacher effects may be specified as either fixed or random effects depending on the intended scope of inference. If teacher effects are treated as fixed, the underlying assumption is the observed teachers are the only units of interest; conclusions drawn apply only to the teachers for whom data were collected. If teacher effects are specified as random, it is assumed the observed teachers represent a random sample from a population of teachers to which conclusions can be applied. Random teacher effects are estimated by best linear unbiased predictors (BLUPs), which are also referred to as shrinkage estimators (Raudenbush & Byrk, 2002; Robinson, 1991). This estimation procedure weights the average deviations of each teacher’s students’ scores from the overall average score. The weighting takes into account sample size and variability within, as well as across, teachers’ classrooms to “shrink” teacher effect estimates to the overall mean of the distribution of teacher effects, assumed to be zero. Each teacher’s random effect is estimated relative to all other teachers in the sample, and the variability of the estimated teacher effects is assumed to represent variability present in the population of teachers. Such shrinkage and relative estimation does not occur in the estimation of fixed effects.

2.3.3 Impact of Model Specification on Teacher Effect Estimates Wright and Sanders (2008) explain the EVAAS model in further detail by comparing it to three other sub-models. The researchers use simple examples to illustrate the effect layered design matrices and/or within-student correlations have on estimated teacher effects. In these examples, Wright and Sanders (2008) use a special case of the

where y is a vector of test scores, X tracks the year of each test score, β is a vector of overall mean scores for each year and Z is the coefficient matrix for u, the vector of random teacher effects, assumed to be normally distributed with E (u) 0 and

the number of students with test scores and s is the number of test scores for each student;

residuals from different students are assumed to be independent, but measures on the same student are assumed to be correlated. The overall variance of test scores is

components are not estimated from the data, because the focus is on the estimation of fixed means and random teacher effects. In each of the examples, the ratio of teacher variance to residual variance, , is fixed at 1000 to reduce the amount of shrinkage in the teacher effect estimates and, subsequently, make the estimates easier to interpret; the within-student correlation, , is specified as either 0 or 0.7. Substituting V0 for V, R 0 for R and I t for D in the estimating equations for fixed and random effects does not change the estimates produced.

One of the examples involves three years of scores for nine different students (Figure 2.3). These data can also be found in Table 10 of Wright and Sanders (2008). In each year there are three different teachers: teachers A, B and C in year one; teachers D, E and F in year two; and teachers G, H and J in year three. As shown in Figure 2.3, the teachers fall into two different tracks: students who have teacher A or B in year one do not share any teachers over the course of the three years with the student who has teacher C in year one; students who have teacher A or B in year one either have teacher D or E in year two and teacher G or H in year three, whereas the student who has teacher C in year one has teachers F and J in years two and three, respectively. Subsequently, the estimated random effects of teachers depend on deviations for students in the teachers’ respective tracks, where deviations are the differences between students’ scores and the corresponding yearly mean. The students’ scores are also assumed to be scaled so changes in scores from year to year are meaningful.

Estimates of random teacher effects are compared for four different models: 1) the non-layered model (NLM) with within-student correlations set to zero ( 0), abbreviated NLM(0); 2) the non-layered model with 0.7, abbreviated NLM(0.7); 3) the layered model (LM) with 0, abbreviated LM(0); and 4) the layered model with 0.7, abbreviated LM(0.7). For the previously described data, Wright and Sanders (2008) provide formulas for calculating the effects of the second and third year teachers using each of the four models. Estimated teacher effects during the first year are not value-added and, while estimated, are not discussed; students do not have previous scores from which to determine the value a year one teacher has added. In the formulas, “M” represents the mean of the deviation scores y Xβ for a group of students, the subsequent integer indicates the year for which the deviations were calculated and the subscript denotes the students’ corresponding teacher(s). For example, M2D represents the average year two deviation score of students taught by teacher D, and M1DE represents the average year one deviation score of students taught by teachers D and E in year two. Differences in teacher effect estimating equations for the four models are discussed using this notation.

The NLM(0) teacher effect estimates are merely unadjusted means of students’ deviation scores. For instance, the estimated random effect of teacher D in year two is the mean deviation score of students taught by teacher D during year two, M2D = Σ(Score – Year Two Fixed Effect Mean) / Number of Students = {(440 – 500) + (380 – 500) + (500 – 500) + (440 – 500)} / 4 = -60.

Similarly, the estimated random effect of teacher G is the average deviation score of the students he or she taught in year three,

= -60.

The NLM(0) estimates do not incorporate expected deviation scores, because withinstudent correlations are assumed to be zero; no basis exists for which to obtain such expected deviations. Consequently, teachers with generally higher achieving students will appear to be more effective than teachers with generally lower achieving students, even when the latter have huge impacts on their students’ learning.

The NLM(0.7) teacher effect estimates adjust the means of students’ deviation scores in a given year to account for students’ deviation scores in other years, as in analysis of covariance (ANCOVA). For example, the estimated random effect of teacher D in year two is the mean deviation score of students taught by that teacher during that time adjusted by the students’ expected deviation scores. The formula,

can be interpreted in three parts: M2D, b(M1D – M1DE) and b(M3D – M3DE). The mean deviation score of students taught by teacher D in year two, M2D, is -60; this is the NLM(0) estimate for teacher D.

However, M2D is adjusted by the expected year one deviation score for students taught by teacher D in year two, b(M1D – M1DE). The coefficient, b, is the pooled withinteacher multiple regression coefficient for predicting students’ year two deviation scores from their corresponding year one and year three deviation scores. In this example, b = / (1 + ) = 0.7 / (1 + 0.7) = 0.411765. The mean year one deviation score of students

**taught by teacher D in year two is:**

M1D = Σ(Year One Score for Students Taught by Teacher D – Year One Fixed

= {(370 – 400) + (350 – 400) + (430 – 400) + (410 – 400)} / 4 = -10.

The expected year one deviation score also incorporates the mean year one deviation score of students taught by teacher D or E in year two, M1DE = {Σ(Year One Score for Students Taught by Teacher D – Year One Fixed Effect Mean) + Σ(Year One Score for Students Taught by Teacher E – Year

= {(370 – 400) + … + (410 – 400) + (330 – 400) + …. + (370 – 400)} / 8 = -30.

Hence, the students taught by teacher D in year two had an above-average deviation score in year one, b(M1D – M1DE) = 0.411765{(-10) – (-30)} = 8.235.

The mean, M2D, is also adjusted by the expected year three deviation score for students taught by teacher D in year two, b(M3D – M3DE). The mean year three deviation

**score of students taught by teacher D in year two is:**

M3D = Σ(Year Three Score for Students Taught by Teacher D – Year Three

= {(470 – 600) + (530 – 600) + (530 – 600) + (590 – 600)} / 4 = -70.

This mean is then compared to the mean year three deviation score of students taught by teacher D or E in year two, M3DE = {Σ(Year Three Score for Students Taught by Teacher D – Year Three Fixed Effect Mean) + Σ(Year Three Score for Students Taught by Teacher E

= {(470 – 600) + … + (590 – 600) + (550 – 600) + …. + (670 – 600)} / 8 = -30, to produce an expected year three deviation score of 0.411765{(-70) – (-30)} = -16.471.

Together, these three parts comprise the NLM(0.7) estimate for the random effect

**of teacher D in year two:**

= -60 – 0. 411765{(-10) – (-30)} – 0.411765{(-70) – (-30)} = -60 – 0. 411765{20} – 0.411765{-40} = -51.8.

Although slightly higher than teacher D’s NLM(0) estimate of -60, the estimate of -51.8 still indicates teacher D had a lower than average teacher effect. The NLM(0.7) estimate is simply an adjusted mean of the students’ deviation scores. Although the adjustments based on expected deviations are conventional, some properties of these adjustments are undesirable. For instance, future scores are used to adjust for earlier scores, even though the covariate is potentially affected by the prior year teacher; this is typically inappropriate for ANCOVA and may also be inappropriate in this context. The NLM(0.7) estimate for the effect of teacher D uses the students’ third year deviation scores to adjust for their second year deviation scores, even though teacher D’s impact may also affect student performance in year three. Additionally, the NLM(0.7) estimates reduce a teacher’s effect if his or her students’ subsequent deviation scores are above-average, i.e., a teacher is penalized if students go on to have higher than expected gains.