«University of Nebraska - Lincoln DigitalCommons of Nebraska - Lincoln Dissertations and Theses in Statistics Statistics, Department of 8-2010 ...»
0.4 0.2 0.2 0.0 0.0 Upper Quartile for the Curve-of-Factors (solid), CA Z-score (dash) and MA Z-score (dotted) Models with Complete and Missing Tests, and MA/CA Z-score (dot-dash) Model with Missing Tests for Specific True Percentiles and Years factors model when all tests are given tends toward that of the alternative Z-score models when a test is missing in both years one and four. With regards to estimating a teacher’s true percentile, assuming a curve-of-factors model structure from which the data are originally simulated loses its statistical advantages, such as smaller RMSE, when tests are missing in some of the years. The results from the Z-score models are not noticeably different from one another for both the complete and missing tests cases. This is expected, because the only difference between a student’s mean-centered MA and CA scores for a given year is the random measurement error, assumed to have constant variance across test and year. The lack of differences between Z-score models in the complete and missing tests cases is also expected. Because only one test can be used as the response in each year, having only one test in a given year instead of two is not consequential. Overall, the curve-of-factors model does not have drastically better performance relative to the alternative Z-score approach, even though the curve-offactors model is the structure from which the data are simulated.
Chapter 5 Conclusions Value-added modeling techniques estimate the contribution of educational factors, such as teachers, to growth in student achievement, while allowing for the possibility to control for the effect of non-educational factors. Several value-added models for estimating teacher effects have been proposed as alternatives to current testbased accountability procedures, such as adequate yearly progress (AYP), but each has its respective advantages and disadvantages. Although these methods have the potential to identify highly effective teachers, teacher effect estimates are sensitive to different modeling specifications, including the persistency of teacher effects. Furthermore, several statistical and psychometric issues exist, and sensitivity of teacher effects to such issues still needs to be explored.
This dissertation includes three chapters that provide an introduction to valueadded methodology and discuss the estimation of teacher effects. Because value-added analyses require high-quality longitudinal data that are often not available, Chapters 3 and 4 proposed methodology for analyzing less-than-ideal student assessment data.
Specifically, Chapter 3 described how to use a value-added model when longitudinal student achievement data are not on a single developmental scale. Although the Z-score approach has limitations, it is an appropriate alternative to using raw data when analyzing less-than-ideal student achievement data across a mixture of norm- and criterion-referenced tests over time. This chapter addressed issues arising when using a layered, longitudinal linear mixed model to analyze gains in standardized scores, including weighting considerations for variance components. Additionally, this chapter proposed methods for estimating teacher effects on student learning before and after teacher participation in professional development programs. Although the specific example used in this chapter did not indicate the Math in the Middle Institute had a significant impact on participating teachers’ effects on student learning, the instruments may not have been designed to detect such changes. Additionally, teacher change from professional development may take time to show an impact on student learning. It is also possible other factors impact a teacher’s instruction and his or her ability or perceived ability to change in ways that align with the professional development program (Kromminga, in progress). When utilizing this methodology, determining whether the goals of the program align with what the instruments assess and acknowledging any existing limitations is essential.
Chapter 4 applied curve-of-factors methodology in a value-added context to extend the analysis of student achievement data to situations in which multiple tests with potentially different scales are given each year in a particular subject. Instead of estimating a teacher’s effect on changes in a student’s scores over time, the curve-offactors model allowed the estimation of a teacher’s effect on changes in some common, latent trait measured by the multiple instruments across years. In the simulation study, the behavior of the curve-of-factors model when all tests were given tended toward that of the alternative Z-score models when a test was missing in both years one and four. With regards to estimating a teacher’s true percentile, assuming a curve-of-factors model structure from which the data were originally simulated lost its statistical advantages, such as smaller RMSE, when tests were missing in some of the years. In general, the curve-of-factors model did not have drastically better performance relative to the alternative Z-score approach, even though the curve-of-factors model was the structure from which the data were simulated.
Together, these two approaches address concerns surrounding the estimation of value-added teacher effects when analyzing less-than-ideal student achievement data.
However, considerations should be made when defining what teacher effects really describe, and teacher effect estimates should be linked to other valid measures of teacher effectiveness. Although standardized tests can provide useful information about a student’s content knowledge in a particular subject, they can only measure a few of the many skills teachers help shape and influence in their students. Teacher evaluation should reflect these many areas of instruction and learning and not be based solely on students’ performance on achievement tests. The uncertainty associated with value-added teacher effect estimates suggests such estimates should not be used in isolation of other measures for high-stakes evaluation purposes and decisions. Instead, value-added teacher effects could be one of many different aspects used for teacher development purposes, fostering high quality teaching and motivating informed improvements in education.
References Ackerman, T., Heafner, T., & Bartz, D. (2006, April). Teacher effect model for impacting student achievement. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
Ball, D. L., Hill, H. C., & Bass, H. (2005). Knowing mathematics for teaching: Who knows mathematics well enough to teach third grade, and how can we decide?
American Educator, 29(3), 14-22, 43-46.
Ballou, D., Sanders, W., & Wright, P. (2004). Controlling for student background in value-added assessment of teachers. Journal of Educational and Behavioral Statistics, 29(1), 37-65.
Borko, H. (2004). Professional development and teaching and learning: Mapping the terrain. Educational Researcher, 33(8), 3-15.
Carey, K. (2004). The real value of teachers: Using new information about teacher effectiveness to close the achievement gap. Thinking K-16, 8(1), 3-32.
Cheong, J., MacKinnon, D. P., & Khoo, S. T. (2003). Investigation of meditational processes using parallel process latent growth curve modeling. Structural Equation Modeling, 10(2), 238-262.
Darling-Hammond, L. (2000). Teacher quality and student achievement: A review of state policy evidence. Education Policy Analysis Archives, 8(1). Retrieved April 7, 2008, from http://epaa.asu.edu/ojs/article/view/392 Desimone, L. M., Porter, A. C., Garet, M. S., Yoon, K. S., & Birman, B. F. (2002).
Effects of professional development on teachers’ instruction: Results from a threeyear longitudinal study. Educational Evaluation and Policy Analysis, 24(2), 81-112.
Doran, H. C. (2003). Adding value to accountability. Educational Leadership, 61(3), 55Drury, D., & Doran, H. C. (2003). The value of value-added analysis. Policy Research Brief for National School Board Association, 3(1), 1-4.
Duncan, S. C., Duncan, T. E., & Strycker, L. A. (2000). Risk and protective factors influencing adolescent problem behavior: A multivariate latent growth curve analysis.
Annals of Behavioral Medicine, 22(2), 103-109.
Fishman, B. J., Marx, R. W., Best, S., & Tal, R. T. (2003). Linking teacher and student learning to improve professional development in systematic reform. Teaching and Teacher Education, 19(6), 643-658.
Frome, P., Lasater, B., & Cooney, S. (2005). Well-qualified teachers and high quality teaching: Are they the same? Atlanta, GA: Southern Regional Educational Board.
Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915-945.
Gilmour, A. R., Gogel, B. J., Cullis, B. R., & Thompson, R. (2009). ASReml user guide, Release 3.0. Hernel Hempstead, HP1 1ES, UK: VSN International Ltd.
Green, J. L., Smith, W. M., Heaton, R. M., Jiao, S., & Stroup, W. W. (under review).
Estimating the impact of a professional development program on student learning using a value-added model. Submitted to Journal of Educational and Behavioral Statistics.
Guskey, T. R. (1994, April). Professional development in education: In search of the optimal mix. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
Hershberg, T., Simon, V.A., & Lea-Kruger, B. (2004). Measuring what matters.
American School Board Journal, 191(2), 27-31.
Hill, H. C. (2007a). Learning in the teaching workforce. The Future of Children, 17(1), 111-127.
Hill, H. C. (2007b). Mathematical knowledge of middle school teachers: Implications for the No Child Left Behind policy initiative. Educational Evaluation and Policy Analysis, 29(2), 95-114.
Hill, H. C., Rowan, B., & Ball, D. L. (2005). Effects of teachers’ mathematical knowledge for teaching on student achievement. American Educational Research Journal, 42(2), 371-406.
Hill, H. C., Schilling, S. G., & Ball, D. L. (2004). Developing measures of teachers’ mathematics knowledge for teaching. The Elementary School Journal, 105(1), 11-30.
Kromminga, B. (in progress). Down the road of change: A mixed methods study of teacher practices and beliefs following intensive professional development.
Unpublished doctoral dissertation, University of Nebraska-Lincoln, Lincoln, NE.
Laird, N. M., & Ware, J. H. (1982). Random-effects models for longitudinal data.
Biometrics, 38(4), 963–974.
Lewis, W. L., Heaton, R., McGowan, T., & Jacobsen, B. (2004). Math in the Middle Institute Partnership. NSF grant proposal project description.
Lissitz, R. W. (Ed.). (2005). Value added models in education: Theory and applications.
Maple Grove, MN: JAM Press.
Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., & Schabenberger, O.
(2006). SAS for mixed models (2nd Ed.). Cary, NC: SAS Institute Inc.
Little, T. D., Bovaird, J. A., & Slegers, D. W. (2006). Methods for the analysis of change.
In D. K. Mroczek & T. D. Little (Eds.), Handbook of personality development (pp.
181-211). Mahwah, NJ: Lawrence Erlbaum Associates.
Lockwood, J. R., Louis, T. A., & McCaffrey, D. F. (2002). Uncertainty in rank estimation: Implications for value-added modeling accountability systems. Journal of Educational and Behavioral Statistics, 27(3), 255-270.
Lockwood, J. R., McCaffrey, D. F., Hamilton, L. S., Stecher, B., Le, V., & Martinez, J. P.
(2007). The sensitivity of value-added teacher effect estimates to different mathematics achievement measures. Journal of Educational Measurement, 44(1), 47Lockwood, J. R., McCaffrey, D. F., Mariano, L. T., & Setodji, C. (2007). Bayesian methods for scalable multivariate value-added assessment. Journal of Educational and Behavioral Statistics, 32(2), 125-150.
Loucks-Horsley, S., Stiles, K., & Hewson, P. (1996). Principles of effective professional development for mathematics and science education: A synthesis of standards. NISE Brief, 1(1), 1-6.
Martineau, J. A. (2006). Distorting value added: The use of longitudinal, vertically scaled student achievement data for growth-based, value-added accountability. Journal of Educational and Behavioral Statistics, 31(1), 35-62.
McCaffrey, D. F., Lockwood, J. R., Koretz, D. M., & Hamilton, L. S. (2003). Evaluating value-added models for teacher accountability. Santa Monica, CA: RAND Corporation.
McCaffrey, D. F., Lockwood, J. R., Koretz, D., Louis, T. A., & Hamilton, L. (2004).
Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29(1), 67-101.
National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National Mathematics Advisory Panel. Washington, DC: U.S. Department of Education.
No Child Left Behind. (2001). Washington, D.C.: U.S. Department of Education.
Pearson. (2008). Metropolitan Achievement Tests®, eighth edition (MAT 8). Retrieved April 28, 2008, from http://harcourtassessment.com/haiweb/cultures/enus/productdetail.htm?pid=E164C Presley, J. B., White, B. R., & Gong, Y. (2005). Examining the distribution and impact of teacher quality in Illinois. Edwardsville, IL: Illinois Education Research Council.
Rasbash, J., & Browne, W. J. (2008). Non-hierarchical multilevel models. In J. de Leeuw & E. Meijer (Eds.), Handbook of multilevel analysis (pp. 301-334).
Raudenbush, S. W. (2004). What are value-added models estimating and what does this imply for statistical practice? Journal for Educational and Behavioral Statistics, 29(1), 121-129.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
Rivkin, S. G., Hanushek, E. A., & Kain, J. F. (2005). Teachers, schools, and academic achievement. Econometrica, 73(2), 417-458.
Robinson, G. K. (1991). That BLUP is a good thing: The estimation of random effects.
Statistical Science, 6(1), 15-51.
Roesch, S. C., Norman, G. J., Adams, M. A., Kerr, J., Sallis, J. F., Ryan, S., Calfas, K. J., & Patrick, K. (2009).
Latent growth curve modeling of adolescent physical activity:
Testing parallel process and mediation models. Journal of Health Psychology, 14(2), 313-325.
Rowan, B., Correnti, R., & Miller, R. J. (2002). What large-scale, survey research tells us about teacher effects on student achievement: Insights from the Prospects study of elementary schools. Teachers College Record, 104(8), 1525-1567.
Sanders, W. L. (2000). Value-added assessment from student achievement data:
Opportunities and hurdles. Journal of Personnel Evaluation in Education, 14(4), 329Sanders, W. L. (2004). A summary of conclusions drawn from longitudinal analyses of student achievement data over the past 22 years (1982-2004). Paper presented at the Governors Education Symposium, Ashville, NC.
Sanders, W. L. (2006, October). Comparisons among various educational assessment value-added models. Paper presented at the Battelle for Kids National Value-Added Conference, Columbus, Ohio.
Sanders, W. L., & Rivers, J. C. (1996). Cumulative and residual effects of teachers on future student academic achievement. Knoxville, TN: University of Tennessee ValueAdded Research and Assessment Center.
Sanders, W. L., Saxton, A. M., & Horn, S. P. (1997). The Tennessee value-added assessment system: A quantitative outcomes-based approach to educational assessment. In J. Millman (Ed.), Grading teachers, grading schools: Is student
achievement a valid educational measure? (pp. 137-162). Thousand Oaks, CA:
SAS Institute Inc. (2008). SAS/STAT user’s guide, Version 9.2. Cary, NC: SAS Institute Inc.