This dissertation examined the effect of contextual features of the classroom environment on measures of teacher quality derived from classroom observation instruments.Using data on 228 teachers observed four times as part of the Understanding Teacher Quality (UTQ) study, scores from the Classroom Assessment Scoring System, the Framework for Teaching, and the Protocol for Language Arts Teaching Observation were analyzed using Generalizability Theory (GTheory) statistical models.The goals of these analyses were to examine how the reliability, validity, and bias of teacher quality estimates shifted as GTheory models successively incorporated a variety of contextual features of measurement present across specific occasions of observation.The contextual features of measurement examined included: (1) observation system design (SD) variables such as time of year and methods of scoring; (2) variables measuring curricular and instructional (CI) practices, such as whether a lesson focused on reading or writing or included discussion or lecture; and (3) variables measuring features of school organization (SO) such as classroom student composition.Through comparison of models that successively adjusted for SD, CI, and SO variables, it was found that for all three observation instruments, over 20% of the variance attributed to teacher effects in traditional GTheory statistical models was due to sampling error stemming from values of the SD and CI variables recorded for a teacher on a given occasion of measurement, implying that traditional GTheory approaches (that do not incorporate SD and CI variables) can result in positively biased estimates of the reliability of teacher quality scores.However, this dissertation also found that teacher quality scores adjusted for SD and CI variables were highly correlated to teacher quality scores from GTheory models that did not adjust for SD and CI variables, in part because occasions of observation were selected at random during the UTQ data collection period.Inclusion of SO variables into GTheory statistical models had more far-reaching consequences. To begin, SO variables (and especially student composition) explained ~40% of the variance in teacher quality estimates, changed point estimates considerably, and reduced the reliability of these estimates to very low levels under normal observational designs.However, the decision as to whether or not to include SO facets into GTheory statistical models necessarily involves assumptions about whether these differences in observed teaching quality are driven by teacher sorting (e.g. better teachers are teaching more advantaged students) or co-construction (e.g., more advantaged students co-construct instruction along with the teacher, making instruction of higher measured quality in more advantaged settings).If one assumes that co-construction drives the observed effects of SO variables on teacher quality estimates, then these variables should be included in a GTheory model, but such inclusion will make teacher quality estimates very unreliable.If, on the other hand, one assumes that teacher selection drives the statistical relationship between SO variables and teacher quality estimates, inclusion of SO variables into a GTheory model would bias teacher quality estimates by removing this selection effect.Overall, the results presented in this dissertation highlight the subtle ways that SD, CI, and SO variables can affect teacher quality estimates and how these subtle differences can affect the reliability, validity, and bias of teacher quality measures derived from classroom observation instruments.
【 预 览 】
附件列表
Files
Size
Format
View
Generalizability of Scores from Classroom Observation Instruments
PDF
download
878987797
028-85220240
