MethodologyData for the evaluation of the Kalamazoo Promise will be collected through five main activities:
Survey of Middle and High School StudentsThe Evaluation Center will survey high school students on an annual basis to establish a baseline with which to compare the effects of the Kalamazoo Promise on future students. The purpose of the survey is to capture student aspirations, teacher expectations for students, and school climate, and to collect information from students regarding their plans following graduation, and intentions to pursue postsecondary education. Students will also be asked to recall and report on changes in their schools since the announcement of the Kalamazoo Promise. Information also will be collected that will help develop and refine the scholarship program. Surveys in future years will include questions about perceived obstacles to using the Kalamazoo Promise. All students will be sampled. The survey will be anonymous and will not be linked to student-level information. It will include key background items so the results can be disaggregated and correlational and multivariate analyses can be conducted. Survey of Middle and High School Teachers and StaffThe Evaluation Center will develop and administer an electronic survey for middle and high school teachers and staff. The survey would collect information on perceived changes in the district, including changes in student aspirations, teacher expectations, responses from the community and from the district itself. Information also will be collected that will help develop and refine the scholarship program. All teachers and staff in the middle and high schools would be included in the sample, and the survey would be confidential, meaning that only evaluators could link information to staff names. The reason we would make this survey confidential and not anonymous is that we would need to know which teachers have not responded to the electronic survey in order to target follow-up reminders. Although no names or personal information will be attached to the findings, we will collect some data on general background items so that the results can be disaggregated and correlational and multivariate analyses can be conducted. Interviews with Key InformantsA purposive sample of 60-70 key informants from the community and from the local school district will be identified for interviews. A semistructured interview protocol will be developed to ensure that relevant and critical topics are covered. Key informants from the community are likely to be civic and local government officials, leaders of community organizations, faith-based groups, and neighborhood associations. Within the district, key informants will include district officials, school board members, principals, and guidance counselors. Data from interviews will be entered and coded for thematic analysis and consistency of patterns within narrative data. Review of Public and District DocumentationProgram and school district documentation would be reviewed for the evaluation. Additionally, a review of media coverage would be conducted in order to help gain an understanding of the scope of the potential impact of the Kalamazoo Promise. Additionally, we will request relevant documentation in connection with the interviews of key informants. The documentation would be thematically coded and summarized. Archiving these public documents allows us to track growth over time within the district, as well as the evolution of community groups’ response to the scholarship. It also serves as a source of data to triangulate our findings from the other data collection activities in this evaluation. Analysis of School-Level Available DataThe Evaluation Center would conduct a secondary analysis of school level data each year. This would track the performance of each of the district schools over the past 3-5 years relative to how demographically similar schools in Michigan perform. We have developed a rather sophisticated methodology called "residual gains analysis" that we would use. This type of analysis uses school level test and demographic data that we can obtain from the state department of education, so the district is not burdened with a request for data. To estimate the effect of the scholarship, we developed a set of statistical “filters” that remove most of the changes in student composition in the Kalamazoo schools over time. The remaining portion of the score changes provides a reasonable (though not foolproof) estimate of program effectiveness. While calculating the filtered scores requires statistical techniques, the basic idea is relatively simple. The filters work by comparing each district school with a set of demographically and geographically similar non-Kalamazoo schools. Instead of focusing on absolute scores or values, the filtered scores focus on the differences between each Kalamazoo school and a specially selected comparison group created using regression analyses. Variables used in the filters will be obtained from state and federal databases and include family income, ethnicity, special education status, urbanicity code, and school enrollment. Inasmuch as the comparison schools will be similar to the district schools in most relevant respects save for not being a Promise school, the filtered (difference) scores will provide the best approximation of the program effect using group or school-level data. Another advantage of the filtered scores is that they have a straightforward interpretation. Since the filtered scores represent the difference between a district school and its specially selected comparison group, a score of zero indicates that the school is performing exactly on par with its comparison group. Unlike most test metrics, filtered or residual scores can take negative values, indicating that a school’s performance is below that of its comparison group. For example, a filtered score of -15 indicates that the average student in a Kalamazoo school scored 15 points lower than predicted or relative to the average student in the school’s comparison group. By contrast, a filtered score of 25 indicates that the average student in a Kalamazoo school scored 25 points higher than the average student in the school’s comparison group. Changes in scores or indicators over time provide a better estimate of value added than a snapshot from a single point in time. Each Kalamazoo school gain in residual scores over time indicates that the average student score or school-level indicator is catching up with the average student or the average school in the school’s comparison group. Similarly, declines in a school’s residual scores suggest that it is falling behind the average school in the comparison group. In addition to this data collection, The W. E. Upjohn Institute for Employment Research also intends to examine changes in student-level indicators. |