Methodology

The identification of indicators and outcome measures was done collaboratively by members of the evaluation team who have extensive experience in either school reform, instructional leadership, professional development of teachers, and/or evaluation. In this collaborative process, we used the already drafted theory of change model to guide the selection of indicators and suitable benchmarks. Staff from The Rapides Foundation were involved before any final decisions were made regarding the selection of indicators and outcome measures. We anticipate that, over time, joint decisions will need to be made regarding the addition, subtraction, or revision of indicators and outcome measures.

It is important to note that the methods detailed in this section are part of a cycle that will be repeated throughout the duration of the evaluation. Each year, the data collection instruments will be revised according to on-the-ground activities.

Many of the school and parish level indicators are readily available and it will be important to design data collection instruments that can capture other necessary indicators and outcome measures. We will use extensive secondary sources of data to cover student achievement as well as a whole array of indicators regarding school/parish demographics, teacher qualifications, and other related performance issues. Specifically, we will develop and use the following key instruments: teacher/principal survey, interview protocols for teachers, school administrators, superintendents, and technical assistance providers, and site visit checklist. Additionally, we will develop a survey in collaboration with TRF staff to evaluate the performance of the evaluation team. This will be administered to a sample of stakeholders at the end of each academic year. The sample size also will be determined collaboratively with TRF staff. Upon completion, we will return the surveys directly to TRF for analysis.

Teacher and principal surveys.  The teachers and principal surveys are the Web-based- equivalent to an 8-9 page survey. The follow-up of nonrespondents will use a combination of e-mail reminders and regular mail paper surveys. The teacher and the principal surveys cover the following topics and areas:

• Key background indicators such as professional background and demographics
• Nature and extent of involvement in professional development activities, including activities not sponsored by the Initiative
• Satisfaction with professional development and support and technical assistance provided by the Systemic Initiative
• Self-reported impact of the professional development and support and technical assistance provided by the Initiative
• Perceived changes in the school and performance of students
• Attitudes toward profession and plans for future
• Areas for further professional development and technical support for subsequent years

While most items will remain the same, as we will be tracking trends, it will be necessary to modify the survey each year to address new questions raised by early findings, and also to address changes in the Systemic Initiative.

Interviews.  Interview protocols will be developed for teachers, principals, superintendents, and technical assistance providers that cover issues related to the quality of and satisfaction with professional development and training in instructional leadership as well as the extent and fidelity of implementation. These interviews also will be used to verify and clarify data collected from other sources, such as the surveys and the analysis of secondary data.

Site visits.  Site visits will (i) observe classrooms using the LINCS classroom observation checklist, (ii) conduct a "walk-through" evaluation using a checklist of items to observe in the school and an abbreviated list of questions to ask staff and students at random, and (iii) conduct an in-depth interview with a key administrator and brief interviews with a small sample of teachers. If possible and when they are available, we would also seek to discuss the systemic initiative with parents, particularly when district grants have activities intended to increase parental involvement. Although there is a limited time frame for data collection in the first year, evaluation team members will avoid scheduling site visits during the time when schools are involved with the state assessments.

The 9- parish service area contains 139 schools; 60,197 students; and an estimated 4,000 classroom teachers. This illustrates that the Initiative is large and seeks to impact a large portion of the state. Because of the importance of determining implementation and capturing school level factors and the need to provide formative feedback, we sample 75 schools that will be visited for data collection twice (timing is explained below). Decisions regarding the final selection of schools will involve input from the Foundation, the Evaluation Advisory Group, and the participating parishes.

As can be seen in Table 1, the sampled schools will be visited twice with 3 years between each site visit. Cohort I will have 35 schools and will be visited in year 1 and again in year 4. Cohort II will have 40 schools and will be visited in years 2 and 5. Because of the plans to focus greater attention on high school intervention, we will wait to visit the high schools in Cohort II. This means that by year 5 these high schools will have their second visit and there will have been more time for implementation and visible impacts. Although 75 schools will be the focus of the site visits, we will also gather data regarding implementation via surveys of staff and key administrators.

Table 1.  Schedule of School and District Site Visits

Sample	Year 1	Year 2	Year 3	Year 4	Year 5
Site visits: Cohort I (35 schools)	X			X
Site visits: Cohort II (40 schools)		X			X
Success cases (10-20 schools)			X
Visits to parishes/districts	X	X	X	X	X
Interviews with technical assistance providers	X	X	X	X	X
Electronic and follow-up paper surveys for teachers	X	X	X	X	X
Electronic and follow-up paper surveys for key administrators	X	X	X	X	X
Survey of stakeholders regarding performance of evaluation team	X	X	X	X	X

In-depth case studies of schools with successful implementation will be conducted in year 3. We will focus on specific teachers, schools, and districts that are successfully implementing new practices that reflect the professional development and instructional leadership provided by the Systemic Initiative. Decisions regarding which cases will be examined will be based on data collected in years 1 and 2, as well as from input from parish offices. It is possible that cases will be highlighted that are among the 75 schools visited in years 1 and 2. A total of 53 days of fieldwork for case studies are planned for year 3. This should be sufficient for capturing and sharing successful implementations of new practices.

Data will be entered and analyzed by members of the evaluation team. The annual survey of teachers and administrators is going to require more in-depth analysis, beyond simple descriptive statistics. Here we will need to use correlational and multivariate analysis as we examine subgroups of staff and schools, and as we look at teacher- and school- level factors and their relationship with immediate, intermediate, and long-term outcomes. Findings from the in-depth analysis of survey data will be included in our annual reports. Initial results from these surveys will be provided back to TRF and-if deemed appropriate-to the districts and schools as soon as they are available. The results will be disaggregated by school and parish, and comparisons can be made with overall findings.