In designing the evaluation strategy, the evaluation team was mindful of the need to balance the interests of evaluators, the client (ISBE), and the charter schools that are the subject of this study.  Given the high profile nature of the reform, charter school officials are subject to requests for information from many sources.  While perhaps flattering at first, such demands can  place significant burdens on the schools. The team has sought to employ data collection methods that are as unobtrusive as possible while fulfilling its obligations to ISBE.  Evaluation teams generally include 4 or 5 persons, allowing the team to collect the information it needs in approximately 2 hours.  Moreover, the evaluation relies heavily on data the schools already submit to the state.  Thus, much of the information presented in this project did not require any extra effort on the part of the charter schools.

The range of evaluation questions addressed in this project requires a multimethod approach.  Questions dealing with student achievement, for instance, are amenable to quantitative analysis of standardized test scores. By contrast, questions about teaching methods, professional opportunities for teachers, and learning opportunities require a mix of quantitative survey analysis and various qualitative techniques.  Data collection strategies employed in the evaluation included the following:
 

• Secondary data analysis
• Surveys 
• Interviews
• Document review

• Secondary data analysis.  All public schools in Illinois including charter schools submit information on a wide variety of topics to ISBE for inclusion in the state's school report card system.  Staff at The Evaluation Center obtained the electronic data files underlying the report cards and extracted a number of variables relevant to the evaluation. These include scores on the state's system of standardized tests, student demographic data, and other school characteristics.  This approach not only reduces the evaluation's burden on the charter schools, but also allows the evaluation team to make comparisons among charter schools and between charter and noncharter public schools. 

        [top]
 
• Surveys.  The evaluation team visits the charter schools approximately once each year in order to administer surveys, conduct interviews with charter school chief administrative officers (CAOs), and collect documentation. Surveys allow evaluators to efficiently gather information on participants'perceptions of the inner workings of charter schools. The Evaluation Center will prepare templates to report the results back to each individual school as well as to ISBE. Thus far, the project has employed two survey instruments: 

 
• Teachers/staff charter school survey (381 KB PDF).  Teacher/staff surveys are administered to all teachers and school personnel who work more than 5 hours per week.  This includes administrative and professional support personnel.  Respondents are asked to complete the questionnaire, enclose it in an envelope, and then return it to a designated person at the school. A cover letter explains the purpose of the survey, and each teacher/staff member  receives an addressed and stamped envelope in which to enclose the survey. To ensure confidentiality, teachers/staff are instructed not to place their names on the questionnaire, although they are asked to check their name off on a list so that the evaluation team can trace and follow up with missing respondents. The completed forms are collected, sealed, and mailed to the evaluation team by a designee at each school. 

 
• Student charter school survey (97 KB PDF).  Student surveys are administered by a member of the evaluation team to approximately three classes of students in Grades 5-12 at each charter school.  This means that a few schools that cater only to lower elementary grades will not be included in the sample. Each team member receives supervised training and masters materials in afield manual before administering surveys.  Team members explain the purpose of the survey and the manner in which the results will be used before students begin completing the forms.  In addition, team members read questions item by item to students in Grades 5 and 6, or wherever teachers and administrators judge that students would have trouble comprehending the questions.  Students in Grades 7-12 typically can complete the questionnaires on their own, after initial instructions.  Team members answer questions posed by students of all grades and have been briefed on standard answers to typical questions before going out into the field. 

 

 
 
 

During the second year of the evaluation the evaluation team will also administer a parent survey and the National Association of Secondary School Principals' School Climate Survey. 
 

• Parent/guardian charter school survey (99 KB PDF).  Parent/guardian surveys are administered to 25 to 35 families, depending on the size of the school.  Families are selected randomly by members of the evaluation team in order to ensure that the sample is representative. A cover letter explains the purpose of the survey, and each parent receives a self-addressed, stamped return envelope in which to enclose the survey.  Each survey packet includes  two dollars as a expression of the evaluation team's gratitude for parents taking the time to complete and return the survey. 

 
• The School Climate Survey is a nationally normed instrument that has been used extensively in public schools, particularly to measure the impact of new interventions and reform strategies. This instrument will be used only in the second year of the study and will be completed by all teachers/staff at each school, as well as a sample of students (3 classes grades 5 and above) and parents (25 per school). This instrument was originally developed at The Evaluation Center, but The National Association of Secondary School Principals (NASSP) now retains the copyright for this instrument.This instrument has been widely used for more than 12 years, and analysis of the results indicate that this instrument is very reliable.  Like the earlier instruments, The Evaluation Center will prepare templates to report the results back to each individual school as well as to ISBE. Each report of survey results will contain the descriptive statistics for the school-based data as well as a chart illustrating how the school performed on 10 subscales in comparison to national norms. 
The summarized results from all of the survey instruments are returned to each school for its own planning purposes.   These reports include responses to the open-ended questions purged of any identifying characteristics.  Schools are also provided with a primer to help them understand and interpret the results for their school.  The reports are designed to encourage charter schools to engage in self-evaluation and to provide objective data for reports they submit to their authorizing agencies. 
      [top]
 
• Interviews.  Each site visit includes an in-depth interview with the school's director/principal.  Some site visits also include interviews/focus groups with teachers and students. 

     [top]
 
• Document review.  During the site visit field researchers collect important documentation on each charter school.  These documents might include copies of the schools' charters, teacher manuals, parent handbooks, descriptions of the schools' curricula and offerings, student work samples, portfolios, and other artifacts, as available. 

     [top of the page]

[IL Charter Schools Study Home]