[Mich. Charter School Evaluation Home]
Description and Plan for the Evaluation

Introduction
General Formative and Summative Evaluation Questions
Primary Evaluation Questions
Selected Assumptions about Charter Schools
General Phases of the Evaluation
Description of Data Collection with Surveys 

Introduction

The legislative and regulatory actions that created the initiative, policies, and procedures to guide the development and operation of charter schools (officially referred to as public school academies in Michigan) provide a clear intent that the initiative is to be monitored and evaluated in a quality manner. In addition, there is a promise to the people of Michigan and a responsibility to the professional practice of education that this innovative initiative be submitted to the highest quality of evaluation and standards. Evaluation and oversight of charter schools are a shared responsibility of school personnel, the authorizers, and the Michigan Department of Education. While the emphases and general audiences for the evaluations are different, it is in this collective manner that the fullest knowledge and understanding of this significant reform effort (the charter school initiative) will be realized. To ensure objectivity and credibility, it is important that a third party evaluator, without vested interests or direct involvement in creating, operating, or providing services to charter schools, be engaged to conduct an overall evaluation of the initiative.

[Top of Page]

General Formative and Summative Evaluation Questions

Since the charter school initiative is relatively new in Michigan and schools are in various stages of development, the evaluation plan incorporates elements of both formative and summative evaluation. The purpose of formative evaluation is to provide information to improve the charter schools by assessing their ongoing activities. It should be conducted continuously throughout the duration of a school's existence. Typical questions that would be asked within the context of formative evaluation are listed below: Within the framework of summative evaluation, the following questions would be appropriate to consider: Further, there is an implicit monitoring element within this plan. However, the purpose is not to monitor the operations or activities of individual schools. This is one of the designated responsibilities of the authorizers and should be an important element in their judging the quality of the educational programs in the schools they have authorized and in their decision to continue or withdraw charters. The monitoring process of the statewide evaluation in this plan relates to the overall initiative.

In recognition that the public school academies are new, operating with small administrative staffs, and of considerable interest to researchers and the general public, the comprehensive evaluation plan is designed to make maximum use of existing/statutorily required data and to be as unobtrusive as possible into the daily activities of the school. However, there is a substantial need to ensure that information derived from secondary sources is accurate and credible.

The evaluation plan is designed to address the primary purposes for which charter schools and the associated legislation and regulations were created. To determine whether or not these purposes have been fulfilled, evaluation questions answerable through this evaluation plan have been created. These questions are included in the following section.

[Top of Page]

Primary Evaluation Questions

  1. Legislation
    1. How have the changes since the original charter school legislation affected the operation of charter schools?
    2. Has the legislation been helpful in responding to individual charter school operations?
    3. Has the legislation provided parents and pupils with greater choices among public schools, both within and outside their existing school districts?
  2. Michigan Department of Education (MDE)
    1. Has MDE staff provided appropriate support and guidance for charter schools and authorizing agencies?
    2. Has MDE provided leadership and supervision for charter schools and authorizing agencies?
    3. Is MDE viewed as reasonable in its data requests?
  3. Authorizers
    1. Are the authorizing agencies providing appropriate oversight of charter schools?
    2. How expensive is the application process?
    3. How long does the application process take?
    4. How many visits have the authorizing agencies made to each charter school?
    5. Are the visits by the authorizing agencies to charter schools helpful?
    6. Is written feedback of the school visitation provided to MDE when requested?
    7. Has MDE been provided with an analysis of the line item expenditures from the authorizing agencies' 3 percent fee, when requested?
  4. Charter Schools - Broad Questions Regarding Effectiveness
    1. How effective are charter schools that have large amounts of start-up moneys, compared with charter schools that have insufficient money to begin operating?
    2. Will the lack of affordable buildings be a significant obstacle to the charter school movement?
    3. What are the affective and effective outcomes for students attending charter schools with a cultural focus?
    4. Have new professional opportunities for teachers been created in a new type of public school in which the school structure and educational programs have been innovatively designed and managed by teachers at the school site level?
    5. Has accountability for the educational performance of students been assumed at the school site level?
    6. To what extent is each charter school fulfilling its performance contract with its authorizing agency?
    7. What are the students' and parents' perceptions and levels of satisfaction with their charter school?
    8. Is the climate of the charter schools perceived by the students and parents as safe and conducive to learning?
    9. What, if any, impact occurred in the local school district in which the charter school is located?
    10. Why have local school districts, intermediate school districts, and community colleges used their statutory authority to authorize public school academies to such a limited extent?
    11. What other obstacles have charter schools faced in organizing and operating?
Additional evaluation questions added to the evaluation plan by The Evaluation Center are listed below.
  1. What, if any, innovative teaching methods or educational practices have been stimulated by the charter schools? To what extent are these transportable to other schools?
  2. To what extent and in what ways have parents become involved in the charter schools attended by their children?
To fully understand the context of the charter schools in the annual descriptive report, basic information on charter schools operating on or before the MDE first student count of the school year under investigation will be included. This includes information about the schools' missions, activities, various demographic figures as well as any additional information that the Department of Education deems appropriate.

[Top of Page]

Selected Assumptions about Charter Schools

In evaluating the charter schools for any given year, there are certain regulations and considerations that should be appropriately recognized in the data collection and interpretation. Among these are the following:
  1. Schools have existed for varying lengths of time and are in different stages of becoming fully developed and maximally efficient and effective.
  2. Each school is authorized for a particular mission and has identified goals and purposes unique to that mission.
  3. Teachers must be certified just as they are at other public schools, except as allowed by laws, rules, or regulations.
  4. Schools may not screen students, but they may limit the number of students they serve. If more students apply than can be enrolled, students are chosen through a random selection process.
  5. Neither the state board of education nor the state legislature mandated a core curriculum for Michigan. However, a Michigan core curriculum was recommended as a framework for all public schools to use.
  6. Charter schools are subject to all laws and regulations that apply to public schools generally, including The Revised School Code, 1995 PA21, Part 6A, and other related laws. These schools may not teach religion.
  7. Charter schools receive the state foundation grant on a per-pupil basis. This level of funding cannot exceed the amount received by the local school district. Charter schools cannot charge tuition. However, they can raise funds through legal foundations, receive grants, etc., as any public school can do.
[Top of Page]

General Phases of the Evaluation

    Planning phase. The planning and preparation of the study is to be completed by mid-December 1997. This includes the development, testing, and revision of data collection instruments, as well as finalizing the plan for data collection.

    Data collection phase. Most of the data collection will take place between December 15, 1997, and May 29, 1998.  There are two rounds of school visits that are planned during which the surveys will be administered and interviews conducted. The first round of school visits extends from December 15, 1997, to February 20, 1998. The second round of school visits is scheduled to take place between April 27 and May 29, 1998. The time between these two rounds of school visits will be devoted to meetings/interviews with representatives from (i) the Michigan Department of Education, (ii) authorizing agencies, (iii) management companies, (iv) intermediate school districts, and (v) traditional public schools.

    Data analysis and review of relevant documentation. Data analysis of the survey results will take place after each round of school visits (summarized results will be returned to each particular school for its own planning purposes). A review of relevant documentation and literature will continue throughout the duration of the project.

    Report writing phase.  Final report writing will take place during the summer and early autumn of 1998. The final report will be submitted in December 1998.

[Top of Page]

Description of Data Collection with Surveys

To ensure the collection of consistent information across all charter schools, we have developed the following schedule for data collection with questionnaires.
Instrument
Period for Administration and Targeted Informants
Description of Data Collection
Basic Information Form Ongoing activity with final copy due by no later than May 29, 1998.
       This form is to compiled by the school principal/administrator.		 This form includes a number of items for which responses will be available at various times during the year. We ask that the administrator of the school be responsible for completing the form and providing supporting or explanatory information as appropriate. By no later than May 27, the completed form should be returned to the external evaluator.
Parent/Guardian Survey 15-20 randomly selected parents sampled between December 15, 1997, and February 20, 1998, and an equal number of parents sampled between April 27 and May 29, 1998. Parents will be randomly selected by the external evaluator together with the school administration.  Specific instructions about how to select parents as how to administer the survey can be found in the document entitled "Instructions for Administering the Parent/Guardian Survey."  Half of the sample parents will receive the Parent/Guardian Survey during the first round of school visits.  The second half will receive this survey as well as the School Climate Survey during the second round of data collection.
       A cover letter is used that explains the purpose of the survey, and each parent receives an envelope in which to enclose the survey.
Student Survey Between December 15, 1997, and February 20, 1998.
       One representative classroom of students will be selected from each grade level (5-12) in each school. All of the students in these selected classes will be asked to complete a questionnaire.
      Due to limitations of time and the coinciding scheduling of MEAP testing, some grade levels were not sampled in a few schools.  Missing classes will be administered the student survey during the second round of data collection.		 The administration of the questionnaire takes between 20 and 40 minutes. The student surveys are administered by a member of the evaluation team. The purpose of the survey and the manner in which the results will be used are explained to the students before they begin completing the forms.  Students in grades 7-12 can complete the surveys on their own, after initial instructions. Because some students in grades 5-7 may have difficulty understanding all the concepts appearing in the surveys the persons administering the surveys, should progress item by item with the class and read each question/item aloud. Students should be assured that they may respond in an honest and straightforward manner without concern about retribution. Before accepting the surveys from the students, each completed survey is checked by the person administering the survey to make sure that all bubbles are completely filled in and no stray marks appear on the forms.
Teacher Survey Between December 15, 1997, and February 20, 1998.  All teachers and school personnel involved with instruction, including administrative and professional support personnel, will complete this questionnaire.
       New teachers or others who could not complete this survey during the first round of school visits will be asked to complete this survey during the second round of school visits.		 All teachers and school personnel involved with instruction, including administrative and professional support personnel, will complete this questionnaire. The respondents will be asked to complete the survey, enclose it in an envelope, and then return it to a designated person at the school. Teachers should not place their names on the survey although they will be asked to check off their name from a list so that we can trace and follow up missing respondents. Since the completed forms are to be collected, sealed, and mailed to the external evaluator by a designated person at each school, there should be ample assurance that the responses are anonymous. Specific instructions about the selection and administration of the teacher survey can be found in the document entitled "Instructions for Administering the Teacher Survey."
       A cover letter is used that explains the purpose of the survey, and each teacher receives an envelope in which to enclose the survey.
School Climate Survey Between April 27 and May 29, 1998 (i.e., the second round of school visits), this survey will be given to a sample of parents, students, and teachers at all participating schools. This is a commercially developed instrument by the National Association of Secondary School Principals. Dissemination and data analysis functions are provided by The Evaluation Center at Western Michigan University. The administration of this instrument will be coordinated by the external evaluators or by a Traveling Observer who will work as part of the evaluation team.
Each school is encouraged to add questions or items to the survey. These would follow on a separate sheet of paper at the end of the surveys. The summarized results from each survey will be returned to each school for its own planning purposes.

Beyond the surveys described above, qualitative data will be collected through interviews and/or focus groups with various stakeholders and through school visits made by the external evaluators or by the Traveling Observers. (Traveling Observers are field workers based in various parts of the state who will assist in the data collection.)

[Mich. Charter School Evaluation Home]   [Top of Page]

Comments and/or questions regarding this page may be addressed to Gary Miron at gary.miron@wmich.edu