Methodology Issues

Methods

The array of questions to be addressed in this study requires multiple approaches for collecting and verifying information and for capturing the various perceptions that may exist.  Our aim is to collect sufficient information to analyze all charter schools individually as well as among and between groups of schools with similar characteristics.  We think it will be important to at least describe the diversity (or similarity) that might exist among the charter schools and among the authorizers.  Decisons made regarding the nature and type of data collected will be made together with the Pennsylvania Department of Education to ensure that this is worthwhile and of interest to decision makers and other interested parties.

We envision that a considerable amount of the contractor's time will be spent in “educating” charter school operators and personnel to the use and value of evaluation.  This task has been placed in the category of capacity building, an element for which we have a strong commitment.  Not only will this be of value to the school in later years, it will also be of value as we gather data and information in collaboration with the schools.

We will utilize the following methods for collecting information.
 

º Surveys of  staff, students and parents	º Reviews of (student) work samples
º Interviews	º Diaries and logs (if available)
º Document reviews	º Portfolios (if available)
º Direct observation	º Focus group meetings
º Analysis of test scores and available    demographic and finanacial data

The procedures and data collection methods we plan to use in this project are clearly described in Tables 1,    2,  and  3,  and in the descriptive sections that follow.

[back to top]

Data Collection Strategies

From our experiences in evaluating programs and schools over many years and particularly those that are new, innovative, and of considerable public interest, such as charter schools, we find these programs/schools are bombarded with requests for information and to serve as subjects for a variety of studies.  In addition, they will undoubtedly be targeted by the public media for stories related to an array of topics.  While this attention may be complimentary and initially well received, it becomes a considerable drain on the resources of the relatively small staffs of charter schools.  Usually, those in administrative roles at charter schools are unaccustomed to these requests and do not have the support services to maintain a responsive position.  These conditions pose potential complications for this study; therefore, we will make every effort to use existing data that may be required for other reports.  However, we will need to seek additional information from students, teachers, administrators, parents, and the broader community.  We will focus on only those issues that are important and necessary for this study and will select those respondents who are considered to be knowledgeable about the issue(s) being addressed and who can contribute to the quality of the information/data that we collect.  With good planning and the development of survey instruments and protocols, we hope the process of obtaining information is viewed as time well spent by the informants and useful by stakeholders, especially the stakeholders directly connected to the local schools.

However, we expect to use the general strategies for collecting information as summarized in Table 1.   Table 2 shows a matrix of the evaluation questions and planned sources of data/information for each question.  In terms of timing, Table 3 shows a matrix of the preliminary plan for data collection by quarter of operating year.  These will be further refined in collaboration with representatives of the Pennsylvania Department of Education, charter schools, and local public school districts that authorize charters.  The evaluation team will engage representatives of these stakeholder groups in discussions for the purpose of confirming initial plans and for making adjustments where appropriate and necessary. 

As noted in the matrices, some of the data will be quantitative in nature and some will be qualitative.  However, it is important to note that information will be collected to answer questions of importance for this study.  In most cases, a combination of information and sources will be used to determine the findings of an investigation or to provide a basis for a response to a question.  We often use the term “triangulation” in our data/information-gathering efforts.  While this implies three indicators or sources, we often employ more than three sources to provide a more detailed and dependable explanation.  For example, we propose to determine the perception of school climate from the vantage point of students, teachers, administrators, and parents.  Likewise, we will consider evidence of academic achievement from test scores, appropriate student work samples that may be provided by teachers, and opinions of parents about their children.  In addition, we will ask teachers about other types of achievement that reflect the mission and goals of the school.  Input about the role and effectiveness of local school districts that granted their charter will be requested from charter school representatives, appropriate Pennsylvania Department of Education personnel, and others who might be identified by the authorizers themselves. 

[back to top]

Data Analyses and Reporting

Quantitative and qualitative data will be collected and analyzed according to professionally acceptable standards of practice.  Data will be encoded for analyses at The Evaluation Center at Western Michigan University.  We consider the data to be confidential, if not anonymous, and the property of the Pennsylvania Department of Education.  Data analyses will be summarized in tables with appropriate explanatory narratives.  Preliminary copies of formal reports will be submitted to the PDE contact for review.  The purpose of the optional review of the reports (formally or informally) by PDE-designated persons and any other personnel is to correct errors and omissions and to ensure readability by stake holders. 

In all cases, the Program Evaluation Standards will be followed in the conduct and operations of this study.  One role of the planned metaevaluation will focus on this important aspect of our planned study of charter schools.

The analyses of the data and information collected according to the specified areas above will be conducted in order to yield sufficient information to make judgments about individual charter schools, groups of charter schools, and the charter school initiative as a whole.   For example, we intend to conduct the analysis with the following comparisons in mind:

º Compare charter schools over time (of course, this will be limited to 2 years for 6 
   of the schools, although additional comparisons can be made with conversion schools).

º Compare groups of charter schools according to similar characteristics dependent 
   on mission, size, location, educational level (elementary, middle, secondary), 
   conversion or nonconversion, local or regional charter, etc.

º Compare each charter school with its host school district in terms of demographics, 
   test scores, revenues and expenditures, etc.

º Compare each charter school with a comparison school with similar background 
   characteristics including demographics, size, location, and educational level. 
   Comparisons would consider test scores, financial data regarding revenues and 
   expenditures, etc.

º Results from the Charter School Survey and School Climate Survey will be merged 
   with school-based data regarding the demographics, size, location, etc.  This will 
   allow comparison of degree of satisfaction and quality of school climate with school 
   characteristics.

It will also be of particular interest to trace developments in terms of background characteristics of students enrolled in the charter schools.  In some states, there is evidence that shifts in enrollments are taking place due to such factors as the increasing involvement of educational management organizations (EMOs), changes in legislation, new requirements by chartering agencies, etc.  Our analysis should be possible to determine how the various background characteristics can impact enrollments in the schools.

We have had experience in conducting such analyses in our charter school studies in Michigan and Connecticut and think that the results have provided substantial feedback that assisted the schools, the authorizing authorities, and the respective departments of education.  We expect that the data on indicators from the PSSA will be the key source of data from the charter schools, host districts, and control schools.

Our plans for evaluation have been designed to reflect the same standards of quality and credibility that have been practiced to this point.  We will work in a cooperative manner with the Pennsylvania Department of Education, authorizers, and charter schools to ensure that this effort provides positive contributions to the information base about charter schools and evaluation of innovative schools/agencies.

In recognition of the various stakeholder groups, decision makers, and interested parties, special efforts will be made to communicate the procedures, findings, conclusions, and recommendations in an understandable form.  To accomplish this goal, there may be multiple versions of certain parts of formal reports.  For example, we plan to submit a final (end-of-project) report, an expanded executive summary, and a highlights report.  All schools will receive the disaggregated results from the surveys adminstered at their own school as well as averages for all charter schools.

[back to top]
 

Metaevaluation

We believe that the evaluation of the charter schools and the initiative in Pennsylvania is of such importance and potential impact that a metaevaluation is not only appropriate but should be mandatory; therefore, we have included this activity as a part of our proposal.  A description of metaevaluation, as taken from The Program Evaluation Standards (Joint Committee on Standards for Educational Evaluation, 1994), is found below.

Metaevaluation is evaluation of an evaluation. The program evaluation should be evaluated by those who design and carry it through. And, inasmuch as program evaluations often are subject to outside scrutiny, criticism, and legal action by program supporters, detractors, and metaevaluators, separate concurrent review of the program evaluation process by external metaevaluators can help program evaluators avoid critical mistakes. Clients can also conduct metaevaluations. Documentation of the effective/ineffective application of program evaluation procedures facilitates the proper interpretation of data. Regular employment of metaevaluation should enhance the credibility of particular program evaluations and the overall evaluation profession (p. 185). 

[back to top]