Autonomy in Exchange for Accountability:
An Initial Study of Pennsylvania
Charter Schools

Executive Summary

Gary Miron and Christopher Nelson

The Evaluation Center
Western Michigan University
Kalamazoo, MI 49008-5237
tel. 616 387-5895
<www.wmich.edu/evalctr/>

October 2000

«[PA Charter Schools Evaluation Reports]

This is the final report of the 17-month initial study of Pennsylvania charter schools. The Western Michigan University Evaluation Center has conducted this evaluation pursuant to a contract with the Pennsylvania Department of Education (PDE). The initial study is a key research component of the Commonwealth's overall accountability plan for charter schools. Section 1728-A of Pennsylvania's charter school law (hereafter known as Act 22) requires an evaluation of the charter school program after 5 years. While the 5-year report is to be largely summative in nature (providing recommendations on the advisability of continuing or amending the program), the initial study is designed to be largely formative and to provide feedback to schools and policymakers regarding changes that can be made to help these schools function more effectively and achieve their anticipated goals. At the same time, this initial study is an important component of the Commonwealth's overall accountability plan for charter schools and it provides a foundation for the 5-year legislatively-mandated evaluation.

Charter schools are a new form of public schooling intended to provide alternative and diverse educational programs, with the goal of improving academic achievement. Charter schools operate under a contractual arrangement with a chartering entity, in the case of Pennsylvania this includes the local school districts. The chartering contract frees schools from most of the rules and regulations that apply to traditional public school systems in exchange for increased accountability-ultimately, high student academic achievement. Charter schools can be formed by a variety of individuals or groups, including educators, parents, community members, for-profit and nonprofit organizations, and institutions of higher education. A charter is signed by its founding members and a chartering agency and details what the school expects to accomplish with respect to student achievement and other outcomes. Unlike traditional public schools, charter schools may be closed by their sponsoring entity if they fail to meet the standards set forth in the charter.

The autonomy-accountability bargain creates for charter schools an "opportunity space" within which they operate. Two general questions follow from this:

How are charter schools using their autonomy?
Are these uses of charter school autonomy leading to the student outcomes for which charters are held accountable?

Indeed, the central evaluation question stated in the request for proposal (RFP) for this study is, "Does increased flexibility in exchange for increased accountability result in improved pupil results?" Moreover, the RFP asked a number of more specific questions about uses of charter school autonomy and their impact on various educational outcomes.

What effect does budget have on student results, nonacademic services, and school facilities?
Are opportunities offered to charter school teachers, parents, and students to influence classroom and school policy significantly different from those offered at traditional public schools?
Are the opportunities (i.e., professional growth, salaries, benefits, employee rights) for teachers and other employees significantly different at a charter school than at a traditional public school?
What is the impact of charter schools as related to district reform efforts?
Is there evidence that, over the term of the charter, student learning has significantly improved?
What are promising practices in charter schools that could be included in district systemic reform?

Methods and Limitations of the Study

The questions addressed in this study required multiple approaches for collecting and verifying information. Thus, the study's methodology represents a blend of quantitative and qualitative methods. Our aim was to collect enough information to analyze all charter schools individually as well as among and between groups of schools with similar characteristics. Since our mandate was to evaluate the charter school reform rather than individual schools, this final report focuses on generalizations across the charter schools and does not make judgments about individual schools. Decisions regarding the nature and type of data to be collected were made with the Pennsylvania Department of Education to ensure that they are worthwhile and of interest to decision makers and other interested parties. All decisions regarding the study's design and methodology were guided by The Program Evaluation Standards developed by the Joint Committee on Standards for Educational Evaluation.

The following data collection and analysis methods were used:

Surveys of staff, students, and parents (charter schools surveys developed by The Evaluation Center and nationally-normed school climate surveys)
Reviews of (student) work samples (when available)
Interviews
Diaries and logs (if available)
Document review
Portfolios (if available)
Direct observation
Focus group meetings
Analysis of test scores and available demographic and financial data

Most of the analyses presented in this report are based on the 31 charter schools operating during the 1998/99 academic year (1 of which closed at the end of the 1998/99 year). Where possible, the study includes secondary data on the additional 17 schools that opened during the 1999/2000 academic year. Surveys were administered and interviews were conducted during both the 1998/99 and 1999/2000 academic years. The evaluation team sampled all teachers and key administrators and a random selected 3 classes of students and between 25-35 families at each school. The evaluation team worked diligently to ensure high response rates across all of the schools. Student response rates for each year were approximately 90 percent, while teacher response rates largely ranged from 73 to 83 percent. Parent response rates were considerably lower (approximately 50 percent).

The study's ability to provide evaluative conclusions was limited by the small number of charter schools and the relative newness of the charter school movement in Pennsylvania. Fortunately, both problems will be self-correcting as the movement continues to grow and mature.

Patterns of Growth in Pennsylvania Charter Schools

One of the key evaluation questions is whether the charter school law is providing students, parents, and teachers with new alternatives within the public school system. Thus, the report began by summarizing growth trends in Pennsylvania charter schools.

As of the 2000/01 academic year, there will be 66 charter schools in operation in Pennsylvania. A 67^th school was closed after its first year of operation. Taken as a whole, these schools will enroll more than 20,000 students, or just over 1 percent of all public school students in Pennsylvania.
Throughout the life of the charter school law, there has been relatively steady growth both in the number of charter schools and charter school students. Most of these schools are smaller in size (i.e., average enrollment is approximately 265 students) than the typical Pennsylvania public schools but larger than the typical charter school nationwide. There is, moreover, evidence of a trend toward larger charter schools in Pennsylvania.
Charter schools are concentrated in certain parts of the Commonwealth, particularly Philadelphia. Indeed, while Philadelphia enrolls approximately 11 percent of the public school students in the state, it has 51 percent of the charter schools and 69 percent of the charter school students.
Charter schools appear to target students of a reasonably wide variety of grade levels. There is, however, a tendency for charter schools to seek to serve students in the lower reaches of the elementary, middle, and high school grades. Similarly, charter schools' mission statements indicate that they intend to serve a wide variety of educational interests and goals. However, there is evidence that a significant proportion seek to serve at-risk students.

Charter School Start-Up

The range of charter school alternatives depends, in the first instance, on founders' commitment to and skill in gaining charters and opening schools. For this reason, the report provides an analysis of the characteristics of the founding coalitions that seek charters and start up charter schools.

Charter schools appear to be born of dissatisfaction with noncharter public schools, as evidenced by low PSSA scores. Lower performing districts, in turn, tend to have higher concentrations of poor and nonwhite students.
The founding coalitions behind charter schools often include public school teachers and administrators, academics, and members of the business community. There is little evidence thus far that parents have played a significant role in founding charter schools (though they do become active in the operation of charter schools).
Founding coalitions' goals have included providing a choice for poor children, creating a venue in which to operationalize ideas and practices hindered by district practices, promoting change in noncharter schools, and inculcating a particular ethnic or cultural perspective. Private conversion charter schools, moreover, are often founded with an eye toward scaling up practices already employed in private schools and attracting a broader base of students.
There is evidence that in order to be successful, founding coalitions often need to muster considerable political resources. In some cases this has involved tapping into support for charter schools among influential community leaders and the more general dissatisfaction with public school bureaucracies. In some instances, however, it appears that host districts have seen advantages in sponsoring charter schools, making such political tactics unnecessary.
Some 78 percent of the charter schools in operation as of the 1998/99 academic year are closely affiliated with some sort of nongovernmental organization, including community development groups, ethnic/cultural groups, and other nongovernmental organizations. Many schools are also built upon the foundation of a preexisting school.

Charter School Finance

One of the most important inputs into charter schools is money. Therefore, the report examined charter school revenues and also provided an analysis of charter schools' expenditure patterns and some simple indicators of financial viability.

Analysis of charter schools' financial reports indicates that the median charter school received approximately 81 percent of its total revenues from LEAs during the 1998/99 school year. There is, however, a large amount of variation among charter schools, with some charter schools receiving as little as 17 percent and some as much as 89 percent of their revenues from other LEAs. Much of this variation, of course, is explained by the fact that some schools spend much more per pupil than others and that the size of the LEA varies with district expenditures. Beyond that, schools vary in their reliance on non-LEA revenue sources.
Next to LEA transfers, the largest revenue source for charter schools is the federal government, mostly through Title I monies and special charter school grants. While there is considerable school-by-school variation, the median charter school received approximately 7 percent of its total revenues from the federal government. The remainder of charter school revenues came from state grants (e.g., start-up grants) and "local" sources, including earnings on investments, charitable donations, and revenues from student activities (e.g., candy sales, car washes, and so on). In addition, a few schools relied on proceeds from extended term financing during the 1998/99 school year.
Analysis of charter schools' financial data indicates that the median charter school spent approximately the same amount per pupil as its host district during 1998/99. Of that total amount, charter schools typically spent a smaller percentage (59 percent) on instructional items than their host districts (66 percent) and a larger percentage on support services (which include renting and maintaining facilities) and on other noninstructional items.
Analysis of financial reports indicates that charter schools appear to be relatively conservative in budgeting, taking in more than expected on the revenue side and spending less than expected on the expenditure side of the ledger. Moreover, there is some evidence that schools benefit from experience, as second year schools had slightly lower expenditure variances than first year schools (there was no discernible relationship on the revenue side).
Analysis of end-of-year balances indicates that most schools ran surpluses for the 1998/99 school year. Seven charter schools (23 percent), however, showed negative balances (deficits), the largest of which constituted 10.7 percent of total expenditures.

Student and Parent Characteristics

Another important set of inputs to charter schools is the people who attend, send their children to, and work/volunteer at the schools.

On the whole, charter schools enroll roughly equal proportions of males and females.
The proportion of low income students enrolled in charter schools is roughly comparable to host districts; in both groups approximately two-thirds of students are eligible for free or reduced-price lunch.
Charter schools, however, enroll a significantly higher proportion of nonwhite students (80 percent) than their host districts (57 percent) and charter schools nationwide (52 percent).
Most charter school students (80 percent) previously attended a public school, while 17 percent attended a private school and 3 percent other schools. The proportion of students coming from private schools roughly corresponds to the overall proportion of private school students in the Commonwealth (15 percent). Nevertheless, there is significant variation among charter schools.
Evidence from surveys suggests that parents choose charter schools mainly because of the perceived quality of instruction, the school's academic reputation, dissatisfaction with their child's previous school, and because they agree with the school's educational philosophy. Indeed, some 88 percent of parents sampled indicated that they are aware of their school's mission.
The most common reason students cited for attending their charter school was that their parents believe the school is good for them. Beyond that, students cite school safety, teacher quality, small classes, and dissatisfaction with their previous schools as the most important factors driving their choice. Approximately 60 percent of students report that they are aware of their school's mission.
Only one-quarter of surveyed parents indicated that they volunteer more than 3 hours per month at their charter school. This is in spite of the fact that approximately half report that their school requires such involvement.

Teacher Characteristics

Teachers and staff represent another important human capital input for charter schools.

The majority of teachers and staff are female (71 percent), while the gender split for administrators and directors is close to equal. This generally matches the gender distribution in noncharter public schools.
Charter school teachers are generally younger than their counterparts in noncharter public schools, with approximately 50 percent under the age of 30 compared with 11 percent in other schools.
Charter school teachers are also more likely to be nonwhite than their counterparts in other public schools. Approximately 40 percent of teachers are nonwhite, with African Americans comprising the largest group, followed by Hispanics, Asian/Pacific Islanders, and Native Americans.
The proportion of charter school staff who report that they are classroom teachers (52 percent) is similar to the proportion in noncharter public schools in Pennsylvania (49 percent) and public schools nationwide (52 percent).
Some 76 percent of teachers surveyed in 1999/2000 reported that they are currently certified to teach in Pennsylvania. This represents a decrease from 82 percent in the previous year. On the other hand, the percentage of teachers certified to teach in other states rose from 2 percent to 4 percent over the same period, as did the percentage of teachers working to obtain certification (12 to 17 percent).
The vast majority (75 percent) of teachers with university degrees had attained a BA as their highest level of education. However, a third of the teachers stated they are working toward another degree.
The average experience level of the teachers surveyed was just under five years.
The most commonly cited reason for joining the faculty of a charter school was an interest in being involved in a school reform effort, followed by the opportunity to work with like-minded educators. Other popular responses included small class sizes, safety, and the school's academic reputation.
Some 97 percent of teachers indicated that they are familiar with their school's mission. Of these teachers, 72 percent believe that their school's mission is being followed "well" or "very well."

Professional Opportunities for Teachers

One of the goals of Act 22 is to provide enhanced autonomy and professional development opportunities for teachers.

Many teachers have come to charter schools seeking autonomy in creating and implementing curriculum.
There is evidence at a number of charter schools that there was a conscious effort to involve teachers in developing curricula.
Teachers indicated that they had autonomy in curriculum decisions and freedom to utilize creative approaches. Indeed, many teachers report that they have considerable flexibility and opportunities for creativity in their day-to-day activities.
There is clear evidence in slightly more than half of the charter schools that the teachers in the schools work collaboratively.
Evidence from teacher surveys indicates that approximately one-quarter to one-third of teachers are satisfied with their school's physical facilities, while one-third to one-half were satisfied with resources available for instruction and other educational functions.
The average charter school paid teachers an annual salary of approximately $30,000, compared with the state average of approximately $48,000 and an average of $40,000 for schools with similar levels of teacher education and experience and similar per pupil expenditures. In spite of this, some 30 percent of teachers report that they are satisfied or very satisfied with their salaries.
There was a measurable difference between initial teacher expectations and current experience on many topics, the largest of which were the effectiveness of leadership and administration, communication between parents/guardians and the school, availability of support services to students, parents' ability to influence the direction and activity of the school, and the extent to which students receive sufficient individual attention. There was also a large gap between expectations and experience in teacher empowerment and the degree to which they are able to influence the steering and direction of the school.
Other areas of concern for teachers included class size, emphasis on academics, and parental involvement.
Despite reported problems, many teachers reported that they are satisfied with their teaching environment, and about 75 percent planned to returned to the school the following year.
Charter schools devote considerably more time to teacher professional development activities than noncharter schools, with the average charter school allotting 7 professional days and noncharter schools 5 days. A strong emphasis on graduate study was frequently reported, with some programs requiring it.

Innovations in Governance, Curriculum, and Instruction

One of the stated goals of Act 22 and other charter school statutes is to encourage the development and diffusion of innovative practices. In this sense, charter schools are to be public education's "R&D." The report examines innovations in leadership, organization, discipline, curriculum, and instruction.

Teachers and parents generally expressed approval of their school's leadership, with 67 percent of parents and 53 percent of teachers indicating that the statement "This school has good administrative leadership" was true.
Approximately 70 percent of teachers agreed that their school's leaders set high standards, communicate them effectively to others, and set a good example by working hard themselves.
Parents, teachers, and students, moreover, often have enhanced opportunities to participate in school decision making. These include (a) involvement in the strategic planning process, (b) attendance at planning meetings, (c) attendance at school board meetings and other avenues of access to board members, (d) input via school surveys and interviews, and (e) formal appeals and grievance processes.
Approximately half of teachers (54 percent) said that they are involved in decision making at their school. A similar proportion of students report that administrators listen to their ideas about the school. Approximately 83 percent of parents, moreover, said that it was true or partly true that "I am able to influence the direction and activities in the school."
Many teachers indicated that their workloads did not leave them enough time to remain involved in school decision making. Parents cited work and family obligations as barriers to their participation.
Notable organizational and disciplinary practices include extended hours and Saturday sessions, inclusion of both middle and high school grades in order to address common problems in making the transition from middle to high school, and preventive disciplinary measures that focus on behavior modification techniques.
Curricular innovations in charter schools often come in the form of a focus on a particular ethnic or cultural perspective. Other schools focus on social and vocational skills, with some offering individualized instructional plans for all students. In some cases, charter schools employ packaged programs such as Success for All reading, Everyday Mathematics, or Discovery Works.
Evidence of charter schools' use of technology was scant and allowed for no firm conclusions. However, the weighted average number of students to computers in charter schools is 4.5, compared with 7.5 for their host districts. At least one school maintains student portfolios on-line, and at least one school offers on-line courses that students can take from home.

Special Education

Special education presents a challenge for all public schools, especially charter schools. Thus, the report provides a preliminary assessment of special education in charter schools.

There is an important conflict between the spirit of charter schools laws, which seek to deregulate charter schools, and the preemption of special education by federal law. To its credit, since enactment of Act 22, the Commonwealth has made considerable headway in clarifying charter school obligations and providing guidance to charter schools in proposing new regulations in this area, and also in providing support and technical assistance to help charter schools provide special education services that are in compliance with IDEA.
The average proportion of students with disabilities in the charter schools was 10.5 percent in 1999-00, which is slightly lower than the statewide average of 12.5 percent. Among the schools that opened during the first two years of the reform, the proportion of students with disabilities was higher (12.1 percent) than for the 17 schools that opened for the 1999/2000 school year (7.8 percent). However, there was great variation among the schools, with some schools reporting as many as 42 percent special education students and some reporting no such students.

Student Achievement in Charter Schools

The charter school "bargain"-autonomy in exchange for accountability-requires that charter schools demonstrate improved student outcomes. Thus, the report provides a preliminary assessment of student achievement in Pennsylvania's charter schools. A number of caveats are discussed in the report regarding our analyses of student achievement. Nonetheless, the evaluation presents some suggestive findings that might be useful in program improvement.

The data on student achievement in charter schools precludes conclusive statements about charter schools' impacts on student learning due to a number of limitations. Among the limitations the following are most noteworthy: (i) the charter school initiative is still quite young, leaving charter schools with little time to demonstrate their ability to improve student achievement; (ii) data was not available for many of the charter schools; (iii) no data exists for such important characteristics as students' precharter school achievement rates, which seriously diminishes the evaluation's ability to provide valid assessments of charter school impact. For these reasons generalizations to the broader movement are tenuous.
Charter schools as a group produced Pennsylvania System of School Assessment (PSSA) scores that were considerably lower than all noncharter public schools in the Commonwealth. Such comparisons, however, probably measure differences in the types of students who choose to attend charter schools more than any impact the charters have on their students.
Four schools for which time series data are available posted gains of more than 100 points (the scale ranges from 1000 to 1600). In doing so, these schools as a group outgained their host districts as a group by some 86 points over the same period. Single cohort, pre/post commercial test data (e.g., ITBS, SAT-9) from a nonrepresentative sample of charter schools also show that a substantial number of charter schools posted gains in student achievement.
A cross-sectional comparison of each charter school with its host district(s) found that charter schools as a group were outperformed by approximately 50 points on the PSSA. However, there is evidence that host districts provide flawed comparisons to charter schools.
Likewise, a cross-sectional comparison of each charter school with demographically similar noncharter public schools found that charter schools as a group were outperformed by their approximately 50 points on the PSSA. However, data limitations restricted the analysis to comparisons on the basis of family income (as indicated by the proportion of students qualifying for free or reduced lunches) only.

Other Indicators of School Quality

There is legitimate debate about precisely what types of student outcomes charter schools should be held accountable for. While most stakeholders seem to agree that student achievement is an important (if not the only) goal of charter schools, others argue that schools should also be judged on their ability to satisfy their customers.

Survey evidence suggests that as a group charter school students report that their level of academic performance has improved since they moved to a charter school. Teachers are perhaps a little less sanguine, with most initially expecting that student achievement would have improved more than it has. Still, more than half the teachers believe that student achievement is on the rise at their school.
Approximately two-thirds of parents and teachers reported that charter schools are serving needs not well served by other schools.
A nonrandom sample of charter schools indicates that the average school has a waiting list of 125 students, or 45 percent of total current enrollment.
A nonrandom sample of student rosters indicates that the average charter school lost 38 percent of its students from Spring 1999 to Spring 2000. Moreover, data from the Philadelphia school district indicates that some 1,800 students left Philadelphia charter schools to return to district schools.
In spite of the significant enrollment instability in some charter schools, more than 90 percent of parents surveyed report that the quality of instruction in their charter school is high and that their child receives adequate attention.
Approximately half of the students surveyed said they would recommend their charter school to a friend.
Charter schools had an estimated attendance rate of 90 percent for both 1998/99 and 1999/2000.
Nearly three-fourths of parents and teachers reported that their charter schools had high expectations for students.
Nearly three-fourths of students said that their teachers encourage them to think about their future.
Approximately one-third of students said that other students at their charter school were more interested in learning than students at their previous school.
Fewer than one-half of respondents report being fully satisfied with school facilities.

Issues for Further Consideration

This formative evaluation seeks to identify program strengths and weaknesses in order to facilitate improvements. It also provides a foundation for a 5-year summative evaluation to be released in 2002. Thus, we conclude by identifying a number of important policy issues and research questions raised by this report.

Issues to be considered in future evaluation and research activities include those listed below:

How effective is the charter approval process as a quality control mechanism?
Do some district approval and oversight processes lead to stronger charter schools than others?
Do charter schools approved on appeal face a hostile environment? If so, does this affect school quality and student outcomes?
What explains variations in charter schools' expenditure patterns?
Do charter schools offer more educational value for the money than noncharter public schools?
Will observed trends in charter school affiliation with nongovernmental organizations continue?
Will the role of private management companies continue to increase over time? And what effects might this have on the autonomy and effectiveness of the schools?
What explains variations in student achievement across charter schools?
Why do students and parents choose to leave charter schools?

Policy issues that should be considered include these:

What are the implications of charter schools' reliance on nongovernmental organizations for school quality and the future growth of the movement?
How should charter schools balance the provision of services for special needs students with the movement's goal of providing head-to-head competition with noncharter public schools?
How should charter schools, local authorizing agencies, and other stakeholders balance market versus other forms of accountability in decisions to grant and renew charters?