[Mich.
Charter School Evaluation Home]
Metaevaluation
Report
of the Evaluation of The Michigan
Public School Academy Initiative
by
Kenneth H. McKinley, Ph.D.
May 1999
Contents
[top of the page]
Michigan Public School Academy
Initiative Evaluation
Metaevaluation
May 1999
This is the metaevaluation report
of the program evaluation of the Michigan Public School Academy (PSA) Initiative
by The Evaluation Center, Western Michigan University (WMU) (hereinafter
referred to as "The Center"). This metaevaluation is conducted and reported
in a summative sense in order to provide feedback to The Center, the Michigan
Department of Education (MDE), and the Michigan PSAs regarding the initiation,
development, operation, progress, and evaluation of the charter school
program in Michigan.
[top of the page]
Methodology
The following activities were
undertaken by this author to conduct this metaevaluation and prepare the
following report:
Studied and became familiar
with the list of evaluation questions posed by the Michigan Department
of Education (MDE) in its public Requests for Proposal (RFP) to evaluate
the Michigan PSA InitiativeReviewed the WMU "Proposal to Evaluate the Michigan
PSA Initiative"Reviewed "A School Self-Evaluation Kit for the Michigan
Public School Academies" prepared by The Center for utilization in the
evaluation of the Michigan PSA Initiative (1997-98)Reviewed sample field
notes prepared by The Center evaluation team and traveling observers (TOs)Reviewed
interview schedules and frameworks prepared and utilized by The Center
evaluation team and resource personsVisited The Center’s World Wide Web
site at http://www.wmich.edu/evalctr/charter/micharter.html.
Read and analyzed the 159-page final report (with its accompanying 23-page
executive summary) entitled: "Evaluation of the Michigan Public School
Academy Initiative" by Horn and Miron, Western Michigan University,
January, 1999Designed and developed the matrix checklist (see Appendix
I) for applying The Program Evaluation Standards (The Joint Committee on
Standards for Educational Evaluation, 1994) to the work of the Center in
conducting the Michigan Public School Academy evaluation and around which
this report is fashioned.
[top of the page]
Standards,
Findings, and Recommendations
The remainder of this metaevaluation
report is organized to give the reader an overview of the findings and
recommendations as they relate to The Program Evaluation Standards from
the Joint Committee on Standards for Educational Evaluation (1994). Each
JCSEE standard will be reported as having been addressed (A), partially
addressed (PA), not addressed (NA), or not applicable (NAP) as it relates
to The Center report of the Evaluation of the Michigan PSA Initiative (see
Appendix I). Following the listing of each standard, findings and recommendations
of the metaevaluator relative to The Center report will be given. Recommendations
for change, improvement, and/or clarification will be shown in bold and
colored print.
[top of the page]
Utility
Standards
U1 - Stakeholder
Identification: (PA)
It is well-understood who the
primary stakeholders are and will continue to be in the current evaluation
effort (e.g., the Michigan Department of Education (MDE), the Michigan
legislature, the Public School Academies (PSAs) {including their students,
teachers, administrators, and parents}, the host public school districts,
the authorizing agencies, etc.). It is recommended
that the authors include a clear definition of the primary and secondary
(maybe even tertiary) stakeholders to this Initiative and this initial
evaluation of its progress. Perhaps the groups identified in Table 1.1,
page 11 of the final report could be cited as the primary stakeholders.
Some secondary or tertiary stakeholders which come to mind might include
state departments of education and legislators in other states, educational
policy makers in Washington, DC, and indeed the entire nation ("the American
people" as the national politicians like to call us) as it seeks ways to
reform and improve the K-12 education system in our country.
[top of the page]
U2 - Evaluator
Credibility: (A)
The Evaluation Center at Western
Michigan University is an internationally known program evaluation contractor.
It offers high-quality program evaluation services to a wide and diverse
variety of public and private sector clients throughout the world. In addition
to its resident research and evaluation staff, The Center is able to draw
on a rich network of highly qualified experts in the field of educational
program evaluation. Thus, the choice of The Center to conduct the evaluation
of the Michigan PSA Initiative in the non-metropolitan region of the state
established evaluator credibility.
[top of the page]
U3 - Information
Scope and Selection: (A)
A well-rounded scope and sequence
of information regarding the Michigan PSA Initiative was chosen by The
Center for inclusion in this time- and resource-limited program evaluation.
Of course, this process was driven in the initial, conceptual stages of
the design and development of the evaluation by the Michigan Department
of Education (MDE) Request for Proposals (RFP). These primary evaluation
questions may be found on pages 4 and 5 of the final report and include
the following categories:
A. Legislation
B. Michigan Department of
Education (MDE)
C. Authorizers
D. Charter Schools - Broad
Questions Regarding Effectiveness
Additional framing questions,
both in the context of formative and summative evaluation (pp. 3, 4), and
additional evaluation questions (p. 5) were posed by The Center evaluation
team to give the study direction and a sense of holistic scope and sequence.
[top of the page]
U4 - Values
Identification: (A)
The perspectives, procedures,
and rationale used by The Center to conduct the current evaluation was
cited early and often in the final report. Reference is made to the Foreword
of the final report wherein the authors laid out not only the principles
and philosophy that guide all Center evaluation studies, but also the specific
criteria and guidelines that were applied in The Center evaluation of the
Michigan PSA Initiative. The study methodology (procedures) were clearly
explicated in chapter I of the final report, and balanced reporting of
the findings, conclusions, and recommendations for future action and research
regarding this sometimes emotionally-charged public education reform movement
(charter schools) were noted throughout the final report.
[top of the page]
U5
- Report Clarity: (PA)
In support of and consistent
with the observation made in the previous statement on Values Clarification,
the authors were uniform in providing the readers and users of the final
report with a contextual framework for the conduct of the study including:
-
characteristics and principles
of the evaluation effort
-
goals and objectives
-
assumptions regarding charter
schools
-
typical formative and summative
evaluation questions which would be applicable in this kind of an educational
program evaluation
-
primary evaluation questions
as specified in the project RFP
-
additional supporting questions
posed by The Center
-
design and methods of the evaluation
-
data collection strategies and
techniques
-
limitations of the study
Following are a number of findings,
followed by selected recommendations (shown in bold print) for improvement
of the clarity of the final report:
The authors of the Western
Michigan University PSA evaluation (1999) did a masterful job of reporting,
in a balanced way, and in the time and with the resources available, a
relatively clear, accurate, and objective picture of the development of
the Michigan charter school movement in the early stages of its infant
history.
-
Although the RFP did not call
for a review of the development of educational management organizations
(EMOs), an insightful discussion of them, at least as they apply to the
implementation of charter schools in Michigan, can be found on pages 67-71
and in the summary on pages 98-99.
A number of terms, phrases,
or word(s) were used interchangeably throughout the report to denote the
charter schools in Michigan. Examples include: public school academies,
charter schools, schools, "these" schools, "new" schools, and "most schools
at this level" (p. 72). It is recommended that
a single term or acronym (e.g., PSA) be used consistently throughout the
report.
-
A number of terms were used
throughout the report to inform the discussion which were not familiar
to third-party readers (i.e., "non-Michiganders"). A definition of terms
section in the methodology chapter is recommended. A
proposed "starter" list of words/terms might include:
-
Authorizing
agencies: Although they are discussed in detail in Chapter 5, they
start showing up in the narrative long before Section 5.3.
-
The
Leona Group (and any other management entities that appear to be making
their presence felt in the Michigan charter school movement)
-
State
foundation grant: Many states, with widely varying K-12 school finance
formulas, are grappling with fair and equitable finance formulas for the
support of charter schools. A more definitive and clear description of
the components of the Michigan foundation grant and how it is "passed through"
to the charter schools would be appreciated. Does it, for example, include
state appropriations only or a combination of state and local/county (ad
valorem, personal property, motor vehicle, etc.) taxes?
-
Start-up
Money: This is apparently a topic of high interest in Michigan. What
constitutes "large amounts of start-up moneys" or adequate start-up funds?
Does it vary from charter school to charter school, by region of the state,
or should it be defined on a head-count basis, similar to average daily
attendance (ADA) or average daily membership (ADM) indices used in state
public school finance formulas throughout the U.S.?
-
"Raising
Factor": Appendices A and B
-
Annual
descriptive report (page 5)
-
A large
number and variety of acronyms (EMOs, ISDs, MEAP, MDE, SVSU, etc.)
most likely have common meaning to a Michigan consumer of this evaluation
report, but not a non-resident. A recommended convention for their use
in a report such as this would be to define them once in the recommended
"definition of terms" section so that the reader could refer to their definition
as needed, or if used repeatedly "spell out" the term (e.g., intermediate
school district [ISD]) when first used in a chapter, then utilize the acronym,
as needed and appropriate, in the remainder of the chapter.
-
This
is a repeat of the comments under Standard U1 - Stakeholder Identification,
but the term "stakeholders" is used in a rather all-encompassing fashion
throughout the report. Perhaps the individuals and groups cited in Table
1.2, page 11 of the report could serve as the definition of the core group
of stakeholders relative to the Michigan charter schools initiative. Others
could be added, as appropriate.
-
As noted in the discussion on
Standard
U4-Information Scope and Selection, the focus for the majority of this
report was the formal evaluation questions posed in the Michigan Department
of Education (MDE) Request for Proposals (RFP). Although all RFP questions
(N=24) were addressed in the final report, there was not a standard format
for this information reporting (i.e., tracking the findings). Some RFP
question responses were clearly responded to with supporting narrative
paragraphs and data in tables and figures; some were restated verbatim
(from the RFP as repeated in Chapter 1), some were not; some answers were
embedded in the text as parts of a major chapter section, while others
constituted the section heading itself. Some responses to the RFP questions
were never clearly restated and then answered directly, but were rather
related to as part of a discussion of other topics within other contexts.
Finally, it was observed that some RFP responses were "rolled" into one
response under different but related phrasing (e.g., the questions related
to the guidance, leadership, and/or direction of the MDE were responded
to under the heading of the "role of the MDE"). Thus,
it is recommended for report clarity that when the RFP questions are first
posed, a sort of "readers guide" to the location of the responses to them
be provided (See Appendix II).
-
Chapter Five contains a discussion
of the findings regarding the current legislation on charter school initiation
and implementation in Michigan. Perhaps it would
be instructive to include a brief overview of the highlights of that current
legislation prior to the report of the interviews in Chapter Five. Such
issues as multiple site charters, authorizing agency accountability, and/or
policy for "shutting down" unsuccessful charter schools could be stressed
so that the reader would have a better understanding of the contextual
framework and historical antecedents leading up to the beginning of the
charter school movement in Michigan.
-
There are a number of minor
typographical errors in the report. Although none of them significantly
detract from the rich data and information baseline that the report, in
total, provides, some of the more noticeable typos include:
-
page 7 (top of the page): "
. . . final report writing took place during the autumn of 1998, with the
final report submitted in January 1998 (actually
1999).
-
page 16 - Figure 2.2 and page
97, third paragraph: " . . . 51 percent of the students were minorities
as compared with the total state enrollments in K-12 schools, which include
approximately 33 percent minorities." (According
to the data in Figure 2.2, it should be 23 percent: 100% minus 77% [white].)
-
page 28 - Middle paragraph:
"Section 5.5, which deals with management companies, explains this more
clearly . . . " (Actually, it is Section 5.4,
the last section in Chapter Five.)
-
page 40 - Footnote to Table
3.1: "Principle" should be "principal."
-
Table clarifications: Several
table data display conventions, if followed, would improve the ability
of the reader to make quick and concise interpretations of the data contained
therein. These include:
-
Display
dollar-denominated data in columns with commas properly placed and vertically-aligned
(Table 2.4, p.37; Appendices L-M, pp. 135-138)
-
Align
decimal points vertically in all appropriate tables and include decimals
to the nearest tenth or hundredth, as appropriate
-
Make
all item response N’s (total and missing) agree with total survey N (in
most cases they vary by + 1; however, in one instance, Appendix
I, Subscale 1: Teacher-Student Relationships, item 4, p. 129, it varies
by an N of 10)
-
Round
all response N’s to nearest whole (see Appendix F, item 14, p. 120)
-
Include
"total" and "missing" N data in the same box at the right-hand margin of
the page. When they are separated (ex., Appendix E), it can lead to confusion
at first glance.
-
Use
parentheses ( ) throughout to indicate negative values.
-
In
Appendix M, it is interesting to note that the average PSA teacher salary
at the Western MI Academy for Hospitality Sciences went from nearly $71K
in 1995/96 to $16.7K in 1996/97. Is this a typographical error or actual
fact? If the latter, it is a dramatic exhibit for the statement made by
the authors that it is, in many cases, too early to make valid and reliable
assertions about the charter schools because of their unstable nature,
structure, and clientele.
Other tabular/quantitative
data clarification observations and recommendations include the following:
-
Arrange
the appendices in the same sequential order as they are first referenced
in the text.
-
The
data reported in Tables 3.3 and 3.4 lead to some confusion when the BS/BA
degree is used as the common point of reference. Both tables are a report
of formal education, as by the highest level of degree work or completion
of the respondents on a continuum from high school diploma attainment to
the doctorate. One suggestion would be to combine them into a single response
set.
-
Explain
the computation of the "raising factor" in Appendices A, B, and C and how
it contributes to an ` understanding of the charter school and climate
survey response rates.
-
Table 2.4, page 37, appears
to be a mixture of charter schools statewide, both those evaluated by The
Center, and those included in the review by Public Sector Consultants (PSC),
the Detroit area evaluator.
-
Clarify the source of the Ns
reported in Table 4.1, p. 51 (students = 280; parents = 188) inasmuch as
the charter school survey Ns were much larger (students = 1801; parents
= 944).
-
There
is a significant amount of quantitative data embedded in the narrative
at the start of Chapter Four - Students and Parents, as it relates to student
and parent/guardian samples. It might improve the reading and interpretation
of that important information if the narrative was further enhanced with
appropriate tabular presentation of the data.
Narrative clarification:
The following observations are made relative to the extensive narrative
in the report:
-
Do the first two sentences in
the fourth paragraph on page 64 regarding the authorizing agency fee mean
the same thing?
-
The statement on accountability
for student performance by charter school personnel (Section 7.4, pp. 82-83)
doesn’t need to be clarified, but rather highlighted as one of the bedrock
findings and conclusions regarding the progress of the charter school movement
in Michigan. Similar conclusions could probably be applied with accuracy
to most of the three dozen other states that are experimenting with charter
schools and school choice.
-
Analysis of the cost to charter
school applicant organizations as well as authorizing agency reviewers
might also be of interest (page 64)
-
It is interesting and striking
to note that, as a group, the PSA students tended to rate their charter
school experience lower than did the PSA parents and teachers and the national
norm group on some of the very items/reasons that parents cited for choosing
a charter school for their child (e.g., security and maintenance, instructional
management, etc.) (pp. 88-89).
-
In the area of school finance,
one wonders where some (substantial percentage) of the PSA money is being
spent in light of the fact that:
a) The PSAs get no less
than the average per pupil state foundation grant.
b) Personnel salaries and
fringe benefits consume well over 80% of most public school budgets.
c) Yet the PSAs, as a group,
are paying strikingly lower salaries and fringes while capturing at least
the average of what their host district receives.
Is it going to start-up
expenses? Profit margins? Etc.?
-
The
discussion under Section 7.8 Actual Use of Evaluation in the PSAs
should be amended to indicate that the discussion is actually referring
to personnel evaluation or teacher/staff performance appraisal and development
rather than program evaluation as implied by the section heading.
[top of the page]
U6 - Report
Timeliness and Dissemination: (A)
The Center did well under the
time constraints of a one-year contract, the lack of a central source of
a consistent database (because the contract period crossed more than one
fiscal year and the charter schools are relatively new), and the broad
geographical region covered (mostly rural, non-metropolitan Michigan) to
deliver and disseminate such an in-depth report on this evolving public
education reform initiative. While this writer is not familiar with the
dissemination requirements imposed on the contractor (The Center), it is
assumed that these were met to the satisfaction of the Michigan Department
of Education (MDE) and other major stakeholders, including representatives
of the mass media. It is common knowledge that selected finding from the
final report are already being quoted across the country through other
print media, albeit sometimes erroneously when taken "out of context."
[top of the page]
U7 - Evaluation
Impact: (A)
Time and circumstances will
dictate what the impact of this evaluation effort and the findings and
recommendations in the final report will be. The information provided therein
will be perceived differentially by different stakeholder groups. However,
it is the belief of this author that The Center evaluation team made a
concerted effort to create a positive climate for the review and use of
the Michigan PSA evaluation information, both during the process of collecting
the data and after the report had been disseminated, and by the primary
stakeholders in the schools (both PSAs and the host public school districts)
as well as secondary and tertiary stakeholders in positions of policy and
decision-making at the state and national level and in the mass media.
The use of traveling observers (TOs) and the attempt to provide a culture
of technical assistance and evaluation capacity building within and among
the PSAs that exceeded the requirements of the contract were outstanding.
[top of the page]
Feasibility
Standards
F1 - Practical
Procedures: (A)
There appeared to be a genuine
attempt to conduct the evaluation in as unobtrusive a fashion as was possible
under the circumstances: many new Michigan PSAs in their first or second
year of operation and others just getting started. It was apparent that
every effort was made to obtain extant data from the MDE, if at all possible,
to answer questions relevant to the evaluation and in response to the RFP.
The only direct intervention into the PSAs and their communities was to
sample the environment on the teacher/staff, student, and parent/guardian
charter school and climate surveys and by triangulating other observations
through selected interviews with PSA personnel. This was a difficult challenge
in that:
-
The primary players in the development
and initiation of a fairly radical new education reform like a charter
school, the teachers and administrators, are extremely busy developing
curriculum; administrative structures; and student induction, assessment,
and placement policies and procedures. The intervention of an outside party
will often be viewed as an unnecessary intrusion if not properly designed,
planned, and executed. The Center appears to have carefully considered
this potential and dealt with it properly.
-
In similar fashion, these "new
education" players do not come from or have experience with a culture of
self-evaluation.
-
The Michigan PSAs already have
two (2) external agencies, the MDE and the authorizing agencies, "looking
over their shoulders" making substantive value judgments regarding their
progress. Thus, they will see the evaluator as an interloper if the process
is not handled carefully and they don’t perceive the downstream value of
participating in the evaluation.
[top of the page]
F2 - Political
Viability: (A)
Much of what was said in the
previous section on evaluation procedures employed also applies to the
standard of political viability. The authors of The Center report point
out in several places their attempt to adhere to a standard of objectivity
in not only conducting the evaluation in the field, but in reporting the
result: "It is - in fact - for us, one of the most complex school reforms
that we have had the opportunity to evaluate. The polarized nature of the
reform and the strong divisions that exist between the proponents and opponents
made our task difficult" (Horn & Miron, 1999) (but apparently not impossible).
What all citizens interested in improving public education in the United
States in the 21st century must guard against now is the improper interpretation
and use of this kind of a baseline report by individuals and groups interested
in furthering their own narrow agendas.
[top of the page]
F3 - Cost Effectiveness:
(A)
Although this author is not
intimately familiar with the budget and financial resources allocated for
this evaluation effort, it appears that the project came in on time and
within budget. The resulting downstream positive utilization of the findings
and recommendations as it informs future legislation and research on charter
schools in Michigan and other states will be the true test of the effectiveness
and benefits of this effort.
[top of the page]
Propriety
Standards
P1 - Service
Orientation: (A)
The Center evaluation team attempted
to "engender the cooperation and assistance of all of the charter school
personnel" in this evaluation project. There was ample evidence that The
Center tried, as it does in all comprehensive evaluations of this type,
to build a culture among the primary stakeholders (in this case, the local
PSA administrators and teachers) whereby program evaluation would become
an integral part of the day-to-day operation and administration of the
school (PSA). The ultimate objective in The Center evaluation model is,
as this writer understands it, for the "client" to become so comfortable
with the concept of program evaluation that he/she will: (a) use evaluation
to identify program effects on PSA students, intended or not; (b) examine
program effects against the assessed needs of the PSA students; and, (c)
periodically (but systematically) inform the other primary PSA stakeholders
about the progress of their program and how evaluation is informing that
progress. The Center conducted the following activities during the life
of this evaluation project to initiate and promote a service orientation
within and among the client group:
-
Provided each PSA with a School
Self-Evaluation Kit (the contents of the Kit are listed on page 8 of the
final report).
-
Conducted six (6) half-day workshops
in The Center’s geographical evaluation area on the use of the Kit, the
project data collection needs and strategies, "and the broader concept
of school self-evaluation."
-
Invited not only PSA representatives
to these workshops, but also representatives of the authorizing agencies
and the Michigan Department of Education (MDE).
-
Hired and placed part-time traveling
observers (TOs) in the field to assist with data collection and provide
on-site technical assistance in the PSA evaluation effort.
The efforts of The Center to
reach out to the PSAs, provide technical assistance, and in so doing build
a culture of program self-evaluation were reported to have been met with
mixed success. The reasons were due, to some significant degree, to those
outlined above under Standard F1 - Practical Procedures, but it
was not for a lack of effort or attention by the contractor, The Center,
in this evaluation effort.
[top of the page]
P2 - Formal
Agreements: (A)
The formal agreement to conduct
an evaluation of the Michigan PSA initiative was between The Center and
the Michigan Department of Education (MDE). The final evaluation plan followed
throughout this educational program evaluation effort was that which was
proposed in The Center proposal in response to the MDE RFP, negotiated
between the parties, and incorporated as the work statement under the contract
between the MDE and The Center. Any deviations therefrom were documented.
Thus, this standard was addressed.
[top of the page]
P3 - Rights
of Human Subjects: (PA)
There was no evidence of a formal
Institutional Review Board (IRB) review being undertaken before the involvement
of human subjects in the research conducted within this evaluation project.
However, and by the same token, there was no evidence that human subjects
were abused or unduly taken advantage of as a result of their participation
in this project. Specific instructions and information were provided to
the participants at all levels as to the purpose of the evaluation, how
participants were selected, and how they were to participate in the data
collection process. This was done through the aforementioned technical
assistance workshops, the materials provided as guidelines with the administration
of the questionnaires, the TOs, and through the project website provided
and maintained by The Center.
[top of the page]
P4 - Human
Interactions: (A)
The Center has conducted a number
of large, comprehensive education and social service agency program evaluations
over the last twenty (20) years. The Michigan PSA Initiative evaluation
project is at least the second, statewide charter schools evaluation undertaken
by The Center. (Connecticut is ongoing and is the other charter school
evaluation contract known by this writer). Thus, it can be documented that
The Center has had a great deal of experience in properly assessing and
understanding "the culture, social values, and language differences of
the participants" in settings such as may be found in charter schools or
PSAs. It can be fairly stated and documented that The Center went appreciably
"above and beyond" the contractual expectations to establish and maintain
good communication and positive human interactions with the major stakeholders
relevant to this evaluation. Those include some strategies already mentioned,
but most likely not limited to:
-
The technical assistance workshops
-
The deployment of the traveling
observers (TOs)
-
A concerted attempt to be cognizant
through words and action regarding the time demands of the PSA personnel,
especially those in the first year of their operation
-
Ample opportunity for all stakeholders
to express themselves through the interview process
-
Establishment and maintenance
of the project electronic website at The Center.
[top of the page]
P5 - Complete
and Fair Assessment: (A)
It is the opinion of this writer
that this standard was met and exceeded by The Center evaluation team and
report authors. In fact, it is this author’s belief, based on a thorough
review and analysis of the final report, and supporting material, that
the terms "complete" and "fair assessment" are the hallmark characteristics/adjectives
to describe the current charter school evaluation effort.
The Center evaluation model
for this project included thorough responses to the contractual RFP questions,
as well as those added by The Center as formative and summative evaluation
questions posited to frame the study. The responses reported throughout
Chapters Two through Seven, Chapter Eight - Major Findings and Recommendations,
and the Executive Summary were open, balanced, and objective. The authors
did not try to shy away from sensitive issues, but rather reported their
professional observations based on the facts and evidence as they were
found at this point in the historical evolution of the charter school educational
reform movement in Michigan. Much can be gained by the proponents and opponents
by carefully digesting the information (findings, conclusions, and recommendations
for further research and policy action) in The Center report.
[top of the page]
P6 - Disclosure
of Findings: (A)
This writer has little knowledge
of activities engaged in by either the client, the Michigan Department
of Education, or the contractor, The Center, after the formal submission
of the final report to "ensure that the full set of evaluation findings
along with pertinent limitations are made accessible to the persons affected
by the evaluation, and any other with expressed legal rights to receive
the results." However, there is evidence that The Center made a reasonable
effort to conduct the formal evaluation in an atmosphere of "directness,
openness, and completeness," to wit:
-
In several places in the final
report, the authors cited difficulties and challenges in the data collection
process and steps taken to try to address those difficulties.
-
A balanced summary of field
notes and interview responses was reported.
-
An in-depth and insightful review
of the educational management organizations (EMOs) currently making their
presence felt in the Michigan charter school movement was rendered.
-
A good review of the strengths
and weaknesses of the charter school authorizing mechanism was given.
-
The Center authors stated openly
beliefs about the strengths and weaknesses of charter schools in Michigan
based on their findings.
-
Finally, there was no noticeable
attempt by the authors to "step outside" or beyond the findings of this
study to draw conclusions and render recommendations that could not be
supported by the evidence collected.
[top of the page]
P7 - Conflict
of Interest: (A)
As stated earlier, The Center
has conducted a large number of comprehensive educational and social service
program evaluations around the world in the last quarter of a century.
In fact, the vast majority of its work takes place outside of the state
of Michigan. Thus, this contractor had nothing to gain from an ethical
or business standpoint by trying to slant or bias the design or conduct
of the evaluation in one direction or another. This evaluation effort was
broad-based, drawing on the resources of a qualified team of experts and
traveling observers (TOs), using a variety of quantitative and qualitative
evaluation research techniques to elicit the maximum amount of valid and
reliable information possible within the time and resources made available
by the client pursuant to completion of this evaluation project.
[top of the page]
P8 - Fiscal
Responsibility: (A)
This writer did not have access
to the expenditure records related to this project. However, having said
that, it is our belief that The Center made optimal use (see previous sections
on use of TOs, utilization of extant quantitative data, etc.) of the resources
available to conduct the study, and as a component of a major university
that manages and accounts for millions of dollars worth of sponsored research
annually, all expenditures were properly accounted for and recorded.
[top of the page]
Accuracy
Standards
A1 - Program
Documentation: (A)
The program being evaluated,
the charter school initiative in Michigan, was clearly described and documented
in the final report. All of the contents of Chapter Two- Public School
Academies; Chapter Five - Legislation, Oversight and Management; Chapter
Six - Innovations in the Charter Schools; and, the introductory sections
of Chapter Three - Teachers and Staff; and, Chapter Four - Students and
Parents carry a more-than-adequate, valid characterization of the Michigan
charter school initiative. The excellent discussion of educational management
organizations (EMOs) (Chapter Five) adds baseline information to the understanding
of the organization and administration of Public School Academies that
exceeds the requirements of the RFP, yet is important to the total understanding
of the reader regarding the evolution of this education reform initiative.
Extant data, including indicators on student performance and teacher profile
information at the local host public school district and state levels,
were used advantageously as reference information to profile the Michigan
charter school programs. In summary, with reference to this evaluation
standard, the authors clearly and consistently stated that it is too early
to ascertain the real impact of this education reform in terms of outputs,
but the inputs and evolving processes were clearly described.
[top of the page]
A2 - Context
Analysis: (A)
This evaluation standard was
satisfactorily addressed in the judgment of the metaevaluator. The following
sub-categories were reviewed:
Geographic location:
Although the Michigan PSA initiative is statewide, the region of the state
included in this evaluation was clearly defined in the methodology frame
(Chapter 1).
Timing: It not clear
in the final report exactly when the charter school program in Michigan
was initiated, although by implication from the narrative in Chapter Two.
Also, there is reference to a Supreme Court ruling on page 57, but no explanation
or background information on how that action impacted the development of
this education reform in Michigan was given.
Political and Social Climate:
A clear sense of this aspect of the climate relative to the establishment
of charter schools in Michigan was gained by reading the background information
and description of the establishment of the authorizing agencies, especially
with reference to the authority of the universities to act as charter school
authorizers.
Staff: Clearly and
well-described in Chapter Three.
Pertinent economic conditions:
The state’s economy was not referenced directly, and inasmuch as one assumes
that Michigan’s economy correlates highly with the national economy (which
is currently very healthy), this was not a mitigating factor.
Other contextual conditions:
-
The authors clearly pointed
out that it is too early to draw permanent conclusions regarding the success
and/or failure of charter schools. They haven’t been in business long enough,
unstable administrative and governance structures prevail in some cases,
matching the curriculum to the avowed mission of the PSA is in evolution,
and host a of other factors are at play.
-
Divisive relationships between
teachers and administrators was reported in some cases.
-
Parental support, as documented,
is generally strong or those parents would not have enrolled their children
in a charter school.
-
Little sign of apathy was apparent;
however, it is clear that some parents may not have had a clear understanding
of what they were buying into when enrolling their children in a charter
school.
-
Little information was given
as to how the charter school movement in Michigan was publicized and promoted.
However, it is assumed that the public and major stakeholders were sufficiently
aware of the legal and ethical underpinnings of charter schools and had
adequate information to act as informed and responsible citizens within
the law and regulations governing their authorization, funding, governance,
and operation.
[top of the page]
A3 - Described
Purposes and Procedures: (A)
The purposes of the evaluation
were described adequately. They obviously related to the work statement
of the Request For Proposals (RFP) as well as addressing the additional
issues and questions posed by The Center evaluation team to give the study
contextual body and direction.
The procedures were well
documented in Chapter One. The quantitative data collection procedures
and analytical techniques were described and followed throughout the study
as reported in the ensuing chapters on teachers and staff, students, and
parents.
The observations this writer
would make regarding the stated procedures as reported in the final report
were twofold: 1) in the qualitative data collection domain (interviews,
observations, and focus group discussions), and 2) the post-survey data
collection decision to eliminate participating PSAs where parent/guardian
response rates on the school climate and charter school survey fell below
45%. With regard to the former, there is an instructive table (1:2 Data
Collection Strategies and Information Sources) found on page 11 of the
final report which outlines the intersection of intended sources of information
for the study with the data collection strategy proposed to access and
collect it (i.e., source x method). However, that table is never referenced
in the narrative or further explained procedurally as to how the evaluation
team proceeded systematically. This is particularly true with reference
to the administration of interview schedules (time, place, items, etc.)
and any focus group interactions that the evaluation team may have had
with the proposed sources (students, parents, community members, etc.).
On the latter, elimination of parent/guardian surveys, there was no discussion
as to why 45% was chosen as the "cut-off" point, and what, if anything,
was done to try to determine if the non-respondent (<45%) group were
materially different in demographic composition or attitudinal mind-set
regarding charter schools or the climate therein.
The only major change in
the negotiated contract between The Center and the MDE was the request
for and approval of a time extension for submitting the final report. This,
of course, did not change or impact the stated purposes or procedures of
the evaluation study.
[top of the page]
A4 - Defensible
Information Sources: (PA)
The primary stakeholders relevant
to the initiation, development, and operation of charter schools in Michigan,
the PSA teachers/staff, parents/guardians, and students, were clearly and
appropriately involved in contributing to the evaluation. This was achieved
primarily through the administration of the charter school and school climate
surveys to these three (3) groups. The sampling procedures were clearly
described and defensible.
The
only recommendation of the metaevaluator with regard to information sources
would be to include, time and resources permitting, more participation
and input from: (a) representatives of the public schools in terms of the
questions regarding the impact of charter schools on the public K-12 system
and, why local school districts and intermediate districts are participating
in the authorizing process to such a limited extent, and (b) representatives
of authorizing agencies, particularly universities, regarding the strengths
and weaknesses, challenges and successes, to date, of the authorizing process.
[top of the page]
A5 - Valid
Information: (A)
The information presented in
the final report appeared to be trustworthy and sound. There is no reason
to doubt or infer the invalidity of the quantitative data gathered through
the surveys. These instruments have been carefully constructed and tested
in other "like-type" settings. The climate survey is a nationally normed
instrument with its own validity and reliability-tested indices. The validity
of the conclusions made by The Center evaluation team and report authors
from the qualitative paradigm (field interviews, observations, and document
reviews) are based on and rooted in the expertise of the resource team
and traveling observers (TOs) who gathered, analyzed, interpreted, and
reported their findings.
[top of the page]
A6 - Reliable
Information: (A)
Much the same kind of metaevaluation
judgment can be made about the reliability of the information reported
from this study as was stated above in Standard A5 - Valid Information
discussion. The informed observer of this evaluation effort must believe
that sound information gathering procedures were used because of (1) the
half-day training and technical assistance workshops offered to the PSA
representatives in the field; (2) the procedures employed to correctly
sample reliable and representative members of the three (3) major PSA stakeholder
populations: teachers/staff, students, and parents/guardians; and (3) the
instrumentation to gather quantitative data, the charter school and climate
surveys. The team approach utilizing part-time traveling observers in the
field helped broaden and strengthen the use of qualitative data collection
and the interpretations of same.
It is assumed that internal
quality control procedures and checks were in place in The Center to ensure
the consistency of scoring, categorization, and coding of both the quantitative
and qualitative data sets.
[top of the page]
A7 - Systematic
Information: (A)
The aforementioned evaluation
training workshops designed to standardize the administration and collection
of surveys, the deployment of TOs, The Center’s Michigan charter schools
website, and the use of extant data from the Michigan Department of Education
(MDE) were all deemed to be helpful in ensuring that the information gathered
and analyzed pursuant to this evaluation was systematically treated. Reference
was made in the final report under 1.5 Limitations to the Evaluation
(p. 11) that up to 15 PSAs may have incorrectly "tampered" with the surveys
before returning them to The Center. It is not clear to this reader what
The Center personnel did to analyze the possible effects of this violation
of instructions, but do believe it did not have a material effect on the
findings for the respondent groups taken as a whole.
[top of the page]
A8 - Analysis
of Quantitative Information: (A)
One of the outstanding features
of this evaluation effort was the analysis of quantitative information,
both that captured from existing databases as well as primary data garnered
from the "self-report" surveys of the three major PSA stakeholder groups.
The authors attempted to make valid comparisons on the relevant indices,
when and where appropriate, between the newly minted public school academies
(charter schools) and their host public school districts, and with statewide
averages. This kind of reporting may raise the temptation of rabid proponents
(and opponents) of educational reform experiments to make technically unsubstantiated
claims about the alleged successes or failures of the reform. The authors
of The Center report provided clear and appropriate cautions to the readers
and users of the findings and conclusions regarding interpretations of
these kinds of comparisons.
Replicability of the data
collection strategies and analytical procedures would be straightforward
and reasonably easy to pursue.
[top of the page]
A9 - Analysis
of Qualitative Information: (A)
A variety of the standard qualitative
data collection strategies were employed in this study to gather information
about the development and status of the Michigan charter schools movement.
They included, but were probably not limited to (1) structured and unstructured
interviews; (2) nonparticipant observations; (3) documenting and recording
reviews; and (4) "unobtrusive data collection of various kinds." The data
yielded from these qualitative domain evaluation activities, coupled with
the studied analysis of the primary and secondary quantitative data included
as a part of the products of the study and the experience and expertise
of The Center evaluation team served to "flesh out" the report findings
and conclusions in the form of "descriptions, logical arguments, interpretations,
and impressions" of the authors who led The Center evaluation team.
The interview data were used
to supplement the quantitative findings, and to expand and clarify the
reader’s understanding of the development and status of charter schools
in Michigan. "Questions of interest" and "boundaries of information" were
obviously driven, in great part, by the Request for Proposals (RFP) and
the resulting contractual Statement of Work (SOW). However, The Center
evaluation team supplemented those areas with additional areas of inquiry:
innovative instructional practices and parental involvement, which helped
to "round out" the study within the context of a holistic and comprehensive
educational program evaluation.
Emergent evaluation questions
were handled with aplomb. Two topics that come to mind in reviewing the
final report include an in-depth review of the emergence and impact of
educational management organizations (EMOs) and, taken as a whole, the
lack of innovative educational practices in the Michigan PSAs, particularly
as it related to the integration and use of technology.
The major concern in the
area of qualitative inquiry in this study was/is the one pointed out earlier
under the standard on information sources. Practically all of the interview
data appeared to come from PSA principals or directors and very little
emanated from teachers, parents, or other stakeholders in the more external
environment of the PSAs (e.g., public school representatives, university
officials, etc.).
[top of the page]
A10 - Justified
Conclusions: (A)
This standard was, on balance,
more than adequately addressed in this evaluation effort. One must "track"
the RFP questions and their response sets throughout the final report to
arrive at this conclusion. A recommendation for accomplishing this is found
in the section on Standard U5 - Report Clarity (page 6-7) of this
report.
The authors did generate
and report a number of plausible alternative explanations and why these
should be discounted. In addition, the authors pointed out a number of
limitations to the study, both methodological and resource-wise. Finally,
one of the real strengths of this report was the constant and consistent
caution to the reader relative to incorrect or ill-advised interpretation
of equivocal findings. Specific locations of note in the final report regarding
this standard included the Foreword (p. i) and Chapter Seven - Demonstrating
Success (pp. 78, 81-82, 88, etc.).
[top of the page]
A11 - Impartial
Reporting: (A)
Again, and finally, it is the
professional opinion of this writer that this was sound, well balanced,
objective evaluation report, based on the facts and data that were gathered,
treated, analyzed, and reported. As previously stated in the discussion
on the propriety standards (e.g., P5 - Complete and Fair Assessment,
P6 - Disclosure of Findings, P7 - Conflict of Interest) and the last
section on A10 - Justified Conclusions, it is the judgment of the
metaevaluator that impartial reporting was one of the hallmarks of this
evaluation effort.
[top of the page]
A12 - Metaevaluation:
(A)
See this report.
[top of the page]
References
Michigan Department of Education.
(1997). State of Michigan public school academy initiative: Request
for Proposals. Lansing, MI.
Evaluation Center, Western
Michigan University. (1997). A proposal to evaluate the public school
academy initiative in the state of Michigan. Kalamazoo, MI.
Evaluation Center, Western
Michigan University. (1997). "A school self-evaluation kit for the Michigan
charter schools". Kalamazoo, MI.
Horn, J. & Miron, G.
(1999, January). Evaluation of the Michigan public school academy initiative.
Kalamazoo, MI: The Evaluation Center, Western Michigan University.
Joint Committee on Standards
for Educational Evaluation. (1994). The program evaluation standards
(2nd. ed.). Thousand Oaks, CA: Sage.
[top of the page]
Appendices
[top of the page]
[top of the page]
[Mich. Charter School
Evaluation Home]