Rural Systemic Initiative Evaluation Study—
Issue Paper
| The Evaluation Center | Western Michigan University |
| Volume 1, Number 1 | February 2002 |
Written by Gloria R. Tressler and Jerry G. Horn
Edited by Sally Veeder
As part of the Rural Systemic Initiative Study being conducted by The Evaluation Center at Western Michigan University, we did further analysis of survey data about perceptions and systemic reform. The surveys of six rural school districts showed that the building level administration was not chosen as the locus of greatest control, or most influential group, for any of the six drivers of educational systemic reform. Based on these perceptions, the loci of control for these reform drivers were divided among teachers and the district level administration.
There was great variation in the sample sizes for the stakeholder groups surveyed. Appendix A contains data on which part of this report was based.
The previous report entitled A Summary of RSI School Personnel’s Perceptions of the Drivers for Educational Systemic Reform (Horn, April 2001) offered analysis of 524 completed surveys from 6 case study sites in the Rural Systemic Initiative program. That report presented 3 major pieces:
1. Self-reported descriptive information about the respondents
2. The extent to which school-related personnel agreed that statements of the NSF-recognized six drivers for educational
reform (see Table 1) were perceived as valid for their schools
3. The choices, based on the highest percentage of responses, for the group/-authority with the perceived greatest control over each driver for educational reform; respondents from each of the six case study sites gave these responses relative to their own school and community.
Those results were based on aggregated data from the five respondent-stakeholder groups as defined by the first question of the Respondent Background Information section of the survey (i.e., (1) science or math teacher, (2) teacher of subject(s) other than science or math, (3) school administrator, (4) other school personnel, and (5) school board member). Those data were reported across all case study sites as well as for each site individually in the above-cited report.
This brief report offers an expanded review of available data that examines these perceptions differently—summarizing by each stakeholder group. This may help determine if there are any “disconnects” between different groups of respondents. Specifically, we are asking the question, “Are stakeholder perceptions in alignment with each other?”
Table 1. Drivers of Educational Systemic Reform as Described by the National Science Foundation (NSF):
1. Implementation of comprehensive, standards-based curricula as represented in instructional practice, including student assessment, in every classroom, laboratory, and other learning experience provided through the system and its partners.
2. Development of a coherent, consistent set of policies that support provision of high quality mathematics and science education for each student; excellent preparation, continuing education, and support for each mathematics and science teacher (including all elementary teachers); and administrative support for all persons who work to dramatically improve achievement among all students served by the system.
3. Convergence of the usage of all resources that are designed for or that reasonably could be used to support science and mathematics education—fiscal, intellectual, materials, curricular, and extracurricular—into a focused and unitary program to constantly upgrade, renew, and improve the educational program in mathematics and science for all students.
4. Broad-based support from parents, policymakers, institutions of higher education, business and industry, foundations, and other segments of the community for the goals and collective value of the program, based on rich presentations of the ideas behind the program, the evidence gathered about its successes and its failures, and critical discussion of its efforts.
5. Accumulation of a broad and deep array of evidence that the program is enhancing student achievement through a set of indices that might include achievement test scores, higher level courses passed, college admission rates, college majors, advanced placement tests taken, portfolio assessment, and ratings from summer employers and that demonstrate that students are generally achieving at a significantly higher level in science and mathematics.
6. Improvement in the achievement of all students, including those historically underserved.
|
Results
If we consider the highest percentage of responses for each category, the following results are revealed. The reader must be cautioned that there was great variation in the sample sizes of the stakeholder groups (see Appendix A). These results should be viewed as broad indicators of perceptions, especially for the stakeholder groups with very small sample sizes (i.e., school administrators and school board members).
§ There appears to be general agreement on the role of teachers for drivers 1, 3, and 6.
§ Four out of five stakeholder groups designated the district level administration as the locus for driver 4. School board members were the exception; they favored the building level administration as the locus for this driver.
§ Four out of five groups also chose the district level administration as the locus for driver 2. In this case, the school administrators chose the building level administration, by a narrow margin.
§ Responses for driver 5 were the most evenly dispersed among the levels of teacher, building, and district. Overall, the district level administration was most often chosen as the locus for this driver.
§ Mindful of the sample size issue, the responses of three groups—“science or math teacher,” “teacher of subject(s) other than science or math” (or “non-science/non-math teachers”), and “other school personnel”—contrast with the choices of the remaining two groups—“school administrator” and “school board member.”
§ The district level administration (local school board) was chosen more often as a locus of control than the building level administration.
§ The teachers’ and “others’” groups do not indicate the building level administration as the locus for any driver.
§ The very small sample size of the school board members’ category offers a glimpse of its thinking on the subject. This group rates teacher level equally with state level, building level administration, and district level administration loci for drivers 1, 3, and 6 respectively.
§ Teacher and other school personnel groups appear to have comparable responses for each driver of educational systemic reform.
§ Although the sample sizes are much smaller, responses from administrators and board members contrast with those of the remaining respondent groups.
Figure 1illustrates
Figure 1
illustrates two
issues about the responses of the five stakeholder groups. The bar graph is constructed to focus on perceptions about the
building level administration only. It
shows(1) overall low percentages of perceptions that
the building level administration is the most influential group for any of the
six drivers, and
(2) that differences among respondent groups tend to fall between the school board members and/or school administrators versus the remaining groups of teachers and other school personnel.
Appendix B is an edited reproduction of a table in a previous report (Horn, April 2001, Table 3, p. 32). It represents the combined responses of all stakeholder groups. It also demonstrates that the building level administration was not indicated as the locus of control for any of the drivers. Graphic representations (Figures 2 - 7) show the aggregate perceptions of the five respondent stakeholder groups. Each figure illustrates responses regarding an individual driver of educational systemic reform. The response options are teacher, building level administration, district level administration/local school board, state department of education or state school board, and U.S. Department of Education or other federal government agency[1]
response
rates by each of the five stakeholder groups with respect to choosing the
building level administration as the locus for the six drivers.
“Are stakeholder perceptions in alignment with each other?” “Do the respondent groups agree on their respective roles?”
There appears to be a “perception-connection” in the following:
§ All stakeholder groups are in alignment on the extensive and important role of the teacher, specifically through drivers 1, 3, and 6.
§ Respondent groups assigned the locus of control for drivers 2, 4, and 5 to the district level administration/local school board.
Figure 1 shows that in some instances there was a “perception-disconnection” by the school administrator and school board member groups versus the teachers and other school personnel groups. One may hypothesize that if the sample sizes of the administrators and board members had equaled those of the teachers and other personnel, the distribution of loci may have been similar across all groups.
Since all groups perceived the teacher’s role to be paramount, it is notable that stakeholder groups then jumped over the building level administration to the district level administration as their next choice of locus of greatest control. No driver of reform was designated as being mostly controlled by the building level administration. It is not clear if the authors intended the drivers of reform to imply this type of division of influence or whether it would be up to each school district to decide how to apportion control. In other words, is there a “correct” distribution of control loci?
Though systemic reform cannot be achieved solely on the basis of one stakeholder group’s efforts, it is a commonly held belief that the building administrator plays a fundamental role in school improvement processes. Indeed, Hall & Hord (2001) identified 12 principles of change. Of these, 2 seem applicable in this instance: “Administrator leadership is essential to long-term change success” and “The school is the primary unit for change.”
An earlier RSI report stated “Policy that supports building level autonomy [emphasis added] is important because state education policy is sometimes made with little regard for rural schools. Rural schools have sometimes found it necessary to depart from district or state policies and regulations to improve instruction. Building level autonomy gives schools the flexibility to do what is in the best interest of their students” (Russon, Paule, & Horn, May 2001, p. 10). Since local school boards and district level administrators are more distanced from the daily activities of an individual school, survey responses that put greater emphasis on the district level administration seem to be in conflict with the building level autonomy concept.
In summary, it may be inconsequential whether stakeholder group responses reflect the way they believe the system should operate or whether they reflect their experiences within the current educational system. These responses led us to some conclusions:
§ The perception-connection that teachers have the greatest control or influence over the majority of the drivers of reform carries with it the assumption that they also bear the greatest level of responsibility. Therefore, they should be the main focus of additional training, support, and accountability measures.
§ Strengthening the educational and leadership skills, interest, and follow-through on the part of the building level administration may significantly enhance a school’s performance in its duties to each of its students.
§ For the NSF-recognized drivers of educational systemic reform to be useful to schools struggling to improve, their intent with regard to implementation and accountability must be fully understood by all stakeholders.
Certainly, all levels of the educational system need to be coordinated to avoid conflicts of interest and wasted efforts and time. Most importantly, stakeholder groups, individually and collectively, must come to terms with their mandate to provide high-quality educational experiences for each student in their charge. We offer the following questions for reformers to consider:
§ If stakeholder groups agree on their respective roles, can this perception-connection be capitalized upon to facilitate systemic reform?
§ If groups are in agreement, are they also in alignment with educational systemic reform? That is, are these perceptions correct in the eyes of those promoting reform?
§ If they are in disagreement, is this a barrier to reform?
§ And if there are barriers due to this type of perception-disconnection, can they be reduced or eliminated?
In essence—use stakeholder perceptions to re-form potholes of change implementation into portholes.
Hall, G., & Hord, S. (2001). Implementing change: Patterns, principles and potholes. Boston: Allyn and Bacon.
Horn, J. G. (2001, April). A summary of RSI school personnel’s perceptions of the drivers for educational systemic reform: Prepared for the NSF rural systemic initiatives evaluation study. Kalamazoo: Western Michigan University, The Evaluation Center.
Russon, C., Paule, L., & Horn, J. (2001, May). The relationship between the drivers of educational reform and the rural systemic initiatives in science, mathematics, and technology education program of the National Science Foundation. Kalamazoo: Western Michigan University, The Evaluation Center.
|
|
Percentage of Responses for Each Locus of Control Level by Stakeholder Group |
||||
|
Locus Level Chosen for Driver 1 |
Science/Math Teacher (n = 127)
% |
Non-Science/Non-Math Teacher (n = 247) % |
Other School Personnel (n = 87)
% |
School Administrators (n = 17)
% |
School Board Members (n = 7)
% |
|
Teacher |
62.9 |
53.6 |
46.6 |
41.2 |
33.3 |
|
Building |
4.0 |
8.5 |
9.1 |
23.5 |
16.7 |
|
District |
17.7 |
20.4 |
22.7 |
23.5 |
16.7 |
|
State |
14.5 |
15.3 |
17.0 |
11.8 |
33.3 |
|
Federal |
0.8 |
2.1 |
4.5 |
0.0 |
0.0 |
|
Locus Level Chosen for Driver 2 |
Science/Math Teacher (n = 127)
% |
Non-Science/Non-Math Teacher (n = 247) % |
Other School Personnel (n = 87)
% |
School Administrators (n = 17)
% |
School Board Members (n = 7)
% |
|
Teacher |
16.9 |
20.1 |
15.3 |
25.0 |
16.7 |
|
Building |
23.4 |
17.1 |
15.3 |
31.3 |
16.7 |
|
District |
37.1 |
42.7 |
34.1 |
25.0 |
66.7 |
|
State |
21.0 |
17.5 |
30.6 |
18.8 |
0.0 |
|
Federal |
1.6 |
2.6 |
4.7 |
0.0 |
0.0 |
|
Locus Level Chosen for Driver 3 |
Science/Math Teacher (n = 127)
% |
Non-Science/Non-Math Teacher (n = 247) % |
Other School Personnel (n = 87)
% |
School Administrators (n = 17)
% |
School Board Members (n = 7)
% |
|
Teacher |
42.5 |
35.3 |
34.9 |
52.9 |
50.0 |
|
Building |
20.8 |
20.3 |
16.3 |
17.6 |
50.0 |
|
District |
27.5 |
33.2 |
32.6 |
29.4 |
0.0 |
|
State |
7.5 |
9.1 |
12.8 |
0.0 |
0.0 |
|
Federal |
1.7 |
2.2 |
3.5 |
0.0 |
0.0 |
|
Locus Level Chosen for Driver 4 |
Science/Math Teacher (n = 127)
% |
Non-Science/Non-Math Teacher (n = 247) % |
Other School Personnel (n = 87)
% |
School Administrators (n = 17)
% |
School Board Members (n = 7)
% |
|
Teacher |
14.0 |
16.2 |
16.3 |
23.5 |
20.0 |
|
Building |
23.1 |
23.1 |
19.8 |
17.6 |
80.0 |
|
District |
49.6 |
45.9 |
51.2 |
41.2 |
0.0 |
|
State |
11.6 |
10.5 |
5.8 |
17.6 |
0.0 |
|
Federal |
1.7 |
4.4 |
7.0 |
0.0 |
0.0 |
|
Locus Level Chosen for Driver 5 |
Science/Math Teacher (n = 127)
% |
Non-Science/Non-Math Teacher (n = 247) % |
Other School Personnel (n = 87)
% |
School Administrators (n = 17)
% |
School Board Members (n = 7)
% |
|
Teacher |
27.6 |
29.8 |
23.9 |
29.4 |
20.0 |
|
Building |
16.3 |
15.8 |
22.7 |
35.3 |
80.0 |
|
District |
28.5 |
31.6 |
30.7 |
23.5 |
0.0 |
|
State |
24.4 |
19.7 |
18.2 |
11.8 |
0.0 |
|
Federal |
3.3 |
3.1 |
4.5 |
0.0 |
0.0 |
|
Locus Level Chosen for Driver 6 |
Science/Math Teacher (n = 127)
% |
Non-Science/Non-Math Teacher (n = 247) % |
Other School Personnel (n = 87)
% |
School Administrators (n = 17)
% |
School Board Members (n = 7)
% |
|
Teacher |
59.0 |
54.1 |
47.7 |
58.8 |
40.0 |
|
Building |
13.1 |
12.1 |
14.0 |
11.8 |
20.0 |
|
District |
12.3 |
19.5 |
20.9 |
11.8 |
40.0 |
|
State |
10.7 |
10.8 |
7.0 |
11.8 |
0.0 |
|
Federal |
4.9 |
3.5 |
10.5 |
5.9 |
0.0 |
|
Appendix B. Summary of All Respondents’ Perceptions of the Locus of Control for the Drivers Indicative of Educational Systemic Reform* |
|||||
Driver |
Teacher Level |
Building Level Administration |
District Level Administration |
State Dept. of Education |
U.S. Dept. of Education |
|
1 |
53.4 |
7.6 |
20.1 |
16.3 |
2.6 |
|
2 |
17.9 |
18.3 |
38.2 |
22.3 |
3.2 |
|
3 |
36.3 |
20.2 |
31.6 |
9.2 |
2.6 |
|
4 |
15.0 |
22.0 |
49.1 |
10.1 |
3.9 |
|
5 |
27.3 |
17.5 |
31.8 |
20.0 |
3.5 |
|
6 |
54.0 |
12.2 |
17.7 |
10.6 |
5.5 |
|
* Percentage of total responses to each item/driver. |
|||||
The Evaluation Center
at
Western Michigan University
Rural Systemic Initiative Evaluation Study
Principal Investigator
Jerry G. Horn
Larry Enochs
Gene Hall
Mary Harris
James Jess
Paul Nachtigal
Steve Oliver
Jack Sanders
Robert Stephens
Daniel Stufflebeam
William Webster
Project Consultants
Brian Lotven
Kenneth McKinley
Louisa Stark
Research Assistant
Gloria R. Tressler
This material is based upon work supported by the National Science Foundation under Grant No. REC-9819347. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).