RURAL SYSTEMIC INITIATIVES’

 SURVEY 2002 REPORT

(Perceptions and Experiences of

 Key Participant Personnel)

Prepared for the

NSF Rural Systemic Initiatives Evaluation Study

by

Jerry G. Horn and Gloria Tressler

The Evaluation Center

Western Michigan University

Kalamazoo, MI 49008-5237

October 2003

This material is based upon work supported by the National Science Foundation (NSF) under Grant REC-9819347. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NSF.

TABLE OF CONTENTS

Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii

 

Perceptions and Experiences of Key Participant Personnel

 

Table 1. States Included in RSIs of the Study

 

Table 2. Survey Distribution and Response

 

Table 3. Administrative Contacts’ Perceptions of

Initial Expectations and Current Experiences

 

Table 3A. Administrative Contacts’ Perceptions of Initial

Expectations and Current Experiences: Comparisons

 

Table 3B. Administrative Contacts’ Perceptions of Impact on

Selected Elements of the Instructional Program

 

Table 4. Principals’ Perceptions of Initial Expectations

and Current Experiences

 

Table 4A. Principals’ Perceptions of Initial Expectations

and Current Experiences: Comparisons

 

Table 4B. Principals’ Perceptions of Impact on Selected

Elements of the Instructional Program

 

Table 5. Local Facilitators’ Perceptions of Initial

Expectations and Current Experiences

 

Table 5A. Local Facilitators’ Perceptions of Initial

Expectations and Current Experiences: Comparisons

 

Table 5B. Local Facilitators’ Contacts’ Perceptions of

Impact on Selected Elements of the Instructional Program

 

Table 6. Summary of Open-End Responses to Critical Issues

 

Findings

 

APPENDIX A (Lists of Contact Persons)

 

APPENDIX B (Cover Letter to Contact Persons)

Foreword

On behalf of the evaluation study team from The Evaluation Center at Western Michigan University, I want to thank the administrative leadership of the Delta, Texas, Appalachia, Coastal, and Michigan Rural Systemic Initiatives for their cooperation and support for this ongoing data collection process. Further, it is critical that participating school districts share their perceptions and experiences with the study team to help us and others understand how this large-scale school improvement effort has impacted the local science and mathematics curriculum and instruction for children and youth. To those people who responded to this survey (administrative contacts, school principals, and local facilitators), we extend our heartfelt gratitude.

We could not have completed the data analysis and development of the report without the able assistance and support of Sally Veeder, Mary Ramlow, Christine Hummel, and Barbara Miller at The Evaluation Center. Finally, I want to extend my appreciation to Ms. Gloria Tressler for work on this project as a Research Assistant. Her dedication to details and “getting it right” are attributes that help ensure that this study is meaningful and an accurate reflection of the data.

Jerry G. Horn

Perceptions and Experiences of Key Participant Personnel

Over the years, the National Science Foundation has created various programs to enhance and improve science and mathematics education for K-12 children and youth. One of the more recent and heavily funded education programs is under the general heading of systemic reform (state, urban, and rural). A description of the Rural Systemic Initiatives, as publicly described by NSF, is found below.

 

The Rural Systemic Initiatives in Science, Mathematics, and Technology Education Program was developed in FY 1994. RSI like the Urban Systemic Initiatives and the Statewide Systemic Initiatives stimulates system-wide educational reform of science, mathematics, and technology.

 

RSI is focused on improved education for students in rural, economically disadvantaged regions of the nation, particularly those that have been underserved by NSF programs; and on sustaining the improvements through encouraging community participation in instructional and policy reform. RSI is tailored to address policy, leadership, and workforce issues related to education, to provide a comprehensive and sustainable framework for science, mathematics, and technology education technology in elementary, secondary, and higher education.

 

RSI targets regions that are highly rural; characterized by significant levels of poverty among their school-age children; and that share common cultural, social, and economic characteristics. Interested individuals or identified leaders within eligible regions form consortia that include large NSF-funded Initiatives (e.g., SSI, EPSCoR, USP). RSI regions can be geographically vast, typically crossing state lines, or may be composed of areas that are geographically separated but linked by a unitary vision and other commonalities.

 

The RSI project scope is divided into two strategic elements. The initial element is a Development phase, to support planning for structuring Implementation (second) phase vision, strategies, and priorities. During the Development period, projects conduct regional self-studies to gather pertinent information regarding target populations, regional strengths and barriers, and policies that enhance or hinder instructional reform. Moreover, successful Development projects should result in a viable evaluation strategy to be utilized during the Implementation phase. The second strategic element, the Implementation phase, focuses on realizing the strategies for systemic improvement in RSI districts, schools, and classrooms. The strategies can include teaching workforce enhancements, curriculum innovation, leadership development among teachers and local district administrators, and innovative and pertinent assessment strategies; and must result in better classroom instruction and higher student achievement.

The goals of the RSI program are listed below:

 

            •     The improvement of science, mathematics, and technology education in rural, economically disadvantaged regions of the nation.

 

            •     The preparation of a technologically competent workforce to enhance the infrastructure of economic development activities within a community or region, by strengthening the science, mathematics, and technology instructional capacities of regional colleges and universities.

 

            •     The enhancement of scientific literacy and science understanding and appreciation among students and the general community in rural, economically disadvantaged regions of the nation.

 

            •     The development of community infrastructure to provide resources to sustain educational improvement.

Further, NSF recognizes “Six Drivers for Educational System Reform”:

 

            1.    Implementation of comprehensive, standards-based curricula as represented in instructional practice, including student assessment, in every classroom, laboratory, and other learning experience provided through the system and its partners.

 

            2.    Development of a coherent, consistent set of policies that supports: provision of high quality mathematics and science education for each student; excellent preparation, continuing education, and support for each mathematics and science teacher (including all elementary teachers); and administrative support for all persons who work to dramatically improve achievement among all students served by the system.

 

            3.    Convergence of the usage of all resources that are designed for or that reasonably could be used to support science and mathematics education—fiscal, intellectual, material, curricular, and extra-curricular—into a focused and unitary program to constantly upgrade, renew, and improve the educational program in mathematics and science for all students.

 

            4.    Broad-based support from parents, policymakers, institutions of higher education, business and industry, foundations, and other segments of the community for the goals and collective value of the program, based on rich presentations of the ideas behind the program, the evidence gathered about its successes and its failures, and critical discussions of its efforts.

            5.    Accumulation of a broad and deep array of evidence that the program is enhancing student achievement, through a set of indices that might include achievement test scores, higher level courses passed, college admission rates, college majors, Advanced Placement Tests taken, portfolio assessment, and ratings from summer employers, and that demonstrate that students are generally achieving at a significantly higher level in science and mathematics.

 

            6.    Improvement in the achievement of all students, including those historically underserved.

In late 1998, the National Science Foundation funded a project at The Evaluation Center at Western Michigan to conduct “The Rural Systemic Initiatives Evaluation Study.” The project, under the direction of Dr. Jerry G. Horn, had four objectives:

 

1.         To develop a system of indicators around each of the identified six drivers of educational system reform

 

2.         To determine the perceived relative importance and value of each of the drivers and indicators for reform in RSI schools in selected communities

 

3.         To determine the status of innovation/reform within selected communities with respect to factors thought to support or serve as barriers to innovation and education reform

 

4.         To determine the ways and the extent to which the perceived importance and value of the drivers and the characteristics of the community impact on systemic reform efforts and student achievement in mathematics, science, and technology

Initially, the WMU study was to focus on three RSI collaboratives: Appalachia, Delta, and UCAN. In Spring 2000, the NSF asked the project director to expand the study to include three newly funded collaboratives (Texas, Coastal, and Michigan). Based on early findings and ques-tions, WMU also sought to investigate four additional questions as a part of the expanded study:

 

•          What new or different forms of student assessment and teacher effectiveness have been developed and used as a result of curriculum transformation and alignment with state or national standards in science and mathematics, and how were these developed and used?

 

•          What contextual factors (within and across collaboratives) serve to support reforms that result in or are associated with the development and implementation of standards-based curricula?

 

•          What processes and conditions are essential for effective partnerships within large-scale collaboratives that are characterized by large geographic distances between entities and multiple/potentially contrasting governance arrangements?

•          What considerations have been given to the use of technology for enhancing the accessibility and effectiveness of math and science instruction and communication within and among collaborative members?

By mid-2002, five of the collaboratives were functioning as a part of their original funding, with a mutually agreeable extension, or with additional funding for an expanded RSI or as a second cycle. The UCAN project was closed, but several elements of the original group resurfaced as leaders or participants in small, but related projects.

Through a series of 15 case studies, surveys, and other data collection procedures, a substantial amount of data regarding the new and additional questions have been accumulated and reported in various forms. However, the individual RSI projects kept evolving, and the participating schools were only then realizing the form and impact of the efforts designed to improve their science and mathematics curricula. However, since many of the projects were “in process” and others were at the earliest stages of development, additional information and evidence seemed warranted. Therefore, The Evaluation Center sought and was approved by NSF for a one-year, no-cost extension of the evaluative study (to May 31, 2004), which will provide local school personnel with the time to reflect on what has happened and where the impact (breadth and depth) of the effort has occurred.

We chose to include five RSI collaborative projects for this survey activity: Appalachia, Coastal, Delta, Michigan, and Texas. UCAN was not included because its organization and the types of schools it served were quite different from the other collaboratives. Additionally, its funds had been discontinued. The principal investigators of the five RSI collaborative projects were asked to supply the name and address of each project site contact person within their purviews (see Appendix A). Appendix B contains a copy of the cover letter that accompanied the surveys to each contact person identified by the principal investigators. None of the collaboratives supplied the contact information for all of the projects under their domains. They selected the projects they deemed appropriate to partake in this survey, i.e., those that had participated with a reasonable level of involvement for the course of two or more years.

Each complete survey package contained three smaller packets and a cover letter to the administrative contact. Each smaller packet was labeled “Building Principal”; “School District RSI Contact Person”; or “Local Facilitator, Teacher Partner, or a Local Person who has substantial involvement in the RSI program” and included a survey for its anticipated respondent along with a postage-paid return envelope. To keep them informed of the process we were following, complete survey packages were sent to each principal investigator.

Table 1 shows details about the numbers of survey packages sent to and later returned by each collaborative. Michigan RSI had the highest response rate for each of the three types of surveys included in the complete survey package. In general, the school principals had the lowest rate of response, except for those from the Coastal RSI.

Table 1. States Included in RSIs of the Study*

States	Rural Systemic Initiatives (RSIs)
	Appalachia	Coastal	Delta	Michigan	Texas
Arkansas			X
Mississippi			X
Louisiana			X
Michigan				X
Texas					X
South Carolina		X
North Carolina	X	X
Virginia	X	X
Ohio	X
Tennessee	X
Kentucky	X
West Virginia	X

*Only selected school districts are included in specific RSIs, not all schools within that state.

Table 2. Survey Distribution and Response

RSI Collaborative	Number of Complete Survey Packages Mailed	Number of Individual Surveys Returned by Each Respondent Group
		Administrative Contacts		School Principals		Local Facilitators
		N	%	N	%	N	%
Appalachian	45	20	44	9	20	17	38
Coastal	9	5	56	4	44	1	11
Delta	78	16	21	7	9	10	13
Michigan*	15	12	80	7	47	8	53
Texas	48	26	54	18	38	19	40

*Michigan included an extra letter written by the PI to the contact persons.

While the return rate is not what was hoped for or expected, there are some explanations. First, the packets of surveys were sent to the “administrative contact” in individual districts with instructions or a request for them to complete the one for them and forward one set to a building principal and one to the local facilitator” Without names and addresses, it was impossible to follow up. Secondly, the collaboratives had just been “hit” with a demand for a substantial amount of data from another NSF-funded research effort, which required some to find monies to hire persons to collect data and report it in specific ways. Lack of coordination by NSF likely resulted in confusion at the local level and lack of responsiveness to the substudy being reported here.

Below is a breakdown of the primary role of the administrative contacts for this study:

 

                  Teacher                                                          8.9 percent

                  Principal                                                         6.3 percent

                  Superintendent                                               16.5 percent

Curriculum Coordinator/Supervisor53.2 percent

                  Other                                                              15.2 percent

Over the course of a year, the administrative contact respondents reported that they distribute their time in the following ways:

 

                  RSI administration/coordination                            14.8 percent

                  Teaching math and/or science                                11.0 percent

                  Teaching courses other than science or math          2.3 percent

                  Other duties                                                            72.1 percent

The principals represent schools at the following levels that have participated in RSI activities. (Note: A district might participate at more than one level; therefore, these do not sum to 100 percent.)

 

                  Elementary                   66.7 percent

                  Middle/junior high       60.0 percent

                  High school                  40.0 percent

                  Other                             4.4 percent

Principals reported these estimates of the distribution of their duties:

 

                  Administration                                  51.2 percent

                  Curriculum leadership                      21.7 percent

                  Professional development                 10.5 percent

                  Public relations                                 12.6 percent

                  Other                                                  3.9 percent 

The primary roles of the local facilitators who responded to the survey are reported below:

 

                  Teacher                                             73.6 percent

                  Principal                                             3.8 percent

                  Superintendent                                   1.9 percent

                  Curriculum Coordinator/Supervisor  5.7 percent

                  Other                                                 15.1 percent 

Of the local facilitators who reported that they are teachers, the approximate amount of their time devoted to teaching the various subject areas is found below.

 

                  Science                                              41.1 percent

                  Math                                                  35.3 percent

                  Computers and technology                9.8 percent

                  Other                                                 13.7 percent

The local facilitators have an average of 18.9 years of experience in education, and they have been involved as facilitators for the RSI effort for an average of 2.8 years.

From the initial discussion with NSF about the purpose and procedures to be used in this study, we emphasized the intent to study the RSI collaboratives from the perspective of local schools and communities. In many cases, this involved on-site observations, face-to-face interviews, and review of actions and documents. This survey was sent to representatives of the three primary groups involved in choosing to participate in the project, carrying out the activities at the local level, and observing the impact of the program. Specifically, these key informants are the administrative contacts, the school principals, and the local facilitator or the person charged with day-to-day contact with teachers. While the response rates do not permit a reasonable statistical analysis between projects, which is not our intention, the responses overall provide us with information about three major areas: expectations, experiences, and impact. In other words, did the originators of the collaboratives pledge one thing and do something different? Did the project originators mislead local planners? Did they promise more than they delivered, etc.? The important question about impact regards whether the project had any influence on practice and results, i.e., increasing student test scores, reducing the number of dropouts, etc.

The data for the remaining portion of this report are grouped by respondent type; 3, 3A, and 3B represent responses from administrative contacts; 4, 4A, and 4B reflect the responses of school principals; and 5, 5A, and 5B come from local facilitators. These three groups are central to the RSI effort, and they should be in positions to know what is happening. Importantly, they are not the target audience for much of the RSI work; therefore, they can be more objective in their responses than if they were actually the focus of the professional development work or had direct responsibility for in-class instruction and student performance.

The narrative in the next section of this report highlights certain points and does not represent an explanation of every data item. However, readers of this report may want to review all of the data summaries to gain a more comprehensive and inclusive perception.

The percentages of administrative contacts who perceived that 17 statements are “False,” “Partly True,” or “True” with regard to initial expectations and their current experience are found in Table 3. (Similar data for school principals are found in Table 4 and for local facilitators in Table 5.) With regard to expectations, the administrative contacts initially perceived these issues or factors to be true:

 

■   The curriculum would be aligned with state standards. (79.7 percent)

■   Student achievement would improve. (74.7 percent)

■   Curriculum development would be a major focus and result of the reform movement. (70.9 percent)

■   Teacher training/in-service would be available to help all teachers. (68.4 percent)

■   On-site assistance for teachers would be provided by RSI personnel. ((67.1)

School principals perceived these items as most true (see Table 4):

 

■   The curriculum would be aligned with state standards. (71.1 percent)

■   Student achievement would improve. (71.1 percent)

■   On-site assistance for teachers would be provided by RSI personnel. (71.1 percent)

■   Monies would be available for teachers to attend professional conferences and meetings. (68.9 percent)

■   A local facilitator would provide support and leadership on a regular basis. (64.4 percent)

A similar list for local facilitators follows (see Table 5):

 

■   The curriculum would be aligned with state standards. (71.4 percent)

■   Student achievement would improve. (66.1 percent)

■   On-site assistance for teachers would be provided by RSI personnel. (62.5 percent)

■   Instruction would become laboratory/activity centered. (58.9 percent)

■   Teacher training/in-service would be available to help all teachers. (57.1 percent)

Clearly, there is strong agreement on three items: curriculum alignment, improved student achievement, and on-site assistance or some form of assistance for teachers.

After participating in the project, the highest rated items with regard to realization or experience (highest percentage that were marked True) for each of the groups are listed below:

Administrative Contacts

 

■   The curriculum would be aligned with state standards. (81.0 percent)

■   Curriculum development would be a major focus and result of the reform movement. (75.9 percent)

■   On-site assistance for teachers would be provided by RSI personnel. (72.2 percent)

■   Teacher training/in-service would be available to help all teachers. (70.9 percent)

■   Planned improvements would be tied to valid interpretation of test data. (70.9 percent)

School Principals

 

■   The curriculum would be aligned with state standards. (86.7 percent)

■   On-site assistance for teachers would be provided by RSI personnel. (80.0 percent)

■   Curriculum development would be a major focus and result of the reform movement. (73.3 percent)

■   Monies would be available for teachers to attend professional conferences and meetings. (71.1 percent)

■   Leadership training would be a major intervention. (66.7 percent)

Local Facilitators

 

■   The curriculum would be aligned with state standards. (80.4 percent)

■   On-site assistance would be provided by RSI personnel. (75.0 percent)

■   Leadership would be a major intervention. (66.1 percent)

■   Monies would be available for teachers to attend professional conferences and meetings. (64.3 percent)

■   Curriculum development would be a major focus and result of the reform movement. (62.5 percent)

Again, there is consistency in the items that appeared in the lists of respondents. An item that was identified as a high expectation and that has now dropped from the top 5 of all respondent groups is “Student achievement would improve.” Yet, the percentage of respondents who most currently marked “True” for that issue/factor ranged from 55.4 percent for local facilitators to 68.4 percent for administrative contacts.

Noticeably low expectations and realizations were recorded for the following items:

 

■   Parents would become involved in substantive ways.

■   The community (not just parents) would become involved in the school.

■   A new math/science program would be implemented.

Another way of looking at these same data is projected in Tables 3A, 4A, and 5A where the differences in the mean of the responses (1 = False, 2 = Partially True, and 3 = True) were calculated (Difference = Mean of the current experience - the mean of the initial expectation). A negative value in the “Difference” column would indicate that the current experience for a particular issue or activity is less than what was initially expected. From the perception of administrative contacts, these items had the largest negative value:

 

■   Instruction would become laboratory/activity-centered. (-0.2)

■   Parents would become involved in substantive ways. (-0.2)

■   Monies would be available for purchasing classroom materials. (-0.3)

School principals rated none of the negative items further from 0 than -0.1. All of the others were 0 or plus. The three with negative values pertained to student achievement, role of local facilitator, and parental involvement. For local facilitators, two with negative values are of note:

 

■   Parents would become involved in substantive ways. (-0.2)

■   Monies would be available for purchasing classroom materials. (-0.4)

Indications that experience exceeded expectations are relatively few, but those of note are listed below under the three categories of respondents.

Administrative Contacts

 

■   Leadership training would be a major intervention. (0.3)

■   A local facilitator would provide support and leadership on a regular basis. (0.2)

School Principals

(None greater than 0.1)

Local Facilitators

 

■   Leadership training would be a major intervention (0.2)

■   Curriculum development would be a major focus and result of the reform movement. (0.2)

■   External audits or reviews of our curriculum would occur. (0.2)

In sum, the initial expectations were not far removed from what was experienced, i.e., the participating schools got what they expected. However, it does seem that the expectations were relatively low, which may be a result of their experiences with outside funded projects or simply a reflection of the culture in which they are located.

Of course, the impact of the program is the more important aspect to consider for the RSI projects. Did they make a difference? Will there be any residuals or evidence of the program after cessation of the interventions/activities and termination of the overall project? At the time of this writing, several RSI collaboratives are nearing the end of their funding period or have terminated and, in some cases, have no organized NSF support to sustain this effort or begin a new set of initiatives.

On a four-point scale (1 = Not at All to 4 = Great Extent) with regard to the perceived extent of impact due to the RSI in which their school participated, the administrative contacts identified the following areas of greatest impact (greater than 3.0). (See Tables 3B, 4B, and 5B.)

 

■   Math/science curriculum aligned with recognized set of standards (3.6)

■   Curriculum more challenging for all students (3.3)

■   Increased requirement and expectations of student in math/science (3.3)

■   Increased use of group and/or cooperative learning experiences for students (3.1)

■   Policies that support high standards and good preparation of new teachers (3.1)

■   Improved achievement test scores (3.1)

■   Advocacy/support for math/science among all teachers (3.1)

School principals perceived the greatest impact on the following:

 

■   Math/science curriculum aligned with recognized set of standards (3.6)

■   Curriculum more challenging for all students (3.4)

■   Increased ownership of the math/science curriculum by teachers (3.3)

■   Content of statewide tests shape the curriculum (3.3)

■   Increased requirement and expectations of students in math/science (3.3)

■   Curriculum more challenging to the best students (3.2)

■   Increased use of group and/or cooperative learning experiences for students (3.2)

■   Policies that support higher quality education (3.1)

■   Policies that support high standards and good preparation of new teachers (3.1)

■   Improved student achievement scores (3.1)

■   Advocacy/support for math/science among all teachers (3.1)

■   Expanded student access to laboratory/activity-oriented instruction (3.1)

Local facilitators perceived these issues as being impacted the most:

 

■   Math/science curriculum aligned with recognized set of standards (3.4)

■   Curriculum more challenging for all students (3.2)

■   Expanded student access to laboratory/activity-oriented instruction (3.2)

■   Increased use of group and/or cooperative learning experiences for students (3.2)

■   Increased requirement and expectations of students in math/science (3.2)

Two items rated highest on all three lists:

 

■   Math/science curriculum aligned with recognized set of standards

■   Curriculum more challenging for all students

Only two of the three groups perceived an issue or area was impacted so little that it produced a mean value of less than 2.0 (Slight Extent). The mean for school principals’ responses was 1.9 for “Lowered dropout rate”; and local facilitators rated “Student engagement in community service projects, especially, those requiring some math/science knowledge and skills” as 1.9.

Other areas also warrant attention with regard to impact, but these will be discussed as a part of the “Findings” in this report when the six drivers are reintroduced in considering the overall reform and school improvement process.

The three groups of respondents were also asked to list the “three most significant changes,” the “three greatest challenges,” and the “three most important indicators of reform.” The responses were reviewed, categorized, and are summarized in Table 6.

In examining these responses, there is a fairly consistent perception of the changes, challenges, and indicators of education reform among these key informants from participating RSI schools. There will be more discussion of these in relation to the case studies developed in the overall evaluative study, but the issues related to attitudes, opportunities, and skills appear time and time again. While student achievement is broadly recognized as an indicator of (positive) reform, there may be less agreement on how achievement is defined. Among the potential indicators of student achievement are course grades in K-12 schools, state or other standardized test scores, acceptance or admittance to higher education, success in various forms of scholastic competition, scholarship awards, acceptance into specialized programs, success in careers and employment, etc.

In sum, all of these may be valid evidence of achievement and, collectively, that may be the best use of the evidence. Time and teacher work habits are considerations primarily expressed by principals and local facilitators with regard to challenges. Because teachers in many of the small schools have multiple assignments, i.e., teaching, supervision, coaching, etc., and have up to 5 or 6 separate preparations, time to prepare for laboratory science is a challenge. While work habits are mentioned, it is not clear what that entails, but it appeared often enough to be recognized as an issue.

In looking at the duties of the administrative contacts with regard to RSI, they reported that their role/responsibilities in that capacity were as follows. (Note: Respondents were asked to indicated all that applied; therefore, the sum does not total to 100 percent.)

 

                  Administration and fiscal management         69.2 percent

                  Curriculum leader                                          70.5 percent

                  Overall change facilitator                               47.4 percent

                  Consultant and resource person                     46.2 percent

                  Public relations/spokesperson                        38.5 percent

                  Other                                                               3.8 percent

     Table 3. Administrative Contacts’ Perceptions of Initial Expectations and Current Experiences

 

       Notes: Missing responses were those that were either blank or not included because they were unusable, e.g., multiple instead of single

                    responses, etc. N = 79. Therefore, rows may not total to 100 percent.

 

Table 3A. Administrative Contacts’ Perceptions of Initial Expectations and Current Experiences

Notes:     Means and standard deviations are calculated only from responses to options “False,” “Partly True,” and “True.” Results from the initial expectations response category are designated by subscript “I”; results from the current experiences response category are designated by subscript “C.”

Table 3B. Administrative Contacts’ Perceptions of Impact on Selected Elements of the Instructional Program

 

Notes: Response options used for calculating the means and standard deviations were “Not at All” (1), “Slight

Extent” (2), “Moderate Extent” (3), and “Great Extent” (4). Data not included in this table were missing and multiple responses or responses that were unusable for one reason or another. Therefore, rows may not total 100 percent.

Table 4. Principals’ Perceptions of Initial Expectations and Current Experiences

 

Notes: Missing responses were those that were either blank or not included because they were unusable, e.g., multiple instead of single

            responses, etc. N = .45. Therefore, rows may not total to 100 percent.

 

 

Table 4A. Principals’ Perceptions of Initial Expectations and Current Experiences

Notes:    Means and standard deviations are calculated only from responses to options “False,” “Partly True,” and “True.” Results from the initial expectations response category are designated by subscript “I”; results from the current experiences response category are designated by subscript “C.”

Table 4B. Principals’ Perceptions of Impact on Selected Elements of the Instructional Program

 

Notes: Response options used for calculating the means and standard deviations were “Not at All” (1), “Slight Extent” (2),

           “Moderate Extent” (3), and “Great Extent” (4). Data not included in this table were missing responses and multiple

           or responses that were unusable for one reason or another. Therefore, rows may not total to 100 percent.

Table 5. Local Facilitators’ Perceptions of Initial Expectations and Current Experiences

 

Note: Missing responses were those that were either blank or not included because they were unusable, e.g., multiple instead of single

           responses, etc. N = 56

Table 5A. Local Facilitators’ Perceptions of Initial Expectations and Current Experiences

 

Notes:    Means and standard deviations are calculated only from responses to options “False,” “Partly True,” and “True.” Results from the initial expectations response category are designated by subscript “I”; results from the current experiences response category are designated by subscript “C.”

 

 

In the survey forms, a number of questions called for open-end responses. These responses are summarized in Table 6.

 

Table 5B. Local Facilitators’ Contacts’ Perceptions of Impact on Selected Elements of the Instructional Program

 

Notes: Response options used for calculating the means and standard deviations were “Not at All” (1), “Slight Extent” (2), “Moderate Extent” (3), and “Great Extent” (4). Data not included in this table were missing responses and multiple

            or responses that were unusable for one reason or another. Therefore, rows may not total to 100 percent.

 

 

 

Table 6. Summary of Open-End Responses to Critical Issues

 

 

The school principals estimated that they spent an average of 3.3 hours per week on RSI-related work and that they communicated with an RSI representative an average of 3.4 times per month. Regarding their involvement with RSI, they estimated the time they devote to RSI is of the following types and percentage.

 

                        Administration and fiscal management          25.7 percent

                        Curriculum leader                                           22.9 percent

                        Overall change facilitator                               25.7 percent

                        Consultant and resource person                      17.1 percent

                        Public relations/spokesperson                         5.7 percent

                        Other                                                              2.9 percent

 

 

 

Findings

 

A number of questions and issues emerged from the evaluative study of selected NSF-supported Rural Systemic Initiative collaboratives. This study, under the direction of Dr. Jerry G. Horn with The Evaluation Center at Western Michigan University, is directed more toward what is happening in the RSIs than as an assessment of individual RSIs or participating schools. The study was designed and is being conducted on the principle that rural communities and the schools that serve them are unique and that there is value in how these reform efforts, such as those undertaken by NSF and its collaborators, are perceived. To gather information that will help us and others (practitioners, administrators, and policymakers) understand these perceptions, a survey was designed with input from the principal investigators/project directors of the selected RSI collaboratives, i.e., Delta, Appalachia, Texas, Michigan, and Coastal. It was thought that the persons who would be the best informants on the questions and issues under consideration at this time would be the administrative contact for the RSI, a school principal, and the local facilitator. Also, the respondent pool (participating school districts) was selected by the RSI collaborative administrator(s) with instructions to select only those districts that had substantive involvement with RSI. While the term “local facilitator” is not universally used, it was operationally defined as the person selected to serve as “teacher partner,” “resource teacher,” or a person with similar responsibilities.

 

The results of this survey, conducted during 2002, are summarized and presented in this report. Copies of the survey instruments and other information pertaining to the administration of the survey are included in the appendices. In this section, we want to identify some of the important findings, as related to the purpose of this evaluative study. Every point or possible finding will not be addressed here. However, those who find use for this information for other purposes may find these data useful.

 

As stated earlier, the NSF identified six drivers as recognition of “Educational System Reform.”

In earlier reports [A Summary of RSI School Personnel’s Perceptions of the Drivers for Educational Systemic Reform (April 2001) and A Summary of RSI School Personnel’s Perceptions of the Drivers for Educational Systemic Reform–Part Two (September 2002)], it was concluded that there is strong support and recognition of the drivers and the indicators developed and validated by the Research Advisory Team for the WMU study. It was agreed that these are reflective of systemic reform and should be the primary focus of ongoing reform efforts. These findings are consistent in the two studies cited, which includes input from six RSI collaboratives (UCAN, Delta, Appalachia, Texas, Coastal, and Michigan). In that regard, the following section provides an analysis of the extent to which each driver is present in the schools represented by the respondents of the current survey.

 

 

Driver 1:        Implementation of comprehensive, standards-based curricula as represented in instructional practice, including student assessment, in every classroom, laboratory, and other learning experience provided through the system and its partners.

 

Status:             Clearly, all respondents expected that the curriculum would be aligned with recognized standards, and this expectation is perceived to be realized in practice. In terms of impact, there is some evidence that there have been more laboratory-oriented, hand-on activities and that these are related to the alignment of the curriculum with standards. While not specifically determined in this survey, most of the states represented in this study have also instituted statewide testing/-assessment programs that are intended to be reflective of the established standards and benchmarks recognized by that state. Whether the RSIs are responsible for the decision to align the curriculum is not a certainty, but it is clear that the RSIs have been instrumental and in many cases the vehicle for the alignment process to proceed in science and math. In some cases, this served as the model for alignment in other subject areas.

 

Driver 2:        Development of a coherent, consistent set of policies that supports: provision of high quality mathematics and science education for each student; excellent preparation, continuing education, and support for each mathematics and science teacher (including all elementary teachers); and administrative support for all persons who work to dramatically improve achievement among all students served by the system.

 

Status:             We do not have evidence that there existed or that there have been changes that resulted in policies pertaining to these issues. However, administrative contacts and principals did rate policy statements among some of the highest rated impacts. What we found is that schools generally follow the rules/policies set forth by their respective states with regard to the required curriculum and graduation requirements. In practice, there is evidence that more courses and more challenging opportunities for studying science and mathematics occurred in many locations. In a small school and especially in rural communities, practice is far more important than policies, and that seems to be a positive outcome or finding. School principals did give a high rating in terms of impact with regard to “Increased requirements and expectations of students in math/science.” In terms of professional development, the RSIs offered or facilitated the offering of a number of opportunities for teachers. The quality of these offerings or the appropriateness to the educational reform/improvement in science and math has not been determined. However, teachers, especially elementary teachers, speak well of these experiences; and the respondents in this study indicated that one of the important outcomes/impact has been improved teacher attitudes and work habits. Administrative support for the RSI effort is at least partially reflected in the ways in which the administrative contacts and school principals were involved, i.e., spending time on behalf of the RSI and indirectly as facilitating professional development efforts, appointing a local facilitator (often taking a recognized and highly successful teacher from the regular classroom for this assignment), etc.

 

Driver 3:        Convergence of the usage of all resources that are designed for or that reasonably could be used to support science and mathematics education—fiscal, intellectual, material, curricular, and extra-curricular—into a focused and unitary program to constantly upgrade, renew, and improve the educational program in mathematics and science for all students.

 

Status:             The results of this survey do not provide adequate information to make a judgment with regard to Driver 3.

 

Driver 4:        Broad-based support from parents, policymakers, institutions of higher education, business and industry, foundations, and other segments of the community for the goals and collective value of the program, based on rich presentations of the ideas behind the program, the evidence gathered about its successes and its failures, and critical discussions of its efforts.

 

Status:             In many respects, this expectation or area of hope has been a disappointment. While many respondents had this expectation, there has been little success in substantive involvement of parents and the community in this reform effort. While there are instances of what has been termed “successful” events—i.e., math or science night, a science fair, etc.—there is virtually no evidence that this driver is present in most of the communities or that it will ever be present using current strategies. In higher education, participation has been largely on the basis of individuals who were contracted to provide specific services: workshop presenter, resource person, etc. However, higher education (colleges and universities), which might be represented by math and science departments’ major involvement and commitment to the goals of RSIs, is not a part of current RSI projects or local reform efforts.

 

Driver 5:        Accumulation of a broad and deep array of evidence that the program is enhancing student achievement, through a set of indices that might include achievement test scores, higher level courses passed, college admission rates, college majors, Advanced Placement Tests taken, portfolio assessment, and ratings from summer employers, and that demonstrate that students are generally achieving at a significantly higher level in science and mathematics.

 

Status:             While this was identified as the second highest rated expectation by all three groups of respondents, none listed it among its highest realizations. Only among school principals did we find that “Improved student achievement scores” received a rating above 3.0 (Impact to Some Extent) on estimated impact, and the calculated value for the school principals group was only 3.1. Granted, there are more indicators of student achievement than test scores, but there are virtually no data to indicate that there is a “broad and deep array of evidence that the program is enhancing student achievement” or that there is an effort to establish this at the local school level. This is not to say that this is not a goal and even an accomplishment at the RSI collaborative level.

 

Driver 6:        Improvement in the achievement of all students, including those historically underserved.

 

Status:             In terms of the RSI projects, other reports of the WMU study have cited the homogeneous nature of most of the communities. Generally, they could be described as economically poor, not well educated, with an adult population of maybe 50 percent with high school diplomas or GEDs, few opportunities for employment and virtually none that require advanced studies in math or science, school populations of one racial type, and relatively low expectations for education and careers/employment outside the immediate area. By definition, the RSIs are supposed to serve poor, rural schools; therefore, the real underserved are the whole school population. However, there are subpopulations within the schools themselves, and the respondents reported that the “curriculum is more challenging for all students” and that this was an expectation, a realization, and an impact. Evidence was not obtained that would allow us to extend this interpretation to subgroups.

 

Following are other findings based on this survey:

 

■        Expectations did not exceed realizations. The exceptions to this are generally confined to the areas of greater expectations for monies from the project to be available at the local level and that parents and the community would be more involved.

 

■        Realizations that exceeded initial expectations are few, but to a substantial extent fall within the areas of leadership training and support and leadership from a local facilitator.

 

■        Alignment of science and math curricula with a recognized set of standards and the provision of a more challenging curriculum for all students are highly notable outcomes or areas of impact. (With the advent of the statewide standards movement and the “No Child Left Behind” program of the George W. Bush administration, the RSIs were timely forms of resources, expertise, and assistance to local schools in these projects.)

 

■        Although likely not the only factor in some cases, the RSI program seems to have had a broad impact on various elements of schools—the students, teachers, and the administration—including increasing the awareness of math/science-related careers, increased student interest in science fairs and other forms of competition, improved student access to laboratory/activity-oriented instruction, more challenging curricula, integration of technology into instruction, advocacy for math/science among all teachers, teacher ownership in the curriculum, policies that support higher quality education, leadership training, etc.

 

It might be useful to consider some of the following comments (edited quotes) as concluding comments about the RSIs from the perspective of the local schools, as reflected in this 2002 survey.

 

Our thinking is that there will be spill over to the grade levels above and below. As the supervisor and evaluator of these individuals, I can attest to the improved instruction and assessment. In addition, student achievement as measured by the teacher-created, on-going assessments, is much more meaningful. Teachers are using this information to drive their instruction. Our students do not take a standardized state assessment for math and science until the upper grades, so it will be a few years before we feel the full impact of the curriculum and instruction improvement.

 

I am convinced that increased student learning and student achievement in math and science impacts all areas of learning in a positive manner. Math and science activities are invitational and exciting. Puts the pressure on.

 

The RSI technical assistant has worked “seamlessly” with other support initiatives.

 

The RSI has had a profound effect on our school district. Given the isolated nature of both the town and its occupants, change has not occurred here in over 30 years. Teachers are resistant and untrustful of research, new instructional techniques, and benchmarks/standards. The RSI has provided the necessary groundwork for my staff to begin working as a team.

APPENDIX A

 

 

Lists of Contact Persons

 

 

ARSI Contact Person List

 

Coastal RSI Contact List

 

Delta RSI Contact Person List

 

MiRSI Contact Person List

 

Texas Contact Person List

 

 

 

APPENDIX B

 

Cover Letter to Contact Persons

 

September 30, 2002

 

 

TO:                   School District Contact Person for [Collaborative Project Title] RSI Project

 

FROM:            Jerry Horn, Principal Research Associate

The Evaluation Center, Western Michigan University

 

RE:                   SURVEY

 

 

As a part of a continuing contract with the National Science Foundation to conduct a study of the Rural Systemic Initiatives Program, we are asking your assistance in completing one of the enclosed surveys (labeled RSI Contact Person) and distributing one (labeled Building Principal for RSI School) of the other two packets to a principal of a building/school that has been actively involved in the RSI program and one (labeled Local RSI Facilitator) to the person with responsibilities as a local facilitator, teacher partner, or to the person whom you consider to be in a similar role or has knowledge of the program and past participation in program activities in your school district. (Since the same survey is being sent to persons in five different RSI projects, there is not a common title given to the person that we have designated as “local facilitator,” but I ask that you consider the roles of those involved in the RSI activities in your school district and distribute it to a person who has good knowledge and experience with the program across the schools in your district.) In some cases, a person might have responsibilities that correspond to more than one of the three persons designated to receive a survey form, e.g., a building principal might also serve as the contact person for the district. In those cases, distribute both packets to that individual.

 

It is important for you to know that this survey/study is being conducted in cooperation with and with support of the [collaborative project title] RSI and that we are not evaluating your school/district or the RSI. Rather, our longitudinal study is designed to gather information that will help us and the National Science Foundation understand the context, needs, and approaches for the improvement of mathematics and science across all of the RSIs. Thus, some of the questions/issues may not have been a part of a specific RSI’s plan, and its appearance on the survey should not be interpreted as an expectation for the past or the future or an inferred assessment of that RSI’s effectiveness.

 

Each of the enclosed packets contains a brief introduction, a survey form, and a self-addressed, postage- paid envelope. By asking for perceptions or understandings, we are not asking respondents to gather information or to spend an inordinate amount of time in completing the survey. However, the information that we gather from these surveys is important in our attempts to document various approaches and understandings across all of the school districts participating in an RSI program.

 

Thank you for your cooperation in completing the “RSI Contact Person” survey and distributing the appropriate packets to a building principal and a local facilitator. If you have questions, please do not hesitate to call me at (405) 707-7143 or Gloria Tressler at (269) 387-5895.

 

 

enc

 

cc [PI of corresponding RSI collaborative project]