An Update of the Rural Systemic Initiatives Evaluation Study by The Evaluation Center at Western Michigan University

Vol. 2, No. 1 September 2001

Project Update and Study Objectives

Focusing on three operating Rural Sys-temic Initiatives collaboratives (Appalachian RSI, Delta RSI, and UCAN RSI), the original objectives of the study were

To develop a system of indicators around each of the identified six drivers of educational system reform
To determine the perceived relative importance and value of each of the drivers and indicators for reform in RSI schools in selected communities
To determine the status of innovation and reform within selected communities with respect to factors thought to support or serve as barriers to innovation and education reform
To determine the ways and the extent to which the perceived importance and value of the drivers and the characteristics of the community impact on systemic reform efforts and student achievement in mathematics, science, and technology.

In Spring 2000, we were asked to expand the study to include three newly funded RSIs (Texas RSI, Coastal RSI, and Michigan RSI) and to investigate the following questions. The target completion date for the project is May 31, 2003.

What new or different forms of student assessment and teacher effectiveness have been de-veloped and used as a result of curriculum transformation and alignment with state or national standards in science and mathe-matics, and how were these developed and used?

What contextual factors (within and across collaboratives) serve to support reforms that result in or are associated with the development and implementation of standards-based curricula?
What processes and conditions are essential for effective partner-ships within large-scale collaboratives that are characterized by large geographic distances between entities and multiple/potentially contrasting governance arrangements?

What considerations have been given to the use of technology for enhancing the accessibility and effectiveness of math and science instruction and communication within and among collaborative members?

We have approached this project as a study of the Rural Systemic Initiative from a community and local school perspective. While we undoubtedly make informal merit and worth judgments about certain practices within and across the six assigned RSIs, we are most interested in describing and understanding how unique and common problems are addressed, the contextual factors that serve as barriers and forms of support for education reform, and the various forms of impact that the program has on schools and the rural communities that they serve.

Several hundred school districts and 15 states are represented in the six RSIs. Further, the RSIs have been operational for varying periods of time, i.e., from less than one year to more than five years. Each state has different student assessment programs and operational procedures/policies for K-12 schools. While these conditions present some interesting and difficult issues for our study, they also demonstrate that elementary and secondary education is the responsibility of the states and that federal intervention, such as the RSI program of the National Science Foundation, must be compatible with state effort for education reform.

Further, while the com-munities are gen-erally rural and poor, there are current and historical conditions that make each one unique. It would be impossible for us to conduct an in-depth study of each school/community. However, we have chosen to conduct case studies in 15 locations. Six case studies have been completed, and the reports are available through the project web site at Western Michigan University <www.wmich.edu/evalctr/rsi>.

Additional project reports and substudies are ongoing, and documents related to these efforts can also be accessed at the web site.

Resource Advisory Team (RAT)

Because of the com- plexity of the schools/ communi- ties and the RSIs, the project is guided by the advice and assistance of the Resource Advisory Team (RAT). This is a group of ten nationally known pro-fessionals, and each member was selected on the basis of his/her specific knowledge and experience. The team assists the project staff by reviewing plans and making suggestions for improvement, interpreting both qualitative and quantitative data in collaboration with project researchers, and serving on site visitation teams to add expertise and credibility to this ongoing process. The current members of the Research Advisory Team are listed below:

Steve Oliver: University of Georgia- science education

Mary Harris: University of North Texas-education reform and standards/curriculum

Gene Hall: University of Nevada, Las Vegas-science education and education reform
Robert (Bob) Stephens: Institute for Regional and Rural Studies in Education (OK)-rural education and regional service centers

James (Jim) Jess: Formerly CAL (IA) Community Schools-rural education, school administration, and education reform

William (Bill) Webster: Dallas (TX) Independent School Dis-trict-testing/achievement assessment

Daniel (Dan) Stufflebeam: Western Michigan Univer-sity-standards/curricula and testing/achievement assessment

Paul Nachtigal: Formerly Annen-berg Rural Challenge (CO) project-rural education and education reform

Jack Sanders: SERVE (Greens-boro, NC)-rural education and education reform

Larry Enochs: Oregon State Uni-versity-science education, standards/curriculum, and educa-tion reform

We are saddened by the untimely and unexpected death of one of the original members of the Research Advisory Team, Dr. Joe Newlin, Executive Director, National Rural Education Association. Joe provided a unique perspective for the project and gave it national recognition through the NREA. We will miss his candor and expert advice.

Project Staff Changes

Dr. Jerry Horn, Principal Research Associate at The Evaluation Center at Western Michigan University, continues to serve as Principal Investigator and Project Director. Dr. Craig Russon, who served as Project Manager, resigned from The Evaluation Center in November 2000. He is currently on the evaluation staff of the W. K. Kellogg Foundation in Battle Creek, Michigan.

Rather than search for another person to replace Craig, we chose to increase the use of Evaluation Center staff members (Dale Farland, Mary Ramlow, Sally Veeder, Christine Hummel, Maxine Robb, and Brian Carnell) and project consultants, Dr. Brian Lotven and Dr. Kenneth McKinley. Dr. Lotven will be coordinating the longitudinal case studies in the Coastal RSI and assisting Jerry Horn with the Michigan RSI. Dr. McKinley will be coordinating the case study activities in Texas.

Dr. Louisa Stark, Director of Science Education for the Genetic Science Learning Center at the University of Utah, continues to work with the project on specific substudies, particularly those related to student achievement and various forms of student impact.

In total, we have adopted a unique staffing model for the study, one that is flexible enough to meet the planned evolution of the procedures and call on specific expertise as appropriate and needed. Additional assistance will be sought from members of the Research Advisory Team or other professional groups/individuals over the next year of the project.

Case Studies

The first set of case studies has been com- pleted for the original RSIs assigned to the study (ARSI, Delta RSI, and UCAN RSI). The study sites were Rockcastle County (KY), Cocke County (TN), Humphreys County (MS), East Feliciana Parish (LA), Wagon Mound Community (NM), and the Gila River Indian Community (AZ). The reports of these studies, based on 3-5 day visits by staff and RAT members, are available on the web site or from The Evaluation Center at WMU.

Since the newly assigned RSIs (Coastal, Texas, and Michigan) have been operating for only a short period of time (2 years or less), we have chosen to conduct longitudinal case studies at three sites in each RSI. A study team of 2-3 persons will make two visits to each site during the period of May 2001-December 2002. The following sites were chosen for this phase of the project:

Texas RSI-Pittsburg, Carrizo Springs, and Clarendon
Coastal RSI-Charles City, Virginia; Marion, South Carolina; and Elizabeth City, North Carolina
Michigan RSI-(tentative) Nah-Ta-Wahsh Public School Academy, Baldwin, and Whittemore-Prescott

We expect the longitudinal case studies to be as rich and informative as the first six, and we expect to be able to describe the ongoing reform process in more detail.

Observations and Tentative Findings

While the RSI program is generally directed at rural and

poor communities/ schools, there are many differences from site to site, but some consistent factors are beginning to emerge as we continue to study the data. It is difficult to separate the effects of socioeconomic status from external interventions or reform efforts such as the RSI. Some of the initial findings that seem to impact or interact with efforts to improve science and math education in these communities are

lack of local employment that encourages stronger backgrounds in math or science

lack of qualified teachers and/or high teacher turnover

low value for education, based on experience and achievement of many parents

instruction driven by state-mandated/high stakes testing programs

disconnect between stated values and local commitments to quality education for all students

Overall, we see that the schools are heavily engaged in aligning their curricula with state guidelines and standards. The RSI projects have provided expertise, resources, and other forms of direct assistance to schools for this effort. Further, we see spin-off effects into other areas.

From survey data, respondents (teachers, administrators, and other stakeholders) from the original case study sites "agree" that the following are valid indicators of systemic educational reform: (1) adminis-trative support for all persons who work to dramatically improve achievement among all students; (2) implementation of comprehensive, standards-based curricula as represented in instructional practice; and (3) development of a coherent, consistent set of policies that supports provisions of high-quality math and science for each student. At the same time, they place the locus of control with the teacher for "implementation of a comprehensive, standards-based curricula" and "the improvement in the achievement of all students, including those historically underserved." While the district level administration or local school board is primarily perceived as responsible for developing "broad-based support from parents, policymakers, institutions . . . for the goals and collective value of the program."

Representatives of participating RSI school districts indicate that some of the most important factors that facilitate education reform are (1) state standards, (2) district administration and principals, (3) technology and science and math materials, (4) teacher preparation and knowledge of subject matter, (5) parent and community involvement, and (6) high expectations for students.

Barriers to education reform w ere perceived to be (1) (lack of) laboratory equipment, (2) teacher turnover, (3) low community support, (4) lack of money and other resources, and (5) poorly prepared teachers who lack subject knowledge.

With RSI involvement, the areas where the most significant change (improvement) occurred are instruction, curriculum, communication, and involvement.

The RSI activity that most contributed to these changes is professional development and training, with resources being placed much further down the list. (However, this is likely to be an indication of where the RSIs have put their efforts and resources.)

Follow-up efforts to identify alternative forms of student assessment and related indicators of achievement produced the following list.

rubrics and scoring guides
open-ended response questions
projects, products, and presentations
performance assessment
journals and learning logs
daily observations of individual students
checklists

portfolios
questioning techniques

A consistent finding may be summarized in the following statement.

The need to improve student scores on the state-mandated high stakes tests is the primary motivation for changes in the curriculum and assessment practices. Schools and teachers have aligned their curricula at each grade level to the state standards so that students will be taught the materials on which they will be tested. The types of questions included on the state test have a strong influence on the types of assessments teachers use in their classrooms.

At the same time, there has been little change in the formal teacher evaluation procedures. But high stakes testing often has built-in rewards and punishments; therefore, teachers and school administrators think they cannot ignore the need for improved student scores on the state-mandated, high stakes tests.

As expected, there was a large range of environmental, societal, and educational factors across these communities. Yet they are clear examples of the diversity in rural America and a reflection of the dedication of professional educators to reform and improve the educational opportunities for their students in science and math. At the same time, we observed some of the limitations, barriers, and problems that the schools and communities have faced in the past and many that they are addressing through the RSI program.

Drafts of the case study reports were submitted to the designated contact person in each community for feedback. Copies of the reports were submitted as a part of the annual report to NSF, and they are available for distribution upon request. We think the case studies will prove to be valuable pieces of information to help inform others about rural communities, including the historical development of the area and the educational system. Also, the case study reports include a summary of each case study team's assessment of the extent to which the drivers have been implemented or observable in each community.

Communications and Distribution of Initial Findings

Efforts are ongoing to make the progress of the project and the various reports related to some of its major activities available to interested parties. The project director made presentations at two national meetings in 2000, and reports are available on the project web site. A panel presentation on the project by Horn, Lotven, McKinley, and Stark is scheduled at the National Rural Education Association's October 2001 annual meet-ing in Albuquerque, New Mexico. Work/ study sessions of the Research Advisory Committee are one important way of sharing the information that we have gained. One such meeting was held in Kalamazoo, Michigan, in July 2001 and another is scheduled for October 2001, prior to the NREA meeting in Albuquerque.

For information or questions about the Rural Systemic Initiatives Evaluation Study, contact:

Dr. Jerry G. Horn
RSI Evaluation Study
The Evaluation Center
Western Michigan University
Kalamazoo, MI 49008-5237

(405) 707-7143 / (405) 707-7148 (fax)

email: jhorneval@aol.com