RSI SURVEY REPORT

Prepared

for

The NSF Rural Systemic Initiatives Evaluation Study

by

Craig Russon, Louisa Stark, & Jerry Horn

The Evaluation Center

Western Michigan University

Kalamazoo, Michigan 49008-5178

April 2000

Rural Systemic Initiatives Survey

In 1994, funding of Rural Systemic Initiatives (RSI), the third in a set of systemic reform initiatives sponsored by the National Science Foundation (NSF), was offered on a competitive basis. The RSIs were created to enhance math, science, and technology education for all students and focus on economically disadvantaged, rural areas through instructional and policy reform and community development activities. RSIs are expected to address policy, leadership, and work force issues related to educational barriers by engaging community involvement in creating a comprehensive and sustainable system of math, science, and technology education that reflects current advancements in these areas. Instruction and performance standards are expected to play a major role for districts involved in these programs.

In August 1998, NSF contracted with The Evaluation Center to evaluate the RSI program. In part, the study was designed to investigate the relationship among the potentially conflicting values in traditional rural communities with the current understanding of education reform and the goals of NSF-sponsored Rural Systemic Initiatives, including the six drivers of educational system reform. More importantly, the Center focused on the impact of these relationships on student learning in science and mathematics within selected RSI projects. The RSIs to be included in the study were the Delta RSI (Mississippi, Arkansas, and Louisiana), the Appalachian RSI (Ohio, West Virginia, Virginia, Kentucky, Tennessee, North Carolina) and the UCAN RSI (Utah, Colorado, Arizona, and Nevada).

In order to achieve its objectives, a mixed method approach was used for the study. Using the User-Friendly Handbook for Mixed Method Evaluations distributed by NSF as a basic set of guidelines, broad-based sets of sets of quantitative and qualitative data were be collected and analyzed. One of the data sets was from a survey of stakeholders in school districts participating in RSIs being studied by The Evaluation Center. The purpose of the survey was to help evaluators identify local conditions that might influence the extent to which the "drivers of educational reform" had been implemented in the district.

In April 2000, a paper and pencil questionnaire was sent to a matrix sample of stakeholders (the designated contact persons for each school district participating in the RSIs). The first five pages of the questionnaire were identical. However, on the last page, each stakeholder received one of three different subsets of questions. After about a month, a second questionnaire was sent to nonrespondents. The final response rates for the RSIs being studied were Delta=31 percent; UCAN=33 percent; ARSI=51 percent. Data were analyzed using quantitative and qualitative methods. The findings from the survey are found below as summarized data and an interpretation of the response for each question.

Question 1--Circle the RSI in which your district participated

Even though ARSI had the highest response rate (51 percent), the respondents from UCAN schools make up the greatest percentage of the total responses (38.8 percent). Delta had the lowest response rate (31 percent) and the lowest percentage of total responses (26.9 percent).

Table 1. Frequency and Percentage of Respondents to the Survey

Question 2--List the names and levels of schools in your district that have participated in some substantial way with this RSI.

Elementary school participation in the RSIs was higher than high school participation, while middle school/junior high participation varied. In all three RSIs, twice as many elementary schools as high schools were reported to have substantial involvement with the RSI (see Table 2). Involvement at the middle school/junior high level was similar to the high school level for ARSI, similar to the elementary level for Delta, and intermediate for UCAN.

Table 2. Number of Participating Schools in Each RSI at Three Educational Levels

These differences in the number of schools involved at each level could be due to the fact that districts frequently have more elementary schools than middle schools/junior highs and more middle level schools than high schools. However, in each RSI a higher percentage of districts reported participation at the elementary level than at the high school level (see Table 3). Middle school participation varied in relation to the lower and upper levels among the three projects.

Table 3. Percentage of Districts in Each RSI Reporting Participation at Three Educational Levels

Question 3--School district information

The data in Table 4 reveal that there is a substantial amount of variation on key characteristics among rural districts participating in RSI. The standard deviations for many of the district characteristics are very high--in some cases as high as the mean. This finding shows that there is no such thing as a "typical" rural district.

Table 4. District Characteristics

Question 4--Which of the following RSI activities or services has your school district participated in or received over the past 5 years? (Check all that apply)

Examination of the data in Table 5 suggests two different strategies employed by RSIs for providing services to school districts. ARSI apparently tried to provide a very broad menu of services to everyone. UCAN and Delta appear to provide core services, such as group/individual professional development and on-site consultation, to everyone and supplement with other services, perhaps tailored to the needs of individual districts.

Table 5. RSI Activities or Services Over Past Five Years

Note: * indicates percent of cases.

The most effective RSI activities reported by districts focused on professional development for teachers, curriculum changes, increased resources, and increased communication and involvement (see Table 6).

Table 6. Most Effective or Exemplary RSI-Sponsored or RSI-Initiated Activities

Professional development was listed most frequently as a particularly effective or exemplary RSI-sponsored activity, with 67 percent of the districts reporting one or more items in this category (see Tables 6 and 7). The Teacher Partner program received high praise from 39 percent of the districts in ARSI. These teachers not only participated in regular professional development meetings themselves, but also trained other teachers in their schools. Several ARSI districts also mentioned professional development focused on inquiry. Districts in UCAN reported professional development activities focused on interpretation of standards, aligning curriculum with standards, assessment, instructional skills and techniques, training to use new curriculum, and leadership development as most effective. One third of the Delta respondents listed training to use new curriculum as the most effective RSI activity. Several respondents also mentioned training in instructional skills and techniques and training for administrators. More than 25 percent of the respondents in UCAN and ARSI, as well as several Delta districts, listed professional development for teachers, with no topics specified, as the most effective RSI-sponsored activity. While most professional development was directed toward teachers, activities also were provided for administrators, counselors, and leadership teams.

Table 7. Professional Development Activities Listed as Effective by Two or More Respondents

RSI activities related to curriculum changes were also listed as particularly effective by many districts (see Tables 6 and 8). Several districts in Delta and UCAN listed assistance aligning their curriculum with standards and with selecting, developing, and implementing new curriculum materials as helpful RSI activities. Almost a third of ARSI districts reported assistance with curriculum audits as particularly useful.

Table 8. Curriculum-Related Activities Listed as Effective by Two or More Respondents

Table 9. Resources Listed as Effective by Two or More Respondents

Table 10. Increased Communication and Community Involvement as a Result of RSI Activities

Question 6--What is the greatest benefit your school district has gained from involvement with the RSI?

The greatest benefits reported by districts focused on professional development for teachers, increased resources, curriculum changes, and increased communication and networking (see Table 11).

Table 11. Greatest Benefit to School District From RSI Involvement

Table 12. Targets of Professional Development Activities

Increased resources were listed by 42 percent of the respondents as a benefit of RSI involvement (see Tables 11 and 13). In addition to unspecified resources, these resources included information, funding, consultants and contact with experts, as well as equipment, kits, and science/math materials.

Table 13. Resources Listed as a Benefit of RSI Involvement

Curriculum changes were listed as a benefit of RSI involvement by 31 percent of the districts (see Tables 11 and 14). These changes included curriculum review and alignment with standards, curriculum selection and implementation, increased use of standards, and the development of a standards-based curriculum.

Table 14. Curriculum Changes Listed as a Benefit of RSI Involvement

For 27 percent of the respondents, RSI involvement led to increased involvement with others outside the school (see Tables 11 and 15). This included increased communication and networking with colleagues, increased communication with community groups such as tribes, and increased family involvement.

Table 15. Increased Involvement with Others Outside the School as a Benefit of RSI Involvement

In addition to the benefits of RSI involvement listed by respondents in all three RSIs, several benefits were listed primarily by districts in ARSI. These included an increased focus on science and math and an increased motivation to pursue a standards-based curriculum (30 percent), improved instruction (26 percent), and the Teacher Partners (22 percent).

Question 7--What factors have facilitated or been a barrier to education reform in your school district? (Check all that apply).

Table 16 contains data on facilitating factors. The most important factor that was found to facilitate education reform across all three RSIs was administrative support from school boards, district administration, and school principals. Another factor that was found to facilitate education reform was state education standards.

Table 16. Factors that Facilitated Education Reform

Note: * indicates percent of cases

Table 17 contains data on barriers to education reform. The factors that were found to be barriers to education reform were money and lab equipment. The barriers were of lower magnitude than the facilitating factors. For example, in Table 17, the highest frequency was 45 (68.2 percent). In Table 16, the highest frequency was 59 (88.1 percent). School consolidations and school boards were not seen as barriers to education reform.

Table 17. Factors That Were a Barrier to Education Reform

Note: * indicates percent of cases

Question 8--What effect has participation in the RSI (UCAN, ARSI, or Delta) had in your school district on the following indicators related to the "drivers of educational reform?"

On question 8, each respondent received one of three different subsets of items. The scale of the Likert-like items was 1= unaffected; 2=somewhat increased; 3 positively increased; and 4=greatly increased. The aggregated descriptive statistics are found in Table 18. Respondents said RSI participation most influenced math assessment alignment with standards (mean=3.58); changing of math activities to improve curriculum (3.52); and lowest ability students' achievements in science and math (3.50). RSI least influenced the percentage of students taking Advanced Placement college examinations (mean=1.67); low income parents' support of their children's learning in science and math (1.80); and minority students' scores on science and math standardized tests (1.85). Because of the matrix sampling, the Ns for each item were lower than they would have been otherwise. Also, when variables are considered on a listwise basis, there are none available for analysis. This precluded the use of more advanced statistical analysis.

Table 18. Result of RSI Participation

Question 9--Which school in your district has had the greatest involvement with RSI?

In the three RSIs combined, schools at the elementary and middle school/junior high levels were cited as having the highest RSI involvement (48 percent, Table 19). Elementary schools had a slightly higher involvement in UCAN and ARSI, while there was a higher involvement at the middle school/junior high level in Delta. In all three RSIs a lower percentage of high schools (36 percent) were listed as the most involved schools. These data correlate with those from Question 1, which show a higher percentage of elementary and middle school/junior high schools involved in each RSI than at the high school level.

Table 19. Grade Level of the School with Highest RSI Involvement in Each District

Question 10--What have been the most important factors contributing to this school's involvement with the RSI?

At the schools in the district with the highest RSI participation, the most important factors contributing to involvement were support by the administration, direct contact, interest, professional development, and resources (see Table 20).

Different factors were most often cited as important by districts in each of the three RSIs. Support and interest by the school's principal or district were cited as important to high RSI involvement by 28 percent of the respondents, but was particularly important in ARSI schools (39 percent). Direct contact by a colleague in the school, particularly a Teacher Partner, was listed as important to high RSI involvement by 70 percent of the ARSI districts, as well as several other districts. Teacher interest and willingness to participate, as well as a positive attitude toward reform, were cited as important to RSI involvement by respondents in all three RSIs (22 percent), but were cited as important in almost a third of UCAN districts. The same number of UCAN districts also listed teacher professional development as important to RSI involvement. Respondents particularly mentioned professional development that was appropriate for the individual and the grade level. Resources such as money, RSI staff, consultant services, and materials were cited as contributing to high RSI involvement in 16 percent of the districts. Other factors cited by at least two districts were perceived need, team effort, teacher stability, networking, and collaboration among schools and family involvement.

Table 20: Factors Important to High RSI Involvement

Question 11--What are the most significant changes in this school [school most involved in the RSI] as a result of RSI involvement?

In all three RSIs, respondents reported that improved instruction was the most significant change (49 percent) that resulted from RSI involvement (Table 21). These instructional changes included more use of inquiry, hands-on activities, and activity-based instruction as well as changes in assessment practices. Curriculum changes were also cited as important (24 percent) by respondents in each RSI. These changes included curriculum alignment to standards, implementation of a math/science program, and a greater use of math/science equipment and kits. Increased communication and involvement were listed as significant changes by 21 percent of the respondents. This included increased family participation, community awareness of math/science, and communication with the community, as well as increased teacher collaboration and networking with other schools. Training and education for school staff were listed as important changes by 19 percent of the respondents. Support for professional development increased, as did the quality of the programs. Several districts in each RSI reported improved student achievement as a significant change in those schools most involved with the RSI (10 percent).

Table 21: Areas of Significant Change in School with Greatest RSI Involvement

Question 12--Which RSI activities or services have contributed most to these changes [in the school most involved with the RSI]?

RSI activities or services that were reported as having contributed most to changes in schools with high RSI involvement included professional development, RSI staff and consultants, curriculum-oriented activities, and resources (see Table 22).

Respondents overwhelmingly (58 percent) cited professional development and training as the RSI activity or service that contributed most to the changes in schools with the highest RSI involvement (see Table 22). This included professional development activities for teachers, Leadership Institutes, team training, and professional development for principals. Having a Teacher Partner in the school was listed by 65 percent of the ARSI districts as contributing to the highest level of school involvement. Since the Teacher Partners both received and delivered professional development, they can be included in the professional development category. If one analyzes the responses in this way, 87 percent of the ARSI districts listed professional development as important to school changes, and 67 percent of the overall responses can be grouped in this category.

Several other RSI activities and services were also cited as contributing to significant changes in the most involved schools. RSI staff and consultants were listed by 15 percent of the respondents, but were seen as most helpful in ARSI (22 percent). Curriculum-oriented activities (13 percent) included needs assessments; program reviews; curriculum changes; and curriculum development, selection, and alignment. Resources cited as contributing to changes included funding to purchase equipment and materials, funding and grants in general, and curriculum materials. Collaboration and teamwork were listed as contributing to change in 10 percent of districts and included collaboration and networking with other districts, the resource collaborative and the UCAN team.

Table 22: RSI activities or services contributing most to significant changes in schools with greatest RSI involvement

Question 13--Which school in your district has had the least involvement with the RSI?

While elementary schools were listed as having among the highest RSI involvement (see Question 9 and Table 19), they were also the level most frequently listed as having the lowest involvement overall (36 percent, Table 23). The responses from UCAN lower the combined mean. This probably has something to do with the large number of schools widely dispersed over a four state area and the resources available to involve all the schools. In the individual RSIs, elementary schools were listed as having the lowest involvement in Delta and ARSI, while high schools were more frequently listed in UCAN.

Note: Percentages at each level were calculated as percentage of respondents in each RSI listing a school at that level. Many districts listed schools at more than one level; therefore, the percentages do not always equal 100 percent.

Question 14--What factors have contributed to this school's [school least involved in the RSI] lower involvement with the RSI?

Factors most often cited as contributing to lower RSI involvement included lack of interest or support, lack of emphasis at that level or school, and lack of time (see Table 24).

In all three RSIs, lack of interest or support was cited most frequently (30 percent) as the factor contributing to lower participation by the school(s) least involved in the RSI. Lack of teacher interest and lack of administrative support were the most frequently listed items in this category. Other factors included teachers resisting change, teachers teaching out of content area, lack of parental involvement, Teacher Partner not allowed to leave the school for meetings, no school representative on the RSI Leadership Team, and lack of coordination between the RSI and the school district.

The next most frequently cited reason for low involvement was a lack of emphasis at that school's level or at that school (16 percent). Other factors were that no Teacher Partner was assigned at this level and a decision that this school needed less support.

A lack of time was cited by 12 percent of the respondents overall, but was listed by 23 percent of the UCAN districts as a factor leading to lower involvement. This category included factors listed as less staff at the school, teachers tied up with after-school activities, and staff with multiple responsibilities.

Although the districts in the three RSIs are located in rural areas, only 8 percent of the respondents listed geographic distance or isolation as a factor leading to lower RSI involvement.

Table 24: Factors Contributing to Lower Involvement in Schools Least Involved with the RSI

Elementary schools have the highest level of participation in the RSI. This finding was corroborated through case studies that were conducted as part of the overall study.

Examination of the data suggests that there are two different strategies employed by RSIs for providing services to school districts. ARSI apparently tried to provide a very broad menu of services to everyone. UCAN and Delta RSIs appear to provide core services, such as group/individual professional development and on-site consultation, to everyone and supplement with other services, perhaps tailored to the needs of individual districts.

Professional development, particularly for teachers, was an important part of all three RSI projects. It was listed as the most effective RSI activity, as the greatest benefit districts gained from RSI participation, and as the RSI activity that contributed most to significant changes in schools with the greatest RSI involvement.

Curriculum-related activities were also reported to be among the most effective RSI activities. Curriculum changes as a result of these activities were seen as a great benefit in UCAN and ARSI. However, only in ARSI were curriculum changes perceived to be among the most significant changes in the schools with high RSI involvement. Professional development was cited as an important activity that led to curriculum changes.

A factor that facilitated education reform across all three RSIs was administrative support in the form of school boards, district administration, and school principals. Another factor that facilitated education reform was state education standards. The factors that were found to be barriers to education reform were money, lab equipment, and expectations for students.

Respondents said RSI participation most influenced math assessment alignment with standards (mean=3.12); emphasis on strengthening science, math, and technology (3.08); and curriculum implementation (3.04). RSI least influenced teacher/student ratios (mean=1.43); minority test scores (1.47); and the percentage of students taking advanced placement courses.

Resources such as equipment/materials, consultants, and funding were another benefit of RSI involvement. They were listed as some of the most effective services by Delta and ARSI districts. Professional development was an important component of learning to use new equipment and materials.

While most RSI activities were not directed toward administrators, administrative support was important for high RSI involvement in Delta and ARSI. Direct contact by Teacher Partners was very effective in increasing school involvement in ARSI. In UCAN, teacher interest and professional development activities were cited as factors leading to high RSI involvement.