A Report on the Identification and

Validation of Indicators of Six Drivers

for Educational Systemic Reform

By

Craig Russon and Jerry Horn

The Evaluation Center

Western Michigan University

Kalamazoo, MI 48008-5178

For

The Rural Systemic Initiatives Evaluation Study

(A project funded by the National Science Foundation)

Executive Summary

The National Science Foundation funded a proposal submitted by Western Michigan University's Evaluation Center to study science and math education systemic reform in rural communities. The study approaches the challenge from the unique perspectives of rural communities. The project will help math and science educators build upon the inherent strengths of rural schools and better understand the factors inherent in rural schools and communities that support or serve as barriers to systemic reform.

The 123 indicators that were identified during the literature review formed the basis of a questionnaire (Appendix 2). The questionnaire was administered to members of the Resource Advisory Team (Appendix 3). The team was instructed to read the description of each indicator and decide to which of the six drivers of educational system reform the indicator most strongly related. If the indicator was not found to correspond to any of the drivers they were to mark "Not Applicable." The results of the first round of the Delphi process showed that 29 indicators were validated (80 percent agreement) by the Resource Advisory Team: 4 for Driver 1, 10 for Driver 2, 3 for Driver 3, 5 for Driver 4, 5 for Driver 5, and 2 for Driver 6 (Appendix 4).

The results of the first round of the Delphi process were compiled along with comments. These were disseminated to the Resource Advisory Team along with a clean copy of the questionnaire. The team members were asked to review their colleagues' responses and complete the questionnaire again, modifying their responses as they deemed appropriate. The results of the second round of the Delphi process show that 75 indicators were validated (approximately 80 percent agreement) by the Resource Advisory Team: 19 for Driver 1, 23 for Driver 2, 10 for Driver 3, 7 for Driver 4, 13 for Driver 5, and 3 for Driver 6.

Introduction

The National Science Foundation funded a proposal submitted by Western Michigan University's Evaluation Center to study science and math education systemic reform in rural communities. The study approaches the challenge from the unique perspectives of rural communities. The project will help math and science educators build upon the inherent strengths of rural schools and better understand the factors inherent in rural schools and communities that support or serve as barriers to systemic reform. The original objectives of the study are listed below:

• To develop a system of indicators around each of the identified six drivers of educational system reform



• To determine the perceived relative importance and value of each of the drivers and indicators for reform in RSI schools in selected communities



• To determine the status of innovation/reform within selected communities with respect to factors thought to support or serve as barriers to innovation and education reform



• To determine the ways and the extent to which the perceived importance and value of the drivers and the characteristics of the community impact on systemic reform efforts and student achievement in mathematics, science, and technology

Phase I of the project was designed to accomplish the first objective of the study. The tasks contained in this phase include the following:

Task I.A. Review the literature for descriptions of evidence that may be reasonably determined to be related to each driver.

Task I.C. Analyze the results of the Delphi procedure and develop a list of validated indicators for each driver.

This report describes of the literature search and reports the results of Rounds 1 and 2 of the Delphi procedure.

Literature Review

An extensive review was conducted of the systemic reform, evaluation of systemic reform, and rural education literature (Appendix 1). The purpose of the literature review was twofold: (1) to identify indicators for consideration in the Delphi procedure and (2) to serve as the basis for a background paper on rural systemic reform (Russon & Horn, 1999). From these sources, and with input from the project's Resource Advisory Team, 172 indicators were identified. The list of indicators was reviewed and duplicates were omitted. The final list contained 123 indicators.

Round 1 of the Delphi Procedure

The 123 indicators that were identified during the literature review formed the basis of a questionnaire (Appendix 2). The questionnaire was administered to members of the Resource Advisory Team (Appendix 3). The team was instructed to read the description of each indicator and decide to which of the six drivers of educational system reform the indicator most strongly related. If the indicator was found not to correspond to any of the drivers, they were to mark "Not Applicable."

Only nine of the ten team members returned the questionnaire during the first round of the Delphi. The results were somewhat difficult to interpret because some team members marked more than one answer. Twenty-nine indicators were validated by 80 percent or more of the team members (Appendix 4):

Driver 1. Implementation of comprehensive, standards-based curricula as represented in instructional practice, including student assessment, in every classroom, laboratory, and other learning experience provided through the system and its partners.

• Curriculum is of high quality and aligned with national standards.
• Teachers reference standards in selecting curriculum.
• Teachers report use of recognized standards.
• Assessments are aligned with standards and curriculum

Driver 2. Development of a coherent, consistent set of policies that supports provision of high quality mathematics and science education for each student, excellent preparation, continuing education, support for each mathematics and science teacher, and administrative support for all persons who work to dramatically improve achievement among students served by the system.

• Professional development program in place to train teachers to implement high quality curriculum.
• Teachers and staff participate in professional development program.
• Policies indicate a coherent vision that encompasses all students.
• Policies strengthen the emphasis on mathematics, science, and technology.
• Policies support the preservice education of teachers.
• Policies assure adequate time for the ongoing professional development of teachers.
• Policies require a tight alignment among curriculum, instruction, assessment, and professional development.
• Professional development requirements have been changed.
• Professional development focuses on program and student needs.
• School policy is effective for guiding practice.

Driver 3. Convergence of the usage of all resources that are designed for or that reasonably could be used to support science and mathematics education--fiscal, intellectual, material, curricular, and extracurricular--into a focused and unitary program to constantly upgrade, review, and improve the educational program in mathematics and science for all students.

• Existing funding sources support systemic reform.

• Technology and telecommunications are used to support education.

• Resource decisions are based/focused on improvement for all students.

Driver 4. Broad-based support from parents, policymakers, institutions of higher education, business and industry, foundations, and other segments of the community for goals and collective value of the program, based on rich presentations of the ideas behind the program, the evidence gathered about its successes and its failures, and critical discussions of its efforts.

• There is coordination between the initiative and other stakeholders.

• A comprehensive effort maximizes broad-based support for program goals.

• The goal of improving the achievement of all students has been embraced by all relevant stakeholders.
  
  
• All relevant stakeholders understood and accepted systematic change as a strategy for improving education.

• Parent organizations, community-based organizations, business-industry, and higher education are collaborating in support of the reform.

Driver 5. Accumulation of a broad and deep array of evidence that the program is enhancing student achievement, through a set of indices that might include achievement test scores, higher level courses passed, college admission rates, college majors, Advanced Placement tests taken, portfolio assessment, and ratings from summer employers that demonstrate that students are generally achieving at a significantly higher level in science and mathematics.

• Data relative to implementation of RSI are being collected.

• Outcomes from science and math programs are available for public scrutiny.
  
  
• Multiple methods are used to assess students and programs.

• Efforts are made to follow up on graduates.

• Percentage of students taking Advanced Placement exams has risen.

Driver 6. Improvement in the achievement of all students, including those historically underserved.

• All students (including historically underserved) show increased scores on measures of learning.

• Standardized tests show improvement in minority student achievement.

Round 2 of the Delphi Procedure

The results of the first round of the Delphi process were compiled along with comments. These were disseminated to the Resource Advisory Team along with a clean copy of the questionnaire. The team members were asked to review their colleagues' responses and complete the questionnaire again, modifying their responses as they deemed appropriate. Emphasis was placed on the importance of marking one response per indicator. Once again, only nine of the ten team members returned the questionnaire. This time, 75 indicators were validated by approximately 80 percent or more of the team members (Appendix 5):

Driver 1. Implementation of comprehensive, standards-based curricula as represented in instructional practice, including student assessment, in every classroom, laboratory, and other learning experience provided through the system and its partners.

• Curriculum is of high quality and aligned with national standards.

• High quality curriculum implemented with all students.

• Extent of standards-based curriculum implementation in classroom.

• Hands-on, inquiry-based instruction is occurring in classrooms.

• Student-teacher-curriculum interactions are taking place at the classroom level.

• Teachers are able to articulate instructional standards.

• Teachers understand the K-12 curriculum.

• Teachers reference standards in selecting curriculum.
  
  
• Teachers report use of recognized standards.
  
  
• Assessments are aligned with standards and curriculum.

• Planning and teaching occurs across disciplines.

• Curriculum/instruction are relevant to the locale of the student.

• RSI courses are taught/offered regularly across K-12 levels.

• Students have opportunities to learn about careers and educational requirements in science and math.

• Curriculum and instruction goals are coordinated with relevant need areas.

• Curriculum activities are organized to promote continuity of learning for students.
  
  
• Activities are selected and arranged to provide a broad and coordinated educational program.

• Instructional activities are systematically planned, organized, and implemented in light of curricular goals.
  
  
• Activities are changed to improve curriculum.

Driver 2. Development of a coherent, consistent set of policies that supports provision of high quality mathematics and science education for each student, excellent preparation, continuing education, support for each mathematics and science teacher, and administrative support for all persons who work to dramatically improve achievement among students served by the system.

• Professional development program in place to train teachers to implement high quality curriculum.
  
  
• Teachers and staff participate in professional development program.
  
  
• Policies indicate a coherent vision that encompasses all students.
  
  
• Policies require that all students be enrolled in high quality and rigorous programs.
  
  
• Policies strengthen the emphasis on mathematics, science, and technology.
  
  
• Policies support the preservice education of teachers.
  
  
• Policies assure adequate time for the ongoing professional development of teachers.
  
  
• Policies require a tight alignment among curriculum, instruction, assessment, and professional development.
  
  
• Policies assure adequate financial and administrative support for the ongoing professional development of teachers.
  
  
• Policies designed to recognize and reward excellence in teaching.
  
  
• Policies support the system's capacity to collect and use data for continuous program improvement.

• Professional development requirements have been changed.
  
  
• Teacher certification requirements were changed.

• Teachers implementing inquiry-based learning receive planning time, mentor teacher assistance, opportunity for continuing professional development.
  
  
• Teachers are trained in the use of assessments.

• Professional development focuses on program and student needs.
  
  
• Teachers are selected and retained on the basis of qualifications and value added to student learning.

• Qualified teachers are encouraged to remain in rural, small schools.

• Procedures for developing and modifying curriculum and instruction objectives are in place.

• Qualifications, assignment, and support of personnel in the curriculum area are appropriate.

• School policy is effective for guiding practice.
  
  
• School policy is implemented adequately.
  
  
• The policy review and revision process is effective.

Driver 3. Convergence of the usage of all resources that are designed for or that reasonably could be used to support science and mathematics education--fiscal, intellectual, material, curricular, and extracurricular--into a focused and unitary program to constantly upgrade, review, and improve the educational program in mathematics and science for all students.

• Existing funding sources support systemic reform.
  
  
• Additional funds have been leveraged in support of improving education.

• Technology and telecommunications are used to support education.

• The budget percentage dedicated to education increased.
  
  
• Percentage of targeted funds such as Title 1, Perkins, and Eisenhower used in direct support of RSI.
  
  
• There have been changes in the student to teacher ratio.

• Facilities provide access to learning technologies.

• Resource decisions are based/focused on improvement for all students.

• Resources are made available for work beyond the school day/year on planned, focused areas.
  
  
• Funds are pooled to enhance professional development and other activities for improvement.

Driver 4. Broad-based support from parents, policymakers, institutions of higher education, business and industry, foundations, and other segments of the community for goals and collective value of the program, based on rich presentations of the ideas behind the program, the evidence gathered about its successes and its failures, and critical discussions of its efforts.

• There is coordination between the initiative and other stakeholders.

• A comprehensive effort maximizes broad-based support for program goals.
  
  
• There are increases in the level of support among all relevant stakeholders.
  
  
• The goal of improving the achievement of all students has been embraced by all relevant stakeholders.
  
  
• All relevant stakeholders understood and accepted systematic change as a strategy for improving education.

• Parent organizations, community-based organizations, business-industry, and higher education are collaborating in support of the reform.

• Students, teachers, and community members share a common understanding of expected outcomes.

Driver 5. Accumulation of a broad and deep array of evidence that the program is enhancing student achievement, through a set of indices that might include achievement test scores, higher level courses passed, college admission rates, college majors, Advanced Placement tests taken, portfolio assessment, and ratings from summer employers that demonstrate that students are generally achieving at a significantly higher level in science and mathematics.

• High-quality courses experienced an increase in student enrollment.

• Number and percentage of students presently affected by global system changes.

• Accountability measures were developed for all system levels.

• Data relative to implementation of RSI are being collected.
  
  
• Data relative to implementation of RSI are used in formative evaluation and in changes in the strategic plan.
  
  
• Site-specific program evaluation is carried out.

• Number of students in targeted programs and the congruence of their curriculum, instruction and student assessment, graduation requirements, and quality of the experience to the RSI.
  
  
• Student performance measures are valid and reliable.

• Job upon graduation or college attendance information are available.

• Outcomes from science and math programs are available for public scrutiny.
  
  
• Multiple methods are used to assess students and programs.

• Efforts are made to follow up on graduates.

• Percentage of students taking Advanced Placement exams has risen.

Driver 6. Improvement in the achievement of all students, including those historically underserved.

• All students (including historically underserved) show success in courses.
  
  
• All students (including historically underserved) show increased scores on measures of learning.

• Standardized tests show improvement in minority student achievement.

Conclusions

The results of the first round of the Delphi process show that 29 indicators were validated by the Resource Advisory Team: 4 for Driver 1, 10 for Driver 2, 3 for Driver 3, 5 for Driver 4, 5 for Driver 5, and 2 for Driver 6.

The results of the second round of the Delphi process show that 75 indicators were validated by the Resource Advisory Team: 19 for Driver 1, 23 for Driver 2, 10 for Driver 3, 7 for Driver 4, 13 for Driver 5, and 3 for Driver 6.

The increase in the number of drivers validated between the first and second rounds of the Delphi process represents an increase of 375 percent for Driver 1, 130 percent for Driver 2, 233 percent for Driver 3, 40 percent for Driver 4, 160 percent for Driver 5, and 50 percent for Driver 6.

As a result of the identification and expert validation of indicators for each NSF driver of systemic reform, the evaluation study of the rural systemic reform initiative will have a set of indicators to guide its work during on-site studies in 6-7 rural communities. The study team will seek evidence of the presence of each of the six drivers and attempt to determine the perceived value of the drivers in these communities.

APPENDIX 1

BIBLIOGRAPHY

Adelman, N. E. (1995). The challenge of linking plans for school-to-work opportunity systems. [Fourth in a series on school-to-work implementation.] Washington, DC: Council of Chief State School Officers. (ERIC Document Reproduction Service No. ED 401 463)

Alexander, G. C. (1995). District initiatives: reforming teacher and principal roles. University of Idaho. (ERIC Document Reproduction Service No. ED 387 923)

Arnold, L. B., & Birne, A. M. (1995, November). Parent university: A new program in Union City, New Jersey, ensures the success of students by educating their parents. Science Teacher, 62(8), 29-31. (ERIC Document Reproduction Service No. EJ 518 885)

Bhaerman, R., Cordell, K., & Gomez, B. (1995). Service-learning as a component of systemic reform in rural schools and communities. Philadelphia: Research for Better Schools, Inc. (ERIC Document Reproduction Service No. ED 391 614)

Boyle, R. A., & Skopp, L. (1998, April 22). Teachers as inquiries: Constructing a model of best practice. Paper presented at the Annual Meeting of the National Association for Research in Science Teaching, San Diego, CA. (ERIC Document Reproduction Service No. ED 417 981)

Bruckerhoff, C. (1998, April 13-17). Lessons learned in the evaluation of statewide systemic initiatives. Paper presented at the American Educational Research Association annual meeting, San Diego, CA. (ERIC Document Reproduction Service No. ED 414 282)

Clune, et al. (1997). Research on systemic reform: What have we learned? What do we need to know? Synthesis of the second annual NISE forum. Volume 2: Proceedings. (Workshop Report No. 4). Madison, WI: National Institute for Science Education.

Cohen, D. K. (1995, December). What is the system in systemic reform? Educational Researcher, 24(9), 11-17, 31. (ERIC Document Reproduction Service No. EJ 519 251)

Council of Chief State School Officers. (1999). How are states disseminating and implementing standards and frameworks? Strategies in mathematics and science. (Available Internet: www.ccsso.org)

Council of Chief State School Officers. (1995). State systemic education improvements. Status report. Washington, DC: Author (ERIC Document Reproduction Service No. ED 387 891)

Davila, N. (1998, November 5). Studying the linkages of system components. Challenges to Evaluating Systemic Reform in Education, American Evaluation Association '98, Chicago.

Dean, C. D., Acker-Hocevar, M., & Laible, J. C. (1997, November 1). A systemic approach to creating and implementing curricular innovation. Paper presented at the University Council for Educational Administration, Orlando, FL. (ERIC Document Reproduction Service No. ED 418 065)

Dilworth, M. E. (Ed.). (1998). Of course it matters. [A volume of essays in response to What Matters Most: Teaching for America's Future, a report of the National Commission on Teaching & America's Future.] Washington, DC: ERIC Clearinghouse on Teaching and Teacher Education. (ERIC Document Reproduction Service No. ED 417 156)

Education Commission of the States. (1995). Scaling-up mathematics, science, and technology education reform: Strategies from the National Science Foundation's Statewide systemic initiatives. Denver: Author. (ERIC Document Reproduction Service No. ED 391 646)

Edwards, P., Abel, F. J., Easton, S. E., Herbster, D., & Sparapani, E. F. (1996, February 24-28). Disadvantaged rural students: Five models of school-university collaboration. Paper presented at the Association of Teacher Educators' 76^th Annual Meeting, St. Louis, MO. (ERIC Document Reproduction Service No. ED 395 904)

Elmore, R. F., & Burney, D. (1997). Investing in teacher learning: Staff development and instructional improvement in community school district #2, New York City. Philadelphia: National Commission on Teaching & America's Future. (ERIC Document Reproduction Service No. ED 416 203)

Fullan, M. G. (1996, February). Turning systemic thinking on its head. Phi Delta Kappan, 77(6), 420-23. (ERIC Document Reproduction Service No. EJ 517 860)

Gibbs, R. M., Swaim, P. L., & Teixeira, R. (Eds.). (1998). Rural education and training in the new economy the myth of the rural skills gap. Ames, IA: Iowa State University Press.

Grummon, P. T. H. (1994, April). Evaluating systemic change in school-to-work initiatives. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans. (ERIC Document Reproduction Service No. ED 372 249)

Heck, D. J. (1998, November 6). Challenges to evaluating systemic reform in education: Resolving conflict in purposes for doing evaluation of systemic reform. Paper presented at the Annual Meeting of the American Evaluation Association, Chicago.

Horn, J. (1999). Notes from the rural systemic initiatives evaluation study resource advisory team/committee (RAT/RAC) meeting. (March 13-14, 1999). Denver, CO. Unpublished manuscript.

Kahle, J. B. (1998, April). Reaching equity in systemic reform: How do we assess progress and problems? (Research Monograph No. 9). Madison, WI: National Institute for Science Education.

Knapp, M. S. (1997). Between systemic reforms and the mathematics and science classroom: The dynamics of innovation, implementation, and professional learning. (Research Monograph No. 1). Madison, WI: National Institute for Science Education.

López-Ferrao, J. E., & Warner, L. S. (1998). 1998 Program effectiveness reviews (PER) report. Statewide systemic initiatives (SSI) program. December 17, 1997. Arlington, VA: National Science Foundation. (ERIC Document Reproduction Service No. ED 417 964)

McGrath, D., & Van Buskirk, W. (1997). Start with the faculty. The Newark faculty alliance for education and systemic educational reform. New York: National Center for Urban Partnerships. (ERIC Document Reproduction Service No. ED 416 282)

National Science Foundation. (1994). Foundation for the future. Washington, DC: Directorate for Education and Human Resources. (ERIC Document Reproduction Service No. ED 370 808)

National Science Foundation Rural Systemic Initiative (RSI) Evaluation RFP Notes. (Available Internet: www.ehr.nsf.gov/ehr/rec/pubs/usi-rsi-eval.htm and www.ehr.nsf.gov/EHR/ESR/RSI.htm//desc)

Norman, J., Stein, M., Moussiaux, S., & Clay-Chambers, J. (1998, April). The effect of the Detroit urban systemic initiative on perceived instructional practice and curriculum adequacy. Paper presented at the Annual Conference of the National Association for Research in Science Teaching, San Diego, CA. (ERIC Document Reproduction Service No. ED 417 979)

Nowakowski, J., Bunda, M. A., Working, R., Bernacki, G., & Harrington, P. (1985). A handbook of educational variables. Boston: Kluwer Academic Publishers.

Radford, D. L. (1998). Transferring theory into practice: A model for professional development for science education reform. Journal of Research in Science Teaching, 35(1), 73-88.

Ridgway, J. (1998). The modeling of systems and macro-systemic change: Lessons for evaluation from epidemiology and ecology. (Research Monograph No. 8). Madision, WI: National Institute for Science Education.

Russon, C., & Horn, J. (1999). The relationship between NSF's drivers of systemic reform and the rural systemic initiatives. Kalamazoo, MI: Western Michigan University Evaluation Center.

Southwest Consortium for the Improvement of Mathematics and Science Teaching. (1994). Guide to reform in mathematics and science education: The big players. Austin, TX: Southwest Educational Development Lab. (ERIC Document Reproduction Service No. ED 387 359.

Wells, A. S., & Oakes, J. Potential pitfalls of systemic reform: Early lessons from research on detracking. Sociology of Education [extra issue], 135-43. (ERIC Document Reproduction Service No. EJ 534 908.

Zucker, A. A., & Shields, P. M. (1995, December). Evaluation of the National Science Foundation's statewide systemic initiatives (SSI) program. Second-year case studies: Connecticut, Delaware, and Montana. Arlington, VA: National Science Foundation. (ERIC Document Reproduction Service No. ED 406 231.

APPENDIX 2

Rural Systemic Initiative

Indicator Questionnaire

Instructions: Please read the description of each indicator in the left-hand column. Decide to which of the six drivers of educational system reform (descriptions of each driver can be found in the Appendix) the indicator most strongly relates. Place a mark corresponding to your selection in one of the right-hand columns. If the indicator does not correspond to any of the drivers, mark NA. Comments are welcome. Return the questionnaire to The Evaluation Center in the self-addressed envelope.

APPENDIX

The six drivers of educational system reform:

1. Implementation of comprehensive, standards-based curricula as represented in instructional practice, including student assessment, in every classroom, laboratory, and other learning experience provided through the system and its partners. (Process)



2. Development of a coherent, consistent set of policies that supports provision of high quality mathematics and science education for each student, excellent preparation, continuing education, support for each mathematics and science teacher (including all elementary teachers), and administrative support for all persons who work to dramatically improve achievement among all students served by the system. (Process)



3. Convergence of the usage of all resources that are designed for or that reasonably could be used to support science and mathematics education--fiscal, intellectual, material, curricular, and extracurricular--into a focused and unitary program to constantly upgrade, review, and improve the educational program in mathematics and science for all students. (Process)



4. Broad-based support from parents, policymakers, institutions of higher education, business and industry, foundations, and other segments of the community for goals and collective value of the program, based on rich presentations of the ideas behind the program, the evidence gathered about its successes and its failures, and critical discussions of its efforts. (Process)



5. Accumulation of a broad and deep array of evidence that the program is enhancing student achievement through a set of indices that might include achievement test scores, higher level courses passed, college admission rates, college majors, Advanced Placement tests taken, portfolio assessment, and ratings from summer employers that demonstrate that students are generally achieving at a significantly higher level in science and mathematics. (Outcome)



6. Improvement in the achievement of all students, including those historically underserved. (Outcome)

APPENDIX 3

The Resource Advisory Team

Members of the Resource Advisory Team were carefully selected to provide a broad base of knowledge and experience that will be used throughout the study and that will serve to enhance the credibility of the findings. Each team member is listed below with institutional affiliation and a brief description of the area of expertise that is applicable to this study. All of these individuals have agreed to participate as defined in the proposal. Their resumes were not included with the proposal, but are available at The Evaluation Center and may be reviewed on request.

Dr. Larry Enochs--professor and director of a science teaching/science education center at the University of Wisconsin-Milwaukee. Dr. Enochs has worked professionally in Kansas, Texas, and Wisconsin in pre- and inservice science education activities. He has worked at NSF and has a good understanding of the goals of the systemic initiatives. With a focus on secondary school science, he will focus his efforts with this study on teacher preparation, the science curriculum, and impact on the secondary school. His interest in using the immediate environment as a laboratory for science instruction will be particularly valuable as we examine the extent and the effectiveness of this in the RSI collaboratives.

Dr. Gene Hall--Research Professor, University of Northern Colorado. Dr. Hall was prepared as a science educator at the doctoral level, and he is well known for his work in science education. His work in developing and using the Concerns Based Adoption Model (CBAM) is well known in efforts to implement innovations and influence school reform. As we identify issues and concerns about implementation of innovations in the curriculum and the operations of schools, his expertise will be highly useful.

Dr. Mary Harris--College of Education and Human Development, University of North Dakota. As dean of a college with major responsibilities for developing teachers as rural educators, she is well experienced in the preparation of teachers for rural areas and for school-university relations and collaboratives. In addition, she has a strong background in multiethnic studies and gender considerations in instruction at the elementary and middle school/junior high school level. Her work in the Dakotas will be particularly helpful in working with rural systemic efforts in that area.

Dr. James Jess--Superintendent, CAL (Iowa) Community Schools. Dr. Jess has served for many years as a superintendent of one of the most highly recognized rural schools in the U.S. He has developed exemplary programs and introduced numerous innovations in a small consolidated rural school district. He has played an important role in the development of legislation and public funding policies for small schools. His involvement will focus on school organizations within the RSI, the development of appropriate curricular responses for rural students, and administrative support mechanisms for innovation.

Mr. Paul Nachtigal--Executive Director of the Annenberg Rural Challenge Project, with national offices in Granby, Colorado. Paul Nachtigal has served as a teacher and administrator in rural schools, but he is probably best known for study of rural education and his work with various foundations, including the Ford Foundation, with an emphasis on studying leadership in poor, rural communities. As head of the parallel rural initiative through the Annenberg project, he will provide a useful link to this effort and the introduction of innovations in rural schools and the development of functional collaboratives.

Dr. Joseph Newlin--Executive Director of the National Rural Education Association (NREA) and professor at Colorado State University. With graduate education in adult and rural education and his experiences with NREA, he will be a valuable resource for the study as it focuses on studying the potential/planned/real impact on rural communities and the development of leadership within the collaboratives. Representing rural education, he has participated on a number of select committees to identify exemplary schools for national recognition.

Dr. Steve Oliver--Department of Science Education, University of Georgia. He has worked as a science educator at the university level in Georgia and Kansas, conducted numerous workshops for teachers and administrators in schools serving rural communities, and is actively engaged in the study of science curriculum for small/rural schools.

Dr. Jack Sanders--currently Executive Director of SERVE in Greensboro, North Carolina. Dr. Sanders has conducted considerable research in rural education and served as an administrator at the Appalachia Regional Laboratory, where the rural ERIC system is located. He has been active in rural organizations, and he will be particularly helpful to this project as we study the use of external services and effective organizational structures of educational institutions and agencies.

Dr. Daniel Stufflebeam--Director of The Evaluation Center, Western Michigan University. Well known for his development and use of the CIPP Model for evaluation, Dr. Stufflebeam is internationally known for his work in program and personnel evaluation. He will serve as a major resource for our study of the context, outcomes, and impact of the various RSIs and the interpretation of the data in relation to the drivers.

Dr. William Webster--Recently retired as head of the research and evaluation unit of the Dallas, Texas, school district and a longtime collaborator with the WMU Evaluation Center. Dr. Webster is nationally known for his work with student achievement data and the application of analysis for studies of teacher and school effectiveness. He brings to the study an exceptional understanding of student achievement and, with this background, he will oversee the interpretation of data and its use in determining the effectiveness of the RSI. Additionally, he will be able to provide us with a perspective of the uniqueness of rural/small schools from the perspective of a large, urban school district.

Alternate: Dr. Angelo Collins--Peabody College, Vanderbilt University. Although not designated as a member of the Resource Advisory Team at this time, she has expressed a willingness to serve during the latter period of the project (Years 2 and 3) as requested and as appropriate for her area(s) of interest and expertise. She has worked as a science educator in the public schools and at the university. She has done extensive work in developing national science education standards, teaching standards (National Board for Professional Teaching Standards), and subject matter assessments for science teachers.

In summary, we have constituted a resource based advisory team with a broad base of expertise and experiences in rural education, science teaching and science education, student achievement, evaluation, innovation/reform, school administration, standards/curricula and testing/student achievements. This is a group of professionals who provide immediate credibility to this study. We intend to make full use of them in the study of RSIs, in the interpretation of findings, and in the various phases and tasks.

In addition, one or more of the team members will accompany study staff/researchers during the site visits to the RSIs. In this role, they will provide ongoing consultation to the staff, in addition to a particular assignment related to their area of expertise. As we better understand individual sites, a long-term plan and schedule of each member's involvement will be developed.

APPENDIX 4