An Analysis of Selected Data from the External Evaluation
of the RMTC Local System Change Project

(January 2000)

Each spring since 1997, teachers participating in the RMTC program were surveyed about a variety of issues related to the teaching and learning of mathematics at the middle and high school level. External evaluators have reviewed the results and conducted analyses to look at change over the three years the surveys have been administered. Additionally, selected items from principal surveys and data from pre- and post- summer workshop sessions were also reviewed.

Results of the analyses are presented in this document. This report should not be seen as a comprehensive evaluation of the RMTC program. It represents only a small part of the data collected through the evaluation of the program. This document was prepared by Chia-Lin Hsieh and Mark Jenness of Science and Mathematics Program Improvement (SAMPI) at Western Michigan University. For more information, call 616-387-3791 or email: jenness@wmich.edu.

The RMTC collaborative consists of 25 middle schools and 19 high schools in 20 school districts. Two hundred thirty-seven teachers have participated in at least some professional development. Participating teachers have received a grand total of 22,666 hours of professional development (PD) since the beginning of the RMTC program. The 158 high school teachers in the project have received a total of 16,793 hours of PD since the project began in 1997, an average of 124.4 hours each. In approximately one year that the middle schools have been involved in the project, 79 teachers have received 5,793 hours of PD, an average of 60.5 hours each.

Classroom Strategies and Level of Preparedness
On the 1997, 1998, and 1999 LSC Core Teacher Surveys, teachers were asked to rate a series of items about classroom strategies on their level of preparedness. The 4-point Likert scale ranged from 1 (not adequately prepared) to 4 (very well prepared). Mean scores from 1997, 1998, and 1999 are shown in the following table.

Teachers' level of preparedness on classroom strategies in 1999 (M=3.2) is statistically higher than in 1998 (M=2.9, p<.00) and 1997 (3.0, p<.00). Teachers felt better prepared on classroom strategies in 1999 than in 1998 and 1997. The strategy teachers felt least prepared to do throughout all three years is "involving parents in the mathematics education of their students."

Mathematics Content and Level of Preparedness
On the 1997, 1998, and 1999 LSC Core Teacher Surveys, teachers were asked to rate a series of mathematics topics on their level of preparedness. The 4-point Likert scale ranged from 1 (not adequately prepared) to 4 (very well prepared). Mean scores from 1997, 1998, and 1999 are shown in the following table.

Teachers' level of preparedness on mathematics content in 1999 (M=3.11) is statistically lower than in 1998 (M=3.13, p<.05) and 1997 (3.23, p<.05). Teachers felt less prepared on mathematics content in 1999 than 1998 and 1997. This may indicate that teachers are more aware of what they really understand in terms of mathematics content than before. Two of the content areas in which teachers felt less prepared are mathematical structures and calculus.

A dissaggregation of data from the 1999 LSC Core Teacher Survey on selected items was conducted comparing responses based on the amount of professional development teachers had received and their perception of preparedness. Four categories of professional development time were created: (1) 0-39 hours, (2) 40-79 hours, (3) 80-119 hours, and (4) 120 hours or more. The following table displays the percentage of teachers who perceived they were well prepared.

In general those with the least exposure to the RMTC professional development tend to rate themselves higher than the other groups. This is a phenomenon that the authors of this report have seen in other projects. As teachers become more familiar with the topics and issues that are the focus of the professional development, they seem to realize they are not as well prepared in those areas as they may have thought before being involved in the professional development. As they receive more professional development and have opportunities to implement what they are learning in the classroom, they usually become more confident of the preparation. The survey results support that pattern.

Mathematics Content and Level of Preparedness  

Mathematics Content	Hours of Professional Development
	0-39 hrs (n = 35)	40-79 hrs (n = 45)	80-119 hrs (n = 31)	120+ hrs (n = 75)
	Percentage well-prepared
Estimation	59	53	65	60
Measurement	68	53	84	67
Pre-algebra	56	76	90	85
Algebra	47	71	87	80
Patterns and relationships	59	58	68	76
Geometry and spatial sense	53	56	53	61
Functions (including trigonometric functions) and pre-calculus concepts	24	40	45	47
Data collection and analysis	65	51	65	54
Probability	39	42	48	38
Statistics (e.g., hypothesis tests, curve fitting and regression)	21	31	39	28
Topics from discrete mathematics (e.g. combinatorics, graph theory, recursion)	9	18	23	29
Mathematical structures (e.g., vector spaces; groups, rings, fields)	9	9	10	9
Calculus	8	13	13	17
Technology (calculators, computers) in support of mathematics	29	18	33	32
Average	43	42	52	49

Pre- and post-tests were given to summer workshop participants as a way to measure their mathematics content learning. The pre-test was made up of actual student assessment items from the Core-Plus Mathematics curriculum, the focus of the workshop. At the end of the workshop, the same test was given to participants again (post-test). The content of tasks was from geometry, algebra, discrete mathematics, and statistics. Results are shown below:

Core-Plus Course 3
The dependent t-test was conducted to evaluate whether the Core-Plus Course 3 workshop makes a difference on participants’ progress between their pretest scores and posttest scores. The findings indicated that the mean posttest score (M=66.6, SD=22.8) was statistically significantly greater than the mean pretest score (M=52.9, SD=22.4), t(15)=-3.8, p<.01. The findings indicate that participants made important gains in their mathematics content knowledge.

Core-Plus Course 4
The dependent t-test was conducted to evaluate whether the Core-Plus Course 4 workshop makes a difference on participants’ progress between their pretest scores and posttest scores. The findings indicate that the mean posttest score (M=90.8, SD=5.8) was statistically significantly greater than the mean pretest score (M=50.5, SD=17.8), t(12)=-9.1, p<.001. The findings indicate that these teachers, too, made significant gains in their mathematics content knowledge.

Teachers' Perception of the Conditions that Encourage Effective Mathematics Instruction
In spring 1999, 18 school districts were involved in the RMTC collaborative at the time the Core Teacher Survey was conducted. One district did not respond. A total of 190 teachers participated in the survey.

Teachers were asked to rate variables that may have an impact on their mathematics instruction.

• the importance that the school places on mathematics,
• the consistency of mathematics reforms efforts with other district reforms,
• the public attitude toward reform,
• the time available for teachers to plan and prepare lessons,
• the time available for teachers to work with other teachers, and
• the time available for teacher professional development.

Please see the following chart for details. Basically, all of the variables impact effective instruction, but to different degrees. Teachers are most likely to think that "importance that the school places on math" (76%) encourages effective instruction. They also think that "time available for teacher professional development" (65%) and "time available for teachers to plan and prepare lessons" (59%) also impact effective instruction.

Among the 6 variables, teachers were less likely to feel that "public attitude toward reform" and "consistency of math reform efforts with other school reforms" affect their teaching. The percentage of neutral responses, 39% and 35% respectively, suggest they feel these are relatively insignificant variables. Although 48% of teachers report that "time available for teachers to work with other teachers" would have a positive effect, 37% believe it would inhibit their teaching effectiveness. This response is worth further study.

*Middle and high school teachers who participated in Renewing Mathematics Teaching through Curriculum (RMTC).

Principals' Perception of the Conditions that Encourage Effective Mathematics Instruction
A comparison of principal responses in 1997, 1998 and 1999 to three of the items asked of teachers in the chart above show the following. It should be noted that 1999 data includes middle school principals.

Principals were asked to indicate the effects of these items on mathematics instruction. The number in each box represents the percentage of principals who perceived it would have a positive effect on mathematics education.

Principals believe that (a) the importance the school places on mathematics, (b) the consistency of the mathematics reform with other district/school reform efforts, and (c) public attitudes toward reform encourage effective teaching.

Teachers were asked to rate to what extent school conditions support their professional development. There were five questions:

• To what extent are you involved in planning your mathematics-related professional development?
• To what extent are you encouraged to develop an individual professional development plan to address your needs and interests related to mathematics education?
• To what extent are you given time to work with other teachers as part of your professional development?
• To what extent are you given time to reflect on what you have learned and how to apply it to the classroom?
• To what extent do you receive support as you try to implement what you have learned?

Please see the following chart for responses.

* Middle and high school teachers who participated in RMTC.