Description of Data
The Kentucky Core Content Test (KCCT) was used as the data source for measuring students’ mathematics and science academic performance. KCCT is a battery of open-response and multiple-choice questions in selected contents at each grade level. The mathematics KCCT is given at the fifth, eighth, and eleventh grades. The science KCCT is administered at the fourth, seventh, and eleventh grades. Raw scores for each battery in the KCCT (e.g., mathematics and science) are translated to a scale score. Scale scores are translated to one of four performance levels: novice, apprentice, proficient, or distinguished (NAPD). The NAPDs get weighted numerically and combined into a numerical scale that ranges from 0 to 140. This computation results in an “academic” index for each content area. The goal of Proficiency translates into an academic index value of 100. (Source: http://www.education.ky.gov/KDE/Administrative+Resources/
Testing+and+Reporting+/CATS/Accountability+System/2003+CATS+Interpretive+
Guide.htm, retrieved March 15, 2004)
The 1999-2003 mathematics and science KCCT academic indices were retrieved from the Kentucky Department of Education’s Web site for the elementary, middle, and high school levels for the 20 ARSI school districts and 20 matching school districts.
The state of Kentucky collects nonacademic data on attendance, retention, dropout, and successful transition to adult life. These four data points comprise the nonacademic component of the Commonwealth Accountability Testing System (CATS). As a part of the No Child Left Behind (NCLB) Act, Kentucky also collects and reports data on graduation rate. For purposes of this study, attendance, dropout, successful transition to adult life, and graduation data are reported.
The dropout rate is the percentage of students that drop out of school. The attendance rate provides the average percentage of attendance for all students. The graduation rate is the percentage of students graduating. The successful transition to adult life is data collected on the status of Kentucky graduates. School personnel collect the data on the prior year’s graduates in the fall of each year. This survey collects data on the number of graduates that have entered full time college, vocational or technical school, the military, or a combination of work and school. This definition of “successful” is used to determine the percentage of students who have successfully transitioned to adult life. All nonacademic indicators are lagged one year. In other words, they are reported in the current school year, but are based on data from the previous school year. (Source: http://www.
education.ky.gov/KDE/Administrative+Resources/Data+and+Reseach/Nonacademic+
Data/Nonacademic+Data+Report+Guidelines.htm, retrieved March 15, 2004)
Data Retrieval and Statistical Analysis
Mathematics and science academic indices at the elementary, middle, and high school levels for 1999-2003 from the 20 ARSI school districts and the 20 matching school districts were downloaded from the Kentucky Department of Education Web site. Attendance rate data from 1999-2003 for all 3 levels for the 40 school districts were also retrieved from the Kentucky Department of Education Web site. High school dropout rates and successful transition to adult life data for 1999-2003 were downloaded from the Kentucky Department of Education Web site. High school graduation rate data were obtained for 2001 and 2002 from the No Child Left Behind adequate yearly progress reports, which were also available from the Kentucky Department of Education’s Web site.
The data were entered into SPSS. Arithmetic means were computed for each variable. The paired-samples t-test procedure was used to compare the means of the ARSI school district and matching school district data for each variable. The paired samples t-test is appropriate to use when comparing two groups in which subjects (in this situation school districts) are matched (Huck et al., 1974).
Study Strengths and Limitations
As with any research study, this effort possesses a number of strengths and limitations. A major strength of this investigation is that it cites data from multiple years. In other words, trends over time are reported, providing a more comprehensive account of mathematics and science achievement and the nonacademic student performance in the ARSI school districts and the matching school districts. This is preferable to data collected and reported from only one test or survey administration, which is limited in scope and cannot provide contextual information about trends. A second strength is the study’s inclusion of comparisons between the ARSI school districts and the state’s performance for each of the variables. This information offers further contextualization of the findings.
Limitations also exist. Graduation rate data were only available for a two year period (2001 and 2002). Disaggregated data on variables of interest such as academic performance of ARSI school district students and matching school district students that qualify for free or reduced price lunch were not available for a five year period. The mathematics and science academic indices were available at the school level (e.g., elementary, middle, and high school) for students qualifying for free or reduced price lunch for 2003 only. The percentage of students scoring at a proficient level or higher was reported for the mathematics KCCT at the district level for 2002 and 2003 only. Nonacademic data were not disaggregated by students qualifying for free or reduced price lunch.
Caution should also be exerted when interpreting small differences occurring in scores from one year to another or between different groups (e.g., ARSI school districts, matching school districts, and the state ). The paired samples t-test was used to test for statistical significance (p<.05) for the ARSI school districts and matching school districts for each variable of interest in this study. Comparisons are also made with the state results, although no statistical tests were performed to determine if differences between the ARSI school districts and the state were statistically significant.
Findings on Student Academic and Nonacademic Performance
Findings of student performance in mathematics and science (e.g., academic indices) are presented first. Within these two content areas, findings are organized by elementary, middle and high school. Data disaggregated by student eligibility for free or reduced lunch are presented next, again organized by student performance in mathematics and science. The findings section concludes with a presentation of data on the nonacademic indicators. Nonacademic indicators include elementary, middle and high school attendance rates, high school dropout rate, high school graduation rate and successful transition to adult life.
Mathematics
The mathematics academic indices for the past five years are presented for the ARSI and matching schools as well as the state. Findings are organized by school level: elementary, middle and high school.
Elementary school. The average mathematics academic index for ARSI elementary schools has steadily increased over the last 5 years (Figure 1). In 1999, the average academic index for the ARSI elementary schools was 51.46 and in 2003 it was 65.23. Although the ARSI school district performance is slightly higher than the matching school districts, the differences are not statistically significant except in 2002. In 2002, the difference in the ARSI elementary schools’ mathematics academic index (64.49) and the matching elementary school districts’ academic index (58.05) was statistically significant (p<.05).
The mathematics academic index for the ARSI school districts consistently has been less than that of the state for the past five years, though the gap has decreased in the last two years.
Figure 1. KCCT Elementary School Mathematics Academic Index Comparison
Middle school. Generally, the ARSI middle schools’ mathematics index has increased over the last 5 years (Figure 2). In 1999, the average mathematics index was 50.99 and in 2003 it was 63.58 for the ARSI middle schools. The exception to this was in 2002, when the academic index decreased. This trend was similar for both the matching school districts and the state.
The ARSI school districts and matching school districts performed similarly over the last 5 years, while the state performance was higher. The largest difference in the mathematics academic index between the ARSI and matching middle schools was in 2003. The ARSI middle schools had a mathematics academic index 4.21 points higher than the matching middle schools. The difference was not statistically significant. Differences between the state’s and ARSI school district’s middle school mathematics academic index was lowest in 2003. The state’s academic index was 65.69 and the ARSI middle schools’ average academic index was 63.58.
Figure 2. KCCT Middle School Mathematics Academic Index Comparison
High school. Differences have been minimal between the eleventh grade mathematics academic index for ARSI school districts and the matching school districts for the past 5 years (Figure 3). The ARSI school districts’ high school mathematics academic index has risen by nearly 10 points in the last 5 years. In 1999 the ARSI high schools’ mathematics academic index was 48.87 and in 2003 it was 57.9. The mathematics academic index for the ARSI school districts has averaged 7.4 points less than the state for the last 5 years.
Figure 3. KCCT High School Mathematics Academic Index Comparison
Science
The science academic indices for the past five years are presented for the ARSI and matching schools as well as the state. Findings are organized by school level: elementary, middle and high school.
Elementary school. The science academic index for ARSI elementary schools has increased by nearly 14 points in the past 5 years (Figure 4). In 1999 the ARSI elementary schools’ science academic index was 68.74 and in 2003 it was 82.65.
For the last three years, the ARSI elementary school districts’ science academic index has been slightly higher (not statistically significant) than that for the matching school districts. The differences in the academic indices between the two groups have been less than 3 points.
The average science academic index for the ARSI elementary schools has been similar to the state for the last five years. In the last two years, the ARSI elementary school districts’ science academic index has been slightly above that of the state.
Figure 4. KCCT Elementary School Science Academic Index Comparison
Middle school. ARSI middle schools’ science academic index has also increased the last five years, although not as dramatically as at the elementary level (Figure 5). The differences between the 2003 and 1999 academic indices were 6.28 points.
The science academic index for the ARSI middle schools has consistently been higher than that of the matching school districts, although the differences are not statistically significant. The differences between the two groups have averaged 1.9 points.
ARSI school districts’ middle school science academic index has been lower than that of the state for the past five years. The difference between the ARSI middle schools’ academic index and the state averaged 2.34 points for this five year period.
Figure 5. KCCT Middle School Science Academic Index Comparison
High school. The science academic indices at the high school for all three comparison groups are lower than those of the elementary and middle schools (Figure 6). The ARSI districts have had academic indices similar to the matching school districts for the last five years. The difference between the academic indices for the two groups has averaged 1.4 points the last five years. The state’s high school science academic has been three to five points higher than the ARSI high schools from 1999 to 2003.
Figure 6. KCCT High School Science Academic Index Comparison
Free/Reduced Lunch Eligibility Academic Performance Comparison
Comparisons of mathematics and science academic indices at the elementary, middle and high school levels for students qualifying for free or reduced lunch are presented for the ARSI and matching schools as well as the State for 2003. Comparisons are also made with students qualifying for free or reduced lunch scoring at or above the proficient level in mathematics for ARSI and the matching school districts and the State for 2002 and 2003.
Mathematics. A disaggregation of 2003 mathematics academic indices by students qualifying for free or reduced lunch revealed that ARSI schools performed better than the matching schools and the state at all three levels (Figure 7). The difference in the elementary mathematics index for ARSI and the matching schools was statistically significant (p<.05). Differences for the ARSI and matching schools were greatest at the elementary level (5.44) and lowest at the high school level (3.64). The differences between the ARSI schools and the state were greatest at the middle school (4.06) and least at the high school (1.02).
Figure 7. Comparison of 2003 KCCT Mathematics Academic Index
For Students Qualifying for F/RL
Additional mathematics data at the district level for the past two years reveals ARSI schools’ have had a higher percentage of students who qualify for free or reduced lunch scoring at or above the proficient level as compared with both the matching school districts and the state (Figure 8). The differences were greatest in 2003. The average percentage of students scoring at or above the proficient level for the ARSI school districts was 22.68 as compared to the non-ARSI school districts average percentage at 18.58 percent. This difference was statistically significant (p<.05).
Figure 8. Percentage of Students Qualifying for F/RL
At or Above Proficient Level on Mathematics KCCT
Science. 2003 science academic indices for students qualifying for free or reduced lunch were higher for ARSI schools at all three levels as compared with the matching schools and the state (Figure 9). Differences were greatest between the ARSI elementary schools’ science academic index and both the matching and state elementary science academic indices. However, the difference in the elementary mathematics index for the ARSI schools and matching schools was not statistically significant (p<.05).
Figure 9. Comparison of 2003 KCCT Science Academic Index
for Students Qualifying for F/RL
Nonacademic Indicators
Findings relative to comparisons of elementary, middle and high school attendance rates for the ARSI and matching school districts and the State for the last five years are presented. The high school dropout and successful transition to adult life rates for the last five years are presented for the ARSI and matching schools and the State. Additionally, the high school graduation rates for the ARSI and matching schools and the State are presented for 2001 and 2002.
Elementary school attendance rate. Attendance rates at the elementary level for ARSI school districts, the matching school districts, and the state have consistently been high for the past 5 years (Figure 10). The average attendance rate for all 3 groups for the past 5 years has been at least 94 percent.
Figure 10. Elementary School Attendance Rate Comparison
Middle school attendance rate. Attendance rates at the middle school have also been quite high for the last 5 years for the 3 comparison groups (Figure 11). Average attendance has been 94 percent for the ARSI middle schools, matching middle schools, and the state for the last 5 years.
Figure 11. Middle School Attendance Rate Comparison
High school attendance rate. High school attendance rates are a slight bit lower (1 to 3 percent) for ARSI high schools, matching high schools, and the state’s high school average as compared with the middle and elementary schools’ attendance rates for these same groups over the same time period (Figure 12). The average high school attendance has consistently been 91 or 92 percent for all 3 groups over the last 5 years.
Figure 12. High School Attendance Rate Comparison
High school dropout rate. The ARSI school districts’ high school dropout rate has declined over the last 5 years (Figure 13). In 1999, the dropout rate was 6.19 percent and in 2003 was 4.78 percent. The ARSI school districts have experienced a slightly higher (although not statistically significant) dropout rate than the matching school districts for 4 of the 5 years. ARSI school districts also have a slightly higher dropout rate than the state over the last 5 years. Typically, the difference between the ARSI school districts’ and the state’s dropout rate averaged 0.8 percent.
Figure 13. High School Dropout Rate Comparison
High school graduation rate. The ARSI school districts’ graduation rate increased slightly between 2001 and 2002, from 76.9 percent to 78.71 percent (Figure 14). The high school graduation rates in 2001 and 2002 for the ARSI school districts, matching school districts, and the state were comparable.
Figure 14. High School Graduation Rate Comparison
Successful transition to adult life. Between 93 and 94 percent of ARSI school district high school graduates have “successfully transitioned to adult life.” Follow-up surveys with the previous year’s graduates found that the vast majority of students have gone on full time to college, vocational or technical school, the military, or a combination of work and school (Figure 15). The percentages of ARSI high school graduates that have successfully transitioned to adult life are statistically significantly higher (p<.05) than that of the matching school districts in 1999-2002. The ARSI school district percentages are similar to those of the state.
Figure 15. Successful Transition to Adult Life Comparison
Summary
ARSI school district performance in mathematics and science as defined by the academic index on the KCCT mathematics battery has steadily increased over the last five years. The greatest increases occurred in mathematics at all three levels (elementary, middle, and high school). In 1999, the elementary school mathematics academic index was 51.46 and in 2003 the mathematics academic index was 65.23. Likewise, the middle school mathematics academic index was 50.99 in 1999 and 63.58 in 2003. In 1999, the high school mathematics academic index was 48.87, and in 2003 it was 57.9.
Although the increases in the science academic indices for the ARSI middle and high schools have not been as great as the mathematic academic indices, the science academic indices were greater to start with than the mathematics academic indices. The ARSI elementary school science academic index was 68.74 in 1999, and in 2003 it was 82.65. The ARSI middle school science academic index was 59.08 in 1999, and in 2003 the academic index was 65.36. Similarly, the ARSI high school science academic index was 55.19 in 1999 and 60.7 in 2003.
Overall, the ARSI elementary, middle, and high schools have performed similarly in mathematics and science to the matching elementary, middle, and high schools. Even though the ARSI schools at all three levels had slightly higher academic indices most of the time in both mathematics and science in 1999 through 2003, the only difference that was statistically significant was the 2002 elementary mathematics academic index. In 2002 the ARSI elementary schools’ average mathematics academic index was higher (6.44 points) than the matching elementary schools’ average mathematics academic index. The greatest disparity between ARSI and the matching school districts’ academic indices was in elementary mathematics. The average difference between the academic indices was 4.01 points. For all other levels in both mathematics and science, the average differences ranged from 1.36 to 1.99 points.
Generally, the state academic indices at the elementary, middle, and high schools in both mathematics and science over the last 5 years have been higher than the ARSI schools. The two exceptions were in 2002 and 2003 when the ARSI elementary science academic indices were slightly above the state’s (less than one point). Differences in the academic indices between the state and ARSI elementary schools averaged 1.28 points over the last 5 years. The greatest differences occurred between the ARSI and state high school mathematics academic indices. The average difference in the last 5 years was 7.37 points.
Disaggregating the 2003 mathematics and science academic indices for students qualifying for free or reduced lunch revealed that ARSI elementary, middle, and high schools had slightly higher academic indices than both the matching schools and the state. The 5.44 difference in the elementary mathematic index for the ARSI and matching schools was statistically significant, with the ARSI elementary schools having a higher mathematics academic index. Differences in the mathematics indices for the ARSI elementary, middle, and high schools and the state ranged from 1.02 (high school) to 4.06 (middle school). Differences in the science indices for the ARSI elementary, middle, and high schools and the state ranged from1.9 (high school) to 4.96 (elementary).
Elementary and middle school attendance rates for the ARSI, matching schools, and the state have consistently been at least 94 to 95 percent the past 5 years. The high school attendance rate for the 3 comparison groups has been slightly lower, 91 or 92 percent.
The ARSI high school dropout rate has been a little higher than the matching high schools and the state from 1999 to 2003, though the differences have been minimal. The greatest difference between ARSI’s dropout rate and that of the matching high schools was 0.38 percent in 1999. The most difference between ARSI’s dropout rate and the state’s high schools was 0.85 percent. The high school dropout rate of ARSI schools declined from 6.19 percent in 1999 to 4.78 percent in 2003.
Comparison of the 2000 and 2001 high school graduation rates of ARSI schools reveals a slight increase over time (1.81%). In 2001 the ARSI high school graduation rate was 76.9 percent, and the 2002 graduation rate was 78.71. Differences in the ARSI high school graduation rates and the matching high school graduation rates were minimal (less than 1%). The state’s high school graduation rate was 2 to 3 percent higher than the ARSI high school graduation rate in 2001 and 2002.
In terms of percentage of high school graduates that have successfully transitioned to adult life, ARSI schools exhibited statistically significant higher percentages than the matching schools for 1999 to 2002. Differences in that 4-year period ranged from 5.31 percent higher in 2001 to 2.98 percent higher in 2000. Although the state’s percentage of high school graduates that have successfully transitioned to adult life is slightly higher, the differences are nominal (1.13-2.17 percent).
In conclusion, it appears that the ARSI program may have had some impact on students’ mathematics and science achievement as measured by the academic index on the KCCT mathematics and science batteries. At all levels assessed (elementary, middle, and high school), the academic indices have steadily increased each year, although it is interesting that the matching school districts displayed similar trends of progress. Because both the ARSI and matching school districts experienced comparable positive increases, other factors besides the ARSI program may have contributed to increases.
ARSI schools’ performance in mathematics and science, although lower than that of the state as a whole, is showing improvement. Again, this suggests that ARSI has had a positive impact on the participating schools. For instance, in 2003 the average difference in academic indices at the elementary, middle, and high school for ARSI schools and the state has been three points. The greatest difference was in high school mathematics (6.49 points).
Examination of the nonacademic indicators shows that the ARSI program may have played a minor part in the small positive changes that have occurred in the ARSI high schools’ dropout rate, graduation rate, and successful transition to adult life. Again, changes are minimal and are akin to changes with the matching schools and the state.
References
Huck, S. W., Cormier, W. H. & Bounds, W. G., Jr. (1974). Reading statistics and research. New York: HarperCollins.
Kentucky Department of Education. (2004). Map of Kentucky school districts. Retrieved March 15, 2004, http://www.education.ky.gov/KDE/About+Schools+and+Districts
/Map+of+Kentucky+School+Districts.htm
Kentucky Department of Education. (2003). CATS interpretive guide. Retrieved March 15, 2004, http://www.education.ky.gov/KDE/Administrative+Resources/Testing+
and+Reporting+/CATS/Accountability+System/2003+CATS+Interpretive+Guide.htm
Kentucky Department of Education. (2003). School and district profiles. Retrieved March 15, 2004, http://www.dpmr.kde.state.ky.us/district%20profile/mainpage.cfm
Kentucky Department of Education. (2003). CATS and NCLB electronic reports. Retrieved March 15, 2004, http://kdemaxport2.kde.state.ky.us
Kentucky Department of Education. (2003). Nonacademic data report guidelines. Retrieved March 15, 2004, http://www.education.ky.gov/KDE/Administrative
+Resources/Data+and+Research/Nonacademic+Data/Nonacademic+Data+Report+Guidelines.htm
Kentucky Department of Education. (2001). Demographic reports. Retrieved March 15, 2004, http://kdemaxport2.kde.state.ky.us
National Center for Education Statistics, U.S. Department of Education. Common core of data 2000-01. Retrieved March 15, 2004, http://nces.ed.gov/ccd/schoolsearch/
National Center for Education Statistics, U.S. Department of Education. CCD school and district glossary. Retrieved March 15, 2004, http://nces.ed.gov/ccd/commonfiles/
glossary.asp