Comparison of Selected Texas Rural Systemic Initiative (TRSI) School Districts and Non-TRSI Districts on Student Academic and Nonacademic Performance Indicators

Prepared

for

The National Science Foundation Rural Systemic Initiatives Evaluation Study

The Evaluation Center

Western Michigan University

Kalamazoo, MI 49008-5237

by

Kenneth H. McKinley

April 2004

This material is based upon work supported by the National Science Foundation (NSF) under Grant REC-9819347. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the NSF.

Table of Contents

 

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .        iii

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .           1

Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .          1

            TRSI Districts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .          1

            Non-TRSI Districts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .          2

Description of Student Performance Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .         4

Data Retrieval and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .          5

Definition of Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .          6

Study Strengths and Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .          7

 

Findings on Texas Student Academic and Nonacademic Performance . . . . . . . . . . . .         8

            Elementary Mathematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .         8

            Middle Level Mathematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .      9

            High School Mathematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .         10

            A Summary of the Comparison of Mathematics Academic Performance . . . . .      11

            A TAAS to TAKS Mathematics Comparison . . . . . . . . . . . . . . . . . . . . . . . . . .        12

            Elementary Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .         13

            Middle Level Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .         14

            High School Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .        15

            High School End-of-Course Biology Exam . . . . .. . . . . . . . . . . . . . . . . . . . . . . .       15

            A Summary of the Comparison of Student Science Performance . . . . . . . . . . . .      17

 

Nonacademic Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .       17

            High School Attrition Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .      18

 

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .      19

 

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .     22

List of Tables and Figures

Tables

 

1.         Demographic Characteristics of TRSI and Matching School Districts . . . . . . .       3

 

2.         Texas Assessments of Academic Performance . . . . . . . . . . . . . . . . . . . . . . . . . .       4

 

3.         A Summary of Mathematics Academic Performance (TAAS): 1999-2002 . . . .       12

 

4.         A TAAS to TAKS Mathematics Academic Performance Comparison . . . . . . . .      13

 

5.         TAKS Elementary Science Academic Performance . . . . . . . . . . . . . . . . . . . . . .     13

 

6.         TAKS High School Science Academic Performance . . . . . . . . . . . . . . . . . . . . . .     15

 

7.         Summary of Science Academic Performance: TAAS (2002) to TAKS (2003) . .    17

 

8.         High School Attendance Attrition Rate: 1999-2000 to 2002-03 . . . . . . . . . . . . .       18

Figures

 

1.         Elementary Mathematics Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .    9

 

2.         Middle Level Mathematics Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .    10

 

3.         High School Mathematics Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   11

 

4.         Middle Level Science Performance (TAAS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   14

 

5.         High School End-of-Course Biology Performance . . . . . . . . . . . . . . . . . . . . . . . . . 16

Executive Summary

On behalf of the National Science Foundation Rural Systemic Initiatives evaluation study, The Evaluation Center at Western Michigan University contracted with the author of this report to conduct a study of the mathematics and science academic performance of students enrolled in selected Texas rural school districts that participated in the Texas Rural Systemic Initiative (TRSI) funded by NSF. Included in this report is a study of student academic performance in mathematics and science in 6 selected TRSI districts with 18 matching non-TRSI school districts, the 4 Texas regional education service centers (RESC) in which those 24 districts reside, and the performance of all students in the state on these variables.

Methodology

Six Texas school districts, nominated by the TRSI management team at West Texas A&M University as exemplary participants in the mathematics and science rural systemic collaborative, served as the primary study group for this research effort. Eighteen rural Texas districts, not members of the TRSI collaborative, were chosen for inclusion in the study and were matched with the TRSI districts on the variables of student enrollment, student body ethnic/racial composition, and percentage of economically disadvantaged students.

The primary data used for analysis in this study were the district pass rates in mathematics and science on the Texas Assessment of Academic Skills (TAAS) from 1999, the first year of the TRSI collaborative activities, to 2002 and on the Texas Assessment of Knowledge and Skills (TAKS), which replaced the TAAS in 2003. These data were downloaded from the Texas Education Agency (TEA) Web site. Weighted average pass rates were calculated for each group. Companion data for the Texas regional education service centers (RESCs) in which the TRSI/non-TRSI districts in the study reside and state data were added to the analysis to provide contextual perspective. A paired-samples t-test procedure was employed in an attempt to determine statistically significant differences between TRSI school districts and non-TRSI districts.

One nonacademic student performance indicator was also analyzed in this study. High school attendance attrition rates were calculated to determine a “school leaving” rate for each of the primary groups included in the study.

Summary

Students in the TRSI school districts showed steady gains in performance on the TAAS mathematics assessment over the 1999-2002 period at each level measured: elementary, middle, and high school. Most profound increases in average pass rates were experienced by the non-TRSI 10th grade group, which achieved a 14.9 percent pass rate increase over the three-year period from 1999-2002, followed by the TRSI 8th grade group, which increased by 12.9 percent.

The TAAS to TAKS pass rate decline, which was expected by state educators due to the change in the nature and scope of the test items, was most pronounced for the TRSI 8th grade group, which fell from a pass rate of 95.5 percent on the 2002 TAAS to 63.9 percent on the TAKS one year later. The magnitude of this decline was nearly matched by the non-TRSI 10th grade group, which experienced a weighted average decrease of 30 percent in mathematics academic performance.

The TRSI 5th grade group declined 13.3 percent on the TAAS to TAKS transition compared with 15 percent for their non-TRSI peers.

Texas introduced a new elementary science assessment with the introduction of the TAKS in 2003. The TRSI and non-TRSI study group school districts experienced failure (nonpass) rates of approximately 35 percent on this new statewide assessment, while the RESC and state groups at the same grade level fared somewhat better at ~25 percent.

At the high school level, student academic performance on the new TAKS science test was even more dismal, with a pass rate of only ~61 percent achieved by both TRSI and non-TRSI study groups. The emphasis in the new TAKS science assessments is clearly on inquiry-based learning, use of higher order thinking skills to solve relatively complex science-related real-world problems.

A review of the high school average daily attendance attrition rate showed a relatively large dimunition of student numbers when a cohort analysis of 9th to 12th grade attendance statistics are tracked from 1996-97 through 2002-03. Attrition rates on the order of 35-40 percent were found for all study groups. Several reasons for this phenomenon were offered by state educational officials: student academic credit accounting causes an inflated number at the 9th grade level; students drop out of high school and take the GED in lieu of the required end-of-course or high school graduation exams; and students transfer in and out of school, making the tracking and maintenance of accurate student attendance records difficult.

Comparison of Selected Texas Rural Systemic Initiative (TRSI) School Districts

 and Non-TRSI Districts on Student Academic and

Nonacademic Performance Indicators

On behalf of the National Science Foundation (NSF) Rural Systemic Initiatives evaluation study, The Evaluation Center at Western Michigan University contracted with the author of this report to conduct a study of the mathematics and science academic performance of students enrolled in selected Texas rural school districts that have participated in the Texas Rural Systemic Initiative (TRSI) funded by NSF. Included in this report is a study of student academic performance in mathematics and science in six selected TRSI districts, 18 matching non-TRSI school districts, the 4 Texas regional education service centers (RESCs) in which those 24 districts reside, and the performance of all students in the state on these variables.

Based on conversations with Dr. Jerry Horn, principal investigator of the NSF Rural Systemic Initiatives evaluation study at the Western Michigan University Evaluation Center, the following units of analysis were identified for study:

 

●          Mathematics and science performance on the Texas Assessment of Academic Skills (TAAS) at the elementary (5th grade), middle (8th grade), and high school (10th grade) levels for the years 1999-2002

 

●          Mathematics and science performance on the new and revised Texas Assessment of Knowledge and Skills (TAKS) at the elementary (5th grade) and high school (10th grade) levels for 2003

 

●          High school attendance attrition rate

Methodology

The first step in the research process was to select the TRSI member school districts for inclusion in the study. In the fall of 2003, the principal investigator, Dr. Horn, asked the NSF Texas Rural Systemic Initiative (TRSI) project administrative team located at West Texas A&M University, Canyon to nominate four to six TRSI member districts that had been participant members in the TRSI project since its inception in 1998-99 and had exhibited exemplary performance.

TRSI Districts

The TRSI project leaders nominated 6 participant school districts (see Table 1). These 6 TRSI-member districts represent less than 10 percent of the number of the member districts participating in the TRSI collaborative at its peak enrollment in 2002-03. To be eligible to receive TRSI services, an NSF Rural Systemic Initiative member district in Texas must have met poverty and population sparsity guidelines as promulgated under the United States Department of Agriculture (USDA) Beale Code; i.e., reside in a county with a poverty rate of 30 percent or greater and a population of 20,000 or less. According to the U.S. Bureau of the Census Web page , all 6 TRSI districts included for analysis in this study, except 1 that had petitioned for special admission to participate in the TRSI project, met the Beale Code guidelines.

Non-TRSI Districts

Three independent PK-12 school districts were selected in each of the 6 TRSI school district regions chosen for inclusion in this study. Thus, there were 6 TRSI districts and 18 non-TRSI districts or a total of 24 Texas independent school districts included in this study. The non-TRSI matching districts were chosen based on the following criteria:

 

●          They are located within a 50-mile radius of the TRSI school district.

 

●          They are most like the selected TRSI district in their region included in this study on the following variables:

 

            •          size: number of students enrolled (average daily attendance: PK-12)

 

            •          ethnic composition: percentage of minority (nonwhite) students

 

            •          economically disadvantaged: percentage of students reported by the district who received free or reduced price lunch(es)

Although it was difficult to find precise “matches” for the six TRSI study schools, an attempt was made to place the majority of the weighting on the latter two independent variables and allow size to vary somewhat within reason. Shown in Table 1 are the results of this selection process including state (Texas) system data and Regional Education Service Center (RESC) data. The Texas RESCs (N=20) are an administrative arm of the Texas Education Agency and provide a variety of services to local school districts: e.g., testing and assessment services for special needs student populations, curriculum planning and development consultation, and technology training and technical assistance. The state and RESC data are included in this analysis to provide further contextualization of the TRSI/non-TRSI data. As displayed in Table 1, the TRSI school districts and their “matching” non-TRSI cohorts reside in four different RESCs.

Table 1. Demographic Characteristics of TRSI and Matching School Districts

	TRSI Member			(in Texas)	Student Body
District/UnitName	Y	N	N/A	Location	Enrollment (PK-12)	% Minority	% ED
Texas			X	Statewide	4,059,619	58%	49.3%
RESC 02			X	SE-Coastal Plains(Corpus Christi)	107,634	71%	56.1%
Beeville	X			SE	3,933	76%	63.6%
Alice		X		SE	5,667	89%	65.1%
Pleasanton		X		SE	3,477	63%	56.8%
Sinton		X		SE	2,176	77%	62.4%
RESC 06			X	East-Piney Woods	135,913	33%	40.9%
North Zulch	X			East	343	4%	37.3%
Lovelady		X		East	528	16%	29.5%
Normangee		X		East	528	15%	37.9%
Bremond		X		East	463	20%	37.4%
Trinity	X			East	1,295	33%	58.0%
Grapeland		X		East	654	36%	47.1%
CS-Oakhurst		X		East	1,837	33%	55.2%
New Waverly		X		East	843	34%	46.7%
RESC 17			X	West-Plains(Lubbock)	79,121	57%	54.0%
Lockney	X			West	714	60%	51.4%
Abernathy		X		West	844	51%	50.9%
New Deal		X		West	710	45%	48.7%
Olton		X		West	766	71%	68.4%
Motley County County	X			West	210	34%	72.9%
Paducah		X		West	324	50%	67.0%
Silverton		X		West	256	38%	52.7%
Turkey.QuitaquQ		X		West	274	46%	63.1%
RESC 20			X	South-S.Antonio San	329,825	72%	61.6%
Charlotte	X			South	487	83%	78.0%
Cotulla		X		South	1,294	88%	76.4%
Dilley		X		South	887	88%	75.6%
Natalia		X		South	1,139	75%	69.6%

Data Sources: Snapshot 2002, School District Profiles, Texas Education Agency. www.tea.state.tx.us/ (March, 2004)

Description of Student Performance Data

The Texas Education Agency administers an annual assessment of student academic performance at various levels of the elementary, middle level, high school continuum. Shown below in Table 2 is a summary of the mandatory student assessments administered in Texas by subject and grade level.

Table 2. Texas Assessments of Academic Performance

The Texas Assessment of Academic Skills (TAAS) and the Texas Assessment of Knowledge and Skills (TAKS) were used as the primary data sources for measuring student mathematics and science academic performance in this study. The TAAS and TAKS are composed of a battery of open response and multiple-choice questions. The TAAS completed a 10-year run in 2002 in the nearly 25-year history of state-mandated, high stakes testing in Texas. After much research, development and field-testing, the TAKS was introduced in the spring of 2003. The TAAS was replaced by the TAKS in order to provide a better alignment of the testing program with the state curriculum standards, the Texas Essential Knowledge and Skills (TEKS), which were revamped in 1997-98.

The state board of education sets the raw score standard for passing and proficiency on each of the content-level tests administered in Texas. They also produce a Texas Learning Index (TLI) for each student, which is essentially a standardized scale score number that provides local curriculum administrators and classroom teachers with technical information about where a student’s instructional needs are most pressing and where students and school districts are relative to previous academic achievement and performance relative to peer groups. The student TLI can be “rolled up” to provide a campus (attendance center) and district TLI report.

The data included for analysis in this study is the “pass rate,” reported as a percentage and computed as the number of students passing a test divided by the number of students taking that test. While various measures can be used to interpret test results and student academic performance, the pass rate utilized in Texas is the single indicator of success used to assign campus and district accountability ratings.

The mathematics and science TAAS and TAKS data used in this study were retrieved from the Texas Education Agency (TEA) Web site . The analysis began with 1999, the first full year of operation of the Texas Rural Systemic Initiative (TRSI) and concluded with 2003, the latest year for which the data were available.

The high school attrition rate data for the four-year period 1999-2003 were also obtained from the TEA Web site. Average daily attendance totals for each grade for each school district are found there.

Data Retrieval and Analysis

Mathematics and science academic indicators (pass rates) at the elementary (5th grade), middle (8th grade), and high school (10th grade) levels for the TAAS for 1999, 2000, 2001, and 2002 and for the 2003 TAKS were downloaded from the TEA Web site for the 6 TRSI school districts, 18 matching non-TRSI districts, the 4 RESCs in which the TRSI districts reside, and the state as a whole. High school attendance data for the 9th through 12th grade classes were downloaded manually from the TEA Web site for each reporting period 1996-97 through 2002-03. This allowed for the analysis of the enrollment and matriculation of 4 ninth grade cohorts through the high school years:

 

●          1996-97 to 1999-2000

●          1997-98 to 2000-01

●          1998-99 to 2001-02

●          1999-2000 to 2002-03

The earliest that the data were available in the format needed for analysis in this study was for 1996-97.

The student academic performance data (total number of students taking each test and total number reported as passing) for each entity included in this study were entered into the Statistical Analysis System (SAS) package and subjected to a paired-samples t-test procedure, which produces a pairwise comparison of two population proportions. This procedure was deemed to be more sensitive to within and between group variances than a nonparametric analysis of the independence of group means (e.g., Chi square).

Definition of Terms

Acronyms and terms pertinent to reading and understanding the information in this report include the following:

 

●          RESC - Regional Education Service Centers. Numbering 20 in total, these entities are strategically located around the state of Texas, are staffed with specialists in various areas, and provide educational services to local school districts in such elements as testing, technology, curriculum, and school finance.

 

●          Scores - Pass rates on the statewide assessment of student academic performance for the various subject content areas at various grade levels. The pass rate is computed by the state for each campus (attendance center) and school district by dividing the number of students passing a given test at the raw score cut-off level predetermined by the state by the number of students taking the test. All quantitative data displayed in the figures and tables in this report are percentage pass rates unless otherwise indicated. Also, these scores represent weighted average scores for the TRSI, non-TRSI, and RESC groups while the state is, of course, reported as the whole population of test takers.

 

●          TAAS - Texas Assessment of Academic Skills. The statewide assessment of student academic performance employed in Texas from 1992-2002. See Table 2 for a summary of subject matter areas and grade levels.

 

●          TAKS - Texas Assessment of Knowledge and Skills. The new and revised statewide assessment academic performance that replaced the TAAS in 2003. See Table 2 for changes in the administration of the test.

 

●          TRSI - Texas Rural Systemic Initiative, housed and administered at West Texas A&M University, Canyon. TRSI is one of several National Science Foundation-funded collaboratives designed to provide assistance to rural schools in selected states and regions in the reform and improvement of mathematics and science education.

 

●          TRSI cohort - A selection of six TRSI participant school districts nominated by the TRSI management team as exemplary members of the collaborative. See Table 1 for a listing and brief quantitative description of the student populations in these districts.

 

●          Non-TRSI matching cohort - A group of 18 Texas school districts chosen for the purposes of this study. Matching criteria included proximity to a TRSI cohort school district, ethnic/racial composition of the student body, and the percentage of economically disadvantaged students (Table 1).

Study Strengths and Limitations

As with any research study, this analysis possesses a variety of strengths and limitations. First, a summary of the potential strengths:

 

●          The study cites data for multiple years. Trends over time are reported, giving a more comprehensive view and possible understanding of the mathematics and science academic performance in the TRSI selected schools and their “matched” peers.

 

●          1999 was chosen for the initial year in the analysis. This coincides with the first full year of the NSF-funded TRSI project in Texas and provides a baseline for analysis of the potential impact of the intervention.

 

●          Regional and statewide data are included to provide a perspective for the reader regarding the performance of the TRSI school districts.

A number of limitations also exist in a study such as this:

 

●          Student academic performance data reported for the newly introduced TAKS test was available only for the last year of analysis in this study and the first year of administration of the test, 2003. This allows for the possibility of the relative instability of the data and is the primary reason that the state did not issue campus and district accountability ratings in the first year after the introduction of the TAKS.

 

●          The inclusion of only six nominated TRSI districts in this analysis and its resulting small sample size invites measurement error, which may cloud the results and temper any conclusions regarding differences in performance when compared with matching districts, the region, or the state. This was a “purposive” sample. Thus, any inferences regarding the achievement of the larger TRSI collaborative should be made with caution. The six nominated districts were offered as exemplary performers in the TRSI collaborative. This may have been based on the observation and perception by the TRSI management team that the educators in these six districts were actualizing the terms of membership in the TRSI collaborative, initiating new programs for students, and engaging the communities (parents, citizens, organizations) in positive ways. In other words, the nomination of the six TRSI districts included in this study was probably most influenced by qualitative indicators of district success with the six NSF drivers of systemic reform and the TRSI program attributes and not quantitative indicators of student performance.

 

●          Caution should be exerted when making comparisons of scores (pass rates) within groups from year to year or between groups for each year. Small differences, especially in the smaller groups (TRSI-nominated districts and their peer matching groups), may be the result of measurement error, while large increments may be influenced by local events during the time of the test administration. As is the case for most states, the indicators reported here are the result of a single test (per subject) given on a single day.

 

●          Tests of significance are influenced by the relative appropriateness of their application to the data subject to analysis. A pairwise comparison of group Ns using a paired samples t-test to test for independence of the means between groups for each year and within groups between years was used to test for significance at p<.05. Descriptive data were used to discuss the nonacademic indicator, high school attrition rate.

Findings on Texas Student Academic and Nonacademic Performance

The major findings of this quantitative research study are presented in the following section. They include data on student performance on the state tests in the selected TRSI and non-TRSI schools, regional education service centers, and the state. These include elementary, middle level, and high school mathematics and science test performance comparisons. Also included are findings related to student performance in math and science on the former state test (the TAAS) as compared with performance on the new Texas state test (the TAKS). One nonacademic indicator is presented: high school attrition rate.

Elementary Mathematics

Shown in Figure 1 are the results of comparing the 5th grade mathematics performance in the TRSI cohort, non-TRSI matching school district cohort, the education services regions in which those 24 districts reside, and the state of Texas as a whole. As shown, in 1999, the first full year of the operation of the TRSI collaborative, the average score (pass rate) for TRSI school districts was slightly below the state and RESC schools. In the second and third year of the collaborative, the TRSI schools had pulled well ahead of the rest of the comparison groups. By 2002, the last year of the administration of the TAAS, all groups were nearly even. The difference in elementary school mathematics performance between the TRSI cohort and the other three groups for the 2000 and 2001 testing period was statistically significant (p<.05).

Middle Level Mathematics

As shown below in Figure 2, the TRSI 8th grade mathematics group started well behind its counterparts in terms of average passing score on the TAAS test in 1999. It ended the 2002 year, the last administration of the TAAS tests, at the head of the pack, having gained nearly 13 percentage points from 1999 to 2002. The most striking gains were made by the TRSI cohort school districts from 1999 to 2000, the second full year of the NSF-funded TRSI intervention in the state. However, this same group of school districts experienced the greatest decline in 8th grade mathematics performance when compared with its peers on the TAKS test. The TRSI school districts included in this study (see Table 1) fell more than 31 percentage points on the average 8th grade mathematics passing score while their non-TRSI counterparts declined just under 25 percent. The score difference between the TRSI cohort and its non-TRSI matching group was statistically significant (p<.05) in the first year on the TAAS (1999) and the last year of the analysis on the administration of the TAKS in 2003.

High School Mathematics

The TRSI cohort schools included in this study (Table 1) started the study period (1999-2003) noticeably behind their peers on the 10th grade mathematics test (TAAS). They had essentially “caught the pack” by 2001 but, for whatever reason(s), their growth trend stalled out in 2002. The decline expected in student academic performance due to the transition from the TAAS to the TAKS was not as pronounced in the TRSI cohort (~20%) as it was among the non-TRSI school districts (30%). In terms of statistical significance, the TRSI school districts were significantly below their non-TRSI peers on the TAAS in 1999 and 2000 and above them in 2003 on the new TAKS (p<.05). How much of this difference can be accounted for by sample size is unknown

A Summary of the Comparison of Mathematics Academic Performance

A comparison of groups’ mathematics scores on the TAAS for the three grade levels over the four years included in this analysis (1999-2002) is displayed in Table 3 below. The TRSI cohort elementary students (5th graders) made the largest increase from 1999 to 2000, the first full year of the introduction of the TRSI collaborative and generally held the lead among the peer-group comparisons through the four years. The 8th grade TRSI cohort test takers started lower than their peers, again made fairly pronounced gains in the second year and held their leading position through the rest of the four-year analysis. The 10th grade TRSI cohort test takers made their most noticeable gain from the second to the third year of the analysis period (2000 to 2001), then interestingly achieved the same score in 2002.

No tests of statistical significance were run on these data. It is, however, interesting to note that the high school students scored lower than did their younger counterparts at the elementary and middle school levels for all groups and test periods (years).

The largest gains in terms of absolute average mathematics passing scores when using the starting point of 1999 as the base were the non-TRSI 10th graders (14.9%), the TRSI 8th graders (12.9%), and the RESC cohort 10th graders (11.1%).

Table 3. Summary of Mathematics Academic Performance (TAAS), 1999-2002

	1999			2000			2001			2002
	5th	8th	10th	5th	8th	10th	5th	8th	10th	5th	8th	10th
TRSI	89.3%	82.9%	78.5%	96.1%	89.5%	80.9%	97.0%	93.7%	88.8%	96.4%	95.5%	88.8%
Non-TRSI	87.4%	86.8%	78.5%	88.2%	89.4%	83.7%	90.8%	92.3%	89.3%	95.0%	93.4%	93.4%
RESC	89.3%	85.5%	80.6%	93.3%	89.6%	85.6%	94.6%	90.8%	88.5%	96.2%	90.8%	91.7%
State	90.1%	86.3%	81.6%	92.1%	90.2%	86.8%	94.6%	92.4%	89.3%	96.2%	92.9%	92.2%

A TAAS to TAKS Mathematics Comparison

A comparison of the initial year of the administration of the TAKS mathematics test scores (2003) compared with the last year of the TAAS (2002) is shown in Table 4 below. As expected, fairly dramatic declines in performance were noted for each grade level and group. The most precipitous decline was experienced by the TRSI 8th graders (31.6%) followed by the non-TRSI 10th graders (30%). The least decline was achieved by the state’s 5th grade students. The TRSI cohort scored better than its non-TRSI peers at the 5th and 10th grade levels.

No statistical tests of significant differences were run on these data because of the vast difference in sample sizes and the potential instability of the data with regard to only one year of administration of the new TAKS test.

Table 4. A TAAS to TAKS Mathematics Academic Performance Comparison

	5th Grade		8th Grade		10th Grade
	2002-TAAS	2003-TAKS	2002-TAAS	2003-TAKS	2002-TAAS	2003-TAKS
TRSI	96.4%	83.1%	95.5%	63.9%	88.8%	68.7%
Non-TRSI	95.0%	80.0%	93.4%	68.9%	93.4%	63.4%
RESC	96.2%	82.8%	90.8%	65.3%	91.7%	68.9%
State	96.2%	86.3%	92.9%	73.2%	92.2%	74.2%

Elementary Science

Spring 2003 marked the first year of the administration of an elementary grade level science test in Texas with the introduction of the TAKS. At the same time, the state discontinued science testing on a mandatory statewide basis at the middle (8th grade) level. Displayed in Table 5 are the results of the performance of the TRSI study group 5th graders and their non-TRSI, RESC, and state peers. The TRSI and non-TRSI matching group school 5th graders were virtually even in this first administration of the TAKS science test, while the score (pass rate) for the RESC school districts and the state were approximately ten percentage points higher.

Table 5. TAKS Elementary Science Academic Performance

Middle Level Science

Shown in Figure 4 are the results of the performance of the TRSI cohort 8th graders on the TAAS science test. As stated above, this test was discontinued at the 8th grade level upon the transition from statewide mandatory testing in Texas under the TAAS to the TAKS (see Table 2). TRSI 8th graders started well behind their study peers in 1999, but actually caught and surpassed them in 2001 only to slip back in 2002, the last year of the administration of the TAAS. The most noticeable gains in the 8th grade science performance of the TRSI study cohort were achieved between 2000 and 2001 (8.5%). The difference in the (weighted) average passing score between TRSI and non-TRSI cohorts was statistically significant (p<.05) only in 1999.

High School Science

Since it discontinued the 8th grade science test requirement, Texas “moved” the next mandatory assessment of student science knowledge and skills to the spring semester of the 10th grade with the implementation of the new TAKS test. Most schools require some combination of biology I and a hybrid physics and chemistry course (PCS) in the first two high school grades (9 and 10) as preparation for the new 10th grade TAKS science test. Shown in Table 6 are the results of the first statewide administration of the high school science test. The TRSI cohort school districts and their matched non-TRSI peers fared relatively poorly on this new exam, with a pass rate in the low 60 percent range. The RESC and state results were slightly better, but still found fewer than 70 percent of these larger groups of students passing.

Table 6. TAKS High School Science Academic Performance

High School End-of-Course Biology Exam

Texas concluded the administration of the TAAS in 2002. The state also terminated the use of end-of-course exams, one of which was the biology I statewide test, which was required for graduation and receipt of a Texas high school diploma in good standing. Shown in Figure 5 is one more display of the performance of the TRSI study cohort schools and their peer groups on this test for the period 1999-2002. The pass rate on this end-of-course exam rose for all groups for the first three years of the study period, then receded, however slightly, in 2002. The TRSI schools trailed their peers throughout the four years of testing, even though they gained slightly in terms of their pass rate, peaking in the third year at 77.5 percent. For whatever reasons, the RESC school districts led the comparison groups on this indicator throughout the four-year study period. The difference in performance between the TRSI and non-TRSI cohorts was not statistically significant for any of the years analyzed (p<.05).

                                                                                                                                                        

A Summary of the Comparison of Science Academic Performance

Shown below in Table 7 is a summary of the science academic performance of students in the study comparison groups on the TAAS and TAKS. As previously mentioned, Texas discontinued the 8th grade science TAAS in 2002 and introduced the new TAKS at the 5th and 8th grade levels in 2003. As revealed in previous figures and in the table below, the relative pass rates on science academic performance declined rather dramatically from the old test administration to the new. This was expected by educators in the state, but expectations regarding the extent of the decline are unknown to this author.

Whereas the scores on the 2002 TAAS were relatively equal across the study groups, the TRSI and matching school district cohort scores dropped rather precipitously when making TAAS to TAKS comparisons between grades; and while there was a comparable decline for the regional and state study groups, it was not as dramatic as for the TRSI/non-TRSI groups. This may be explained in part by the disparity in sample size.

The difference in scores between TRSI and non-TRSI comparison groups was not statistically significant for any of the three grades levels for the two different test administrations (TAAS/TAKS) (p<.05).

Table 7. Summary of Science Academic Performance: TAAS (2002) to TAKS (2003)

 

Nonacademic Indicators

A variety of nonacademic indicators of the performance of the Texas public schools are reported in the Academic Excellence Indicator System (AEIS) found on the Texas Education Agency Web site . Such indicators as attendance rates, 4-year dropout rates, and graduation rates with respect to the reported student body profiles are reported but do not reveal a great deal of variance across school districts.

High School Attrition Rate

One statistic that does intrigue this author is the attrition rate of the high school class cohort, i.e., 9th-to-12th grade “leaving rate.” As shown in Table 8 below, when one tracks the ninth grade population through to the completion of that cohort in the 12th grade, there is a noticeable loss of head count. Attrition in high school head count ranges from a low of 31.6 percent in 2002-03 for the non-TRSI group to a high of 40.3 percent for the TRSI study group. This means that of the total number of 9th graders enrolled in the TRSI study group school districts in 1996-97, fewer than 60 percent were still in school as high school seniors in 1999-2000. Although there was slight improvement in the four-year trends across groups studied, the high school attrition rate remains relatively high at the local, regional, and state levels.

Table 8. High School Attrition Rates: 1999-2000 to 2002-03 

							2002-03 vs. 4-Yr.Ave.
	1999-00	2000-01	2001-02	2002-03	4-Yr.Ave.	3-Yr.Chnge
TRSI	40.3%	37.2%	40.4%	38.6%	39.1%	1.2%	0.5%
Non-TRSI	35.2%	34.9%	32.6%	31.6%	33.8%	2.2%	2.2%
RESC	35.2%	35.7%	35.6%	33.3%	35.0%	0.3%	1.7%
State	36.7%	36.8%	35.6%	33.8%	35.7%	1.0%	1.9%

Reasons for this seemingly and alarmingly high attrition rate are varied and somewhat complex :

 

1.         The relatively high (starting point) enrollment numbers at the 9th-grade level may inflate the attrition rate percentages. Many school districts retain students at the 9th grade until they have completed a set number of academic credits. Secondly, if the students were officially enrolled as 10th graders, they were required to take the end-of-course subject matter exams, the performance on which are counted in the campus and district state accountability ratings.

 

2.         Some students opt to simply drop out. A subset of that population then takes the General Education Development (GED) test as evidence of completion of a high school education. A successful GED score is accepted by Texas for meeting the high school completion requirement for admission into the state higher education system.

 

3.         Students who appear to have dropped out of the high school system may, in fact, transfer to another district within the state education system or to a school in another state. Thus, they are in school, but may be counted as dropouts.

Another related point is that the state reports an official four-year longitudinal dropout rate. It is much lower than the data reported here. The official computation of the state dropout rate is “the number of students in the cohort whose final status is a dropout divided by the final number of students in the cohort after four years, allowing for in-and-out migration.”

Summary

Texas administers an annual statewide assessment of student academic achievement in the subjects of mathematics, reading, science, social studies, and writing. The state introduced a new, more rigorous assessment program in the spring of 2003, the Texas Assessment of Knowledge and Skills (TAKS), replacing a testing program known as the Texas Assessment of Academic Skills (TAAS), which had been in place since 1992.

The TRSI cohort academic performance in mathematics, as defined by the pass rates on the TAAS, increased steadily for the four-year period of analysis, 1999-2002. Performance by all groups of students at all three levels increased in the last four years of the administration of the TAAS. The most profound increases in weighted average pass rates were achieved by the TRSI elementary and middle level students (5th and 8th grade, respectively) in the second full year after the NSF-funded TRSI collaborative was initiated in the state (1999-2000). Interestingly, a similar increase was exhibited by the TRSI high school students (10th grade) on the TAAS mathematic test in the following year, 2000-01. TRSI students at all three grade levels leveled off somewhat in their mathematics academic performance in the last year of the administration of the TAAS, 2002.

When comparing the academic performance of all groups on the new, revised, and predictably more rigorous TAKS mathematics test first introduced in 2003, all groups declined in terms of the average pass rate. The biggest decline among TRSI test takers were the 8th graders (31.6%) and the non-TRSI 10th grade students (30%).

In the area of science academic performance assessment, the state of Texas completely restructured its student academic performance assessment program with the transition from the TAAS to the TAKS. Science testing at the 8th grade level was eliminated, and a new elementary science test was introduced at the 5th grade level. At the high school level, the state eliminated the end-of-course examination program and instituted a graduation test requirement beginning with the 10th and 11th grades. Thus, 10th grade science testing under the TAKS was introduced in 2003. This test is designed to measure student knowledge and skills after taking two required high school science courses which, for most districts, means introductory biology and some combination of physics and chemistry.

Student academic performance on the 5th grade science test ranged from a pass rate of 65.9 percent for the TRSI cohort students to 76.7 percent for their 5th grade peers in the four regional education service centers included in this analysis.

While there are no direct grade-to-grade science academic performance comparisons, students in the TRSI cohort declined slightly less than did their non-TRSI counterparts in the transition from the 8th grade science TAAS to the 10th grade science TAKS (29.3% to 31.2% respectively). Both of these groups were below their RESC and state peer groups on these transition measures.

The TRSI middle level science test takers made substantial gains on the TAAS science test between 2000 and 2001. This represents a period two-and-one-half years into the introduction of the TRSI collaborative. The TRSI group actually led the other three groups: non-TRSI, RESC, and state on the middle level science, however slightly, in 2001. However, they relinquished that lead, again slightly, in 2002, the final year of the administration of the TAAS.

On another indicator of student science academic performance, the TRSI group trailed its peers all four years in the analysis of the TAAS end-of-course biology exam. This test was taken by approximately 25 percent of the grade-eligible students in any given year. It was eliminated along with other end-of-course tests in mathematics, social studies, and history with the completion of the administration of the TAAS in 2002.

On the one indicator of nonacademic student performance, the attrition rate for high school attendance was analyzed. This measure was computed by tracking 9th grade cohorts through the 12th grade over four years beginning in 1996-97, the first year the data were made available electronically. Attrition rates approaching 40 percent were discovered when comparing the 9th grade attendance rate with that same cohort three years later in the 12th grade. Several reasons were given:

 

●          state and local school district policy regarding the completion of adequate academic credits at the 9th grade (which may inflate the head count at that level)

 

●          dropouts who do not wish to complete high school because of the graduation test requirements

 

●          students who transfer between schools or in and out of state

This problem is more pronounced in school districts with higher proportions of minority students. The issue has also caused the state to revise its accountability rating model to include a four-year graduation rate measure in lieu of a dropout statistic.

In final summary, it appears that the TRSI intervention made some noticeable impact in student academic performance, especially in mathematics after one and two years of programming. Now that the testing program in Texas has been revised, it may take two or more years of statewide assessment to more fully understand the impact of the TRSI program on future student academic performance. The TRSI cohort chosen for analysis in this study was a purposive sample, not randomly selected. Thus, inferences about the performance of the population of students enrolled in the school districts participating in the TRSI collaborative should be made with caution.

References

 

Shavelson, R. J. (1996). Statistical reasoning for the behavioral sciences (3^rd edition). Boston: Allyn and Bacon.

 

Texas Education Agency. (2003). Snapshot 2002: School district profiles, 2001-02. Austin: Author.

 

Texas Education Agency. (2004). Academic Excellence Indicator System reports. Retrieved March 22, 2004, http://www.tea.state.tx.us/perfreport/aeis/index.html

 

West Texas A&M University. (2004). Texas Rural Systemic Initiative (TRSI). Retrieved March 31, 2004, http://www.texasrsi.org/

 

U.S. Bureau of the Census. (2004). State and county quickfacts. Retrieved March 22, 2004, http://www.quickfacts.census.gov/qfd/