An Introduction to Value-Added Analysis

For the last 3 years, more than 80% of the respondents to the Annual Phi Delta Kappa/Gallup Poll of the Public’s Attitudes Toward the Public Schools have stated that they would rather see a school’s performance measure based upon “improvement shown by students” than the “percentage passing the test” (Rose & Gallup, 2007, p. 35). If this were to become the norm, the next question would be what “improvement” is signifi cant? Educators need to understand “valueadded” if they are going to use “improvement” to show that schools are improving student achievement.


H
summarize that "indicators continue to suggest that Catholic elementary and secondary school students, on average, outperform their public school counterparts" (p.53).Unfortunately, this statement tells us little about the impact our schools have on infl uencing student achievement.Do Catholic schools have students with higher ability, greater parent support, and the other factors that infl uence a single point in time comparison of student achievement or are our students really achieving at a higher rate?
As the federal government considers the reauthorization of No Child Left Behind (NCLB) legislation, there is a realization that emphasis on profi ciency is not creating the desired level of increased student achievement.This creates both opportunities and challenges for Catholic schools.Under NCLB, parents of children in failing schools are now being given opportunities to choose a "better" school for their children.Whether Catholic schools follow state standards or not, the challenge is to provide evidence that Catholic schools are better able to maximize student achievement.
The federal government is in the process of piloting the impact of a growth model in 9 states, requiring schools to demonstrate necessary profi ciency and increased learning for all students in order to meet the goals of NCLB.The common element among the approved states is that they use a growth model approach or value-added.Value-added models were fi rst applied to school analyses by William Sanders at the University of Tennessee.The approach uses test data to measure "growth," or "value-added," meaning a student's improvement from one year to the next.The results are then aggregated at the teacher, grade, and school levels.
Past reporting of standardized test achievement results has concentrated on comparing buildings and systems: the student scores in one school to another school, Catholic schools to public schools, states to other states, and other types of comparisons.Previous to value-added, a school or a system could look good on average, but might not meet the needs of many individual students because comparing averages does not indicate if a student is achieving growth from one year to the next.The focus of value-added "for measuring student achievement appeals to administrators and policymakers at all levels of education because they quantify the gains that students make from one school year to the next rather than relying on reporting percentages of students who get passing or profi cient scores at the end of the school year" (Viadero, 2008, p. 12).
In 1999 the Archdiocese of Indianapolis, as part of a larger project, became interested in teacher improvement and began using value-added assessment for teacher performance-pay.The archdiocese received a school improvement grant from Lilly Endowment Incorporated and with matching funds raised by an archdiocesan corporate and business campaign, the Offi ce of Catholic Education implemented many school improvement initiatives, one of which was the Teacher Advancement Program (TAP) developed by the Milken Family Foundation, which is "dedicated to attracting, developing, motivating and retaining high-caliber educators in order to raise achievement levels for all students" (National Institute for Excellence in Teaching, 2008, Mission Statement).TAP began in 4 pilot schools in the fall of 2002 and expanded to 13 schools in the 2006-2007 school year.The performancebased pay amounts are calculated using a formula that includes scores from teachers' evaluations, value-added gains made by the students in the teachers' classrooms, and the school wide value-added gains.The archdiocese has used value-added analyses for 5 years to measure these gains and to calculate the performance pay for teachers in 13 TAP schools.There have been four performance-based bonuses awarded to archdiocese TAP teachers and principals partially based on these value-added gains.In using the data it became apparent that it also provides the archdiocese with a tremendous tool for examining student achievement and using data to drive school improvement decision making.Beginning in February 2007, the Offi ce of Catholic Education had value-added reports calculated on all of our students in order to assist schools in improving their programs and making data-driven decisions.The reports were calculated on over 13,000 students and in 69 archdiocesan schools, which include urban, suburban, and rural demographics.
Value-added assessment can demonstrate the impact Catholic schools have on student achievement.The purpose of this article is to explain value-added and how it is different from comparing student achievement as a single point in time on an annual basis; how schools in the Archdiocese of Indianapolis use value-added analysis to improve teaching and learning; and why it should be used.

Value-Added and How It Is Different from Single Point in Time Achievement Data
Value-added is a statistical method for computing the gains or progress students make in a pretest-to-posttest scenario (e.g., from one testing cycle to the next).If only a single point of measurement is used versus a pretest-to-posttest measurement, then there is no determination upon the amount of gains a student has made during a specifi ed time frame.These value-added gains can be attributed to the school's impact on student learning, because they are not affected by external characteristics, such as students' family or neighborhood.These gains tell schools the impact they have on students who attend their institutions regardless of students' level of achievement or measured proficiency.By using statistical models that combine the gains of numerous students, one can calculate the gains attributable to classroom teachers, schools, districts, and even entire states.
In previous profi ciency models, student achievement data is treated as an annual event because it is not linked back to past student achievement.Through more than 20 years of research, Sanders and Rivers (1996) found that classroom teachers are the most important factor in achieving gains in student achievement.But Jencks and his colleagues (1972) found race, poverty, parental education, family structure, family interaction, parenting skills, and neighborhood factors account for 80 to 90% of that achievement.Jencks concluded, "the character of a school's output depends largely on a single input, namely the characteristics of the entering children" (p.256).According to David Berliner (2006), "if the educational opportunities available to White students in our public schools were made available to all our students, the US would have been the 7th highest scoring nation in mathematics, 2nd highest scoring nation in reading, and the 4th highest scoring nation in science" (p.969).Relying on single point in time standardized tests, therefore, refl ects more the students' background than the schools' impact.Furthermore, if we only look at whether a student is above or below a set of profi ciency expectations, then we miss the progress that the student is making over time.Figure 1 is an example of achievement for two students.Looking only at student profi ciency, one can assume that student B was performing better than student A until the 9th grade when their performance level is the same.At the 10th grade, student A is performing better than student B. In this scenario if individual student growth or progress is not monitored, then increases for the achievement of student A and the decline for student B over the years would have been missed.

How Value-Added Works
In order to determine effectively the impact of the school or teacher on a student's learning, individual student achievement data must be matched from one year to the next, then an expected level of growth is calculated based on the sample.The sample can consist of a district, multiple districts, a state, or even a nation.There are several needed conditions to compute value-added estimates: 1.Each student must have two consecutive years of test data from a reliable and valid test (3 years is ideal) which is correlated to expected standards for the student.Remember, value-added analysis is the longitudinal growth in achievement of the same set of students over time.2. Each student's test scores must be linked to his or her teacher.When more than one teacher teaches a student, as is the case in most middle and high schools, teachers need to be linked to the student's achievement in the subject area they teach.3. Districts and schools must use tests with suffi cient vertical range between the scores from one year to the next to measure growth (any state test or normreferenced test meets this criteria).
Value-added statistics use mathematical modeling techniques to make the results more accurate and reduce the error associated with test scores.Valueadded gains are estimations, not exact calculations, of the changes from the pre-and post-assessments for the students in question.Estimates are used because often there is data missing due to absence during testing, change in schools, and other various reasons.The value-added approach uses statistical models to account for missing data to make precise estimates of student gains.In addition to computing these for districts and schools, these estimates can be computed on other variables, including race, gender, free lunch eligibility, Limited English Profi ciency status, special education status, and others.These student-level variables can show how these variances impact the changes in student achievement.
To compute a value-added estimate, it is best to assemble a large and diverse reference group of students and calculate their gains to serve as the context in which one interprets all gains.This group may be all the students in a school, district, state, or a sample of students across the nation, as is the case for norm-referenced tests.The size, scope, and geography of the reference group will affect value-added estimates.If a reference group only has 10 schools in it all from the same district, it is far less diverse and representative than a reference group of 1,000 schools from across the state.To reiterate, the reference group provides a context for comparison, so when the context changes, so do the estimates.

How the Archdiocese of Indianapolis Uses Value-Added Results
The Archdiocese of Indianapolis has received value-added reports for the last 5 years.The analyses were fi rst conducted by the Education Value Added Assessment System (EVAAS) Institute in North Carolina developed by William Sanders using the Terra Nova test and then the Indiana Statewide Test of Educational Progress (ISTEP).The diocese then contracted out its value-added assessment with Teaching Doctors using its value-added analysis network software.The archdiocese found the Teaching Doctors' Value-Added Analysis Network's reports were easier to understand and use.In addition, their reports explain how the calculations are determined (see Thum, 2003).
Gain reports can be developed for all tested subjects (language arts, mathematics, science, etc.), and for each subtest within a subject (e.g., reading comprehension, literature response, writing, etc.), so districts and schools can make more focused instructional and planning decisions.Figure 2 is an example of how gains are reported to teachers and administrators using reports from Teaching Doctors.
A zero (0) percent gain means that the district, school, or teacher's students gained no more or less than the representative sample.The bar is white when the average gain is above 0 and gray when it is below.The length of the bar represents all the scores that fall within a 75% certainty level of the mean reported gain.This provides an estimated gain of improvement based upon a range of scores that is not distorted by the extreme scores either side of the mean score.The center of the white or gray bars represent the average percent gain above the district average; the ends of the bars show the spread or confi dence bands around the gain.For math, this example shows that the school's average percent gain was -30, or 30% less than the representative gain.The range of the gain had a low of -70 and a high of -10.The fi nal issue is how the interpretation of reports are used to change practices in our schools.
Value-added reports are made available to archdiocesan schools in an electronic web format.This includes district-, school-, and grade-level reports  To have signifi cant positive gains, the school range of estimates had to be above the 0 line.The estimated average gains are based upon all students in the Archdiocese of Indiana schools in grades 3 through 10, which is slightly more than 13,000 students, who took the Indiana standards-based assessment (ISTEP).Any range of scores touching the 0 line would be average gains whether the bar was white or gray, and ranges of scores below the 0 line are negative.The smaller the range of scores shown in the bar around this mean (length of the bar for each school), the greater the consistency of the school on impacting the growth of students.In the archdiocese, 18 schools had value-added gains.
Teaching Doctors reports are further disaggregated by creating schooland grade-level reports.Figure 4 displays a school level report by grade level for the two subject areas-English language arts and mathematics.English language arts has 6 subtests, and mathematics has 7 subtests.
In the example for English language arts, a fourth grade-level report for one school, the highest gain score is in reading vocabulary, which has a value-added range above the 0 line.The lowest is in reading comprehension, which has a mean below the 0 line, but the total range for English language arts is not signifi cantly different because the bar touches the 0 line, which means student achievement gains at the subject level is not different than average gains estimated for the total population taking the standards-based assessment at this grade level.Therefore, value-added gains had not been achieved for this school in English language arts.All of the mathematics scores are positive and above the 0 line and are value-added gains.Similar reports are created at the individual grade levels to analyze student growth further.
Making interpretations based on these value-added reports is a relatively straightforward process.In addition to viewing value-added reports by tested subjects at the district, school, grade, and teacher levels, further sub group reports are available by ethnicity; free and reduced lunch; high, middle, and low achievers; English language learners; special education and other student populations.This information on specifi c populations will allow school leadership teams and teachers to plan for the specifi c needs of student populations.It must be kept in mind that the purpose of attaining value-added reports is to help administrators and teachers make informed and accurate decisions that can signifi cantly impact classroom instruction and learning.Value-added analyses can assist schools in prioritizing when creating improvement plans, because they provide data to inform responses to the following questions: 1. On how many areas for improvement should we focus? 2. Do the areas require a district-, school-, grade-, or teacher-level focus? 3. What is a reasonable target for improvement in each area?
For each area in need of improvement that a school identifi es, it is important also to analyze the reasons behind the results.One area for analysis of reasons is curriculum alignment and breadth of instructional content.Students must be provided opportunity to master the content standards on which they will be assessed.This assumption is based upon the belief that the assessments are aligned with the content standards that students should posess.This alignment should not be taken for granted, but should be monitored as value-added results become a part of curriculum alignment.Another area for analysis is instructional and assessment strategies.Are the students being taught effectively, are groups of students getting the content and learning strategies they need to be successful?Finally, are time allocations adequate?How much teaching time is devoted to this area and is the use of time effi cient and effective?

Why Schools Should Use Value-Added
Value-added reports should be used in making decisions about our improvement efforts in schools because now there is a measure of the expected level of improvement, not just a guess that the improvement is adequate.These estimated gain scores are a more accurate assessment of student growth because they provide a pre-and post-assessment for each student, which eliminates the assessment error caused by single point in time comparisons with missing data for students who did not take both assessments.The value-added data allows for decision making based on whether students are making gains in specifi c areas to determine priorities in regard to professional development and generates data to inform responses to questions such as: What are the new strategies, methods, approaches, and materials needed to improve teaching and student learning?How should schools and dioceses decide on the need and allocation of current resources (texts, computers/software, specialist teachers, after school programs, etc.)?What are the needs of different populations of students and what is the best way to reach them (ELL versus non-ELL, high versus low achievers, free lunch versus not free, minority versus majority)?It is a matter of Catholic social teaching; meeting each student's needs is critical to each student's future success.
Value-added analyses provide administrators, teachers, and schools with talking points to fashion school improvement efforts.They measure school and teacher impact on students' learning, not family and neighborhood impact.Value-added results help schools more accurately determine where they are, and aid them in crafting informed plans for where they want to go this year and in subsequent years.In the end, it will allow the school to determine whether they made a year's worth of value-added growth in student achievement.It is what every child deserves.

Concerns about Value-Added
As with any model there are proponents and opponents.Proponents argue, as we do, that without a sophisticated method of comparing student pre-and post-achievement, real changes in student achievement go unnoticed, unmeasured, and unexamined.Opponents criticize that value-added "has fl aws that must be addressed…[including] a shortage of external reviews and validity studies of the model, its insuffi cient user-friendliness, and methodological issues about missing data, regression to the mean, and student background variables" (Amerin- Beardsley, 2008, p. 65).It is beyond the scope of this paper to settle this debate, rather we argue that in our experience, value-added has allowed us to identify more clearly areas where students need to improve and determine whether student achievement has improved after implementing interventions to address those areas.We will continue to monitor our decisions and their impact based on the use of value-added data.

Conclusion
The need for Catholic schools to validate that they make signifi cant (valueadded) gains on student achievement is great.As stated in the Fordham B. Institute report Who Will Save America's Urban Catholic Schools (Hamilton, 2008), since 1990 over 1,300 Catholic schools have closed, displacing more than 300,000 students into public schools.If this trend is going to be reversed, Catholic schools need to overcome the "nostalgia and face these problems head-on" (p. 6).One of the recommendations is to "promote efforts to collect data, foster transparency and astutely 'market' Catholic schools" (p. 1).In the Fordham study, they could not cite one Catholic school system that would have been willing to be that "transparent."Value-added promotes that transparency.If we are going to delve deeper in trying to demonstrate that our Catholic schools do make a difference in improving individual student achievement, then we need more sophisticated methods of determining growth to ensure that we are not missing the real reason for growth.
There are those who would argue that we really cannot measure student growth.Furthermore, there would be those who would state that this does not apply to Catholic schools.In this era of increased accountability and greater parental choice in selecting the school their child attends, we need to be thinking about how we can prove that our schools make a difference in maximizing student achievement.

Figure 2 .
Figure 2. Value-added gain report for a school on mathematics.
Catholic schools in the archdiocese.Administrators and teachers review the reports and use the data to inform and improve instruction.The district-level chart from Teaching Doctors below demonstrates each school's mathematics gains.Each bar represents one school.White bars are positive gains, gray bars are negative gains.School names are removed to maintain anonymity, but one can see from this report how one could look across all schools, and quickly assess their gains in the tested subject.The district report had 15 tests and subtests.Only mathematics gains are shown here.

Figure 4 .
Figure 4. Fourth grade value-added report by subject area.