Understanding Scale Scores
What is a scale score
The NZCER marking service uses scale scores to report student achievement on PAT:Reading (comprehension and vocabulary), PAT:Listening Comprehension, PAT:Mathematics and STAR (revised 2011). A scale score represents the conversion of a raw test score to a location on a described equal-interval scale designed to measure progress over several year levels. There are three separate PAT scales: the PAT:Mathematics scale, the PAT:Reading Comprehension scale and the PAT:Reading Vocabulary scale. The PAT:Mathematics scale, along with a sample of descriptors from different content categories, is shown below.
The process used to convert raw scores to scale scores takes into account the difficulty of the questions in the tests. This means that scale scores can be compared directly, regardless of which test in a series was originally administered.
Each PAT scale uses its own measurement unit to measure progress. For instance, the PAT:Mathematics scale uses a unit called “patm” (short for PAT:Mathematics unit. Each unit on a PAT scale represents the same amount of progress no matter where the student is located on the scale. For example, a student who increases from 20 to 30 patm units is considered to have made the same amount of progress as a student who moves from 50 to 60 patm units. This equal-interval property makes the PAT scales ideal for tracking student achievement over time. Students’ scale scores can be recorded from test to test to show their growing levels of knowledge and skill.
Comparing achievement with national norms
Once a raw score has been converted to a scale score it becomes possible to compare a student’s achievement with the achievement of national reference samples at different year levels. A scale score of 60 patm units, for instance, represents very high achievement for a Year 4 student (stanine 9), but represents below average achievement for a student in Year 10 (stanine 4). The NZCER Marking service allows the user to choose which reference year will be used for these comparisons. It is important to remember that data from the reference samples were collected in March.
The scale description
When each of the new reading and mathematics PAT tests was developed the relative difficulty of every question in each test was located on the appropriate scale. This allowed the scales to be described in terms of the knowledge and skill associated with questions at different scale locations. A student’s achievement can therefore be reported in terms of the knowledge and skill required to correctly answer questions that are located at or below the student’s own scale score location. Students are more likely to have developed the knowledge and skills described at and below their own scale location. The scale is defined so that a student whose scale score is at the same position as a particular set of questions is expected to answer correctly 50 percent of these questions, and far more than 50 percent of the questions located further down the scale.
Using the PAT scales to indicate curriculum levels
By assigning each question to a specific curriculum level it has been possible to work out where the different curriculum levels fall on the PAT:Mathematics and PAT:Reading Comprehension scales. The graphic of the PAT:Mathematics scale shown below displays these curriculum locations using shaded bars. It is important to note that questions from each curriculum level cover a section of the scale, rather than a fixed point, and that these sections overlap.
The curriculum level a student is working at can be estimated by comparing the student’s scale score with the display of curriculum levels. For instance, using the PAT:Mathematics scale on the facing page we can see that the middle of the Level 4 range is at approximately 60 patm units on the scale. This means a student whose scale score is 60 patm units is expected to be able to answer about 50 percent of the questions in PAT:Mathematics considered to represent Level 4 of the national curriculum. He or she will be expected to answer correctly a much higher percentage of the Level 3 questions, but a much lower percentage of the Level 5 questions. This student therefore could be considered to be capable of working at Level 4. A similar graphic showing curriculum levels for PAT:Reading Comprehension is presented in the PAT:Reading teachers manual.
Click on the report to view full size image