e-asTTle – Reference material

The deep and surface scores which are used in e-asTTle Reading, Maths, Pānui, and Pāngarau are based on the SOLO taxonomy. SOLO stands for "Structure of observed learning outcomes".

The taxonomy has five levels to describe the quality of student responses. In e-asTTle, prestructural was removed and the remaining four levels have been grouped into two levels – surface and deep.

Summary of the taxonomy in e-asTTle

• Unistructural – one aspect of a task is picked up or understood serially, and there is no relationship of facts or ideas.
• Multistructural – two or more aspects of a task are picked up or understood serially, but are not interrelated.
• Relational – several aspects are integrated so that the whole has a coherent structure and meaning.
• Extended Abstract – that coherent whole is generalised to a higher level of abstraction.

Each question in e-asTTle has been assigned a level from the SOLO taxonomy. This has then been further classified into either deep or surface.

There are examples in the table.

Looking at the surface/deep ratio in a test

When you create a test, you can see the number of questions that are surface or deep on the Test Summary. This is listed under the Cognitive Processing heading.

Image description: A screenshot of a Pānui test summary. Five headers separate the information: Test Identification, Curriculum Strand, Curriculum Level, Cognitive Processing, Slider Settings. The Cognitive Processing is circled in red.

Reporting on surface/deep
 

You can look at students’ surface/depth scores in three ways.

Image Description: Two report types. The first is an Individual Learning Pathway report showing three dials at the top and a table underneath. The table has the column headings: Overall, Surface. Deep, Processes and Strategies, and Ideas. The row headings are: This student. Level, and Year 9 mean. The first 3 columns are circled in red. The second report type is Console and Console-Comparisions. This shows a box and whisker graph with the main heading Depth of Thinking at the top.

Image Description: A tabular report. The column headings are Gender, Language, Ethnicity, Overall Score, Overall Level, Surface Score, Surface Level, Deep Score, Deep Level. The last four columns are circled in red.

This section explains some of the underlying test theory used in e-asTTle.

Comparison to conventional testing

In classical test theory, percent correct is used for scoring. An issue with this approach – which the Rasch model attempts to solve – is that the difficulty of the test isn't considered.

Consider two reading tests. The first has 32 questions, all at curriculum Level 2. The second has 24 much harder questions, entirely at curriculum level 4. The easy test is given to Tohu at the beginning of the year, and the harder test, at the end of the year. Tohu got all questions right on the first test, but only eight questions correct on the second. Has Tohu improved?

Using classical test theory, Tohu gets a 100% score at the beginning of the year, and 33% at the end of the year. Of course, it is not reasonable to use the percentages to assess Tohu’s progress. It is obvious the tests are not comparable, and when using percentages, there is no way of working out what progress Tohu has made. The Rasch model provides a score for each test, which is still essentially based on percent-correct – however, it is adjusted for the test’s difficulty. These scores can be used for comparison, even across test papers that are very different.

e-asTTle uses the Rasch model to assign difficulty values to test questions and equivalent ability levels to students. The difficulty values are later translated to curriculum levels.

Probabilities and the Rasch model

The Rasch model is about probabilities. It is based on the idea that we can model how likely it is a student responds correctly to a test question based on the difference between test question difficulty and student ability. (Wright, B. & Stone, M. (1979). Best test design)

Mathematically, the logistic curve is used to model how the probability of a correct response relates to how far apart the student and question are in terms of difficulty.

Image Description: A curved line chart. The y-axis title is ‘probability’ with the labels being numbers from 0 – 1, increasing by 0.1 with each step. The x-axis title is ‘distance between question difficulty and student ability’ with the labels starting from -4, increasing by 1 each step to 4. A curved line starts from the bottom left of the graph and finishes in the top right.

If the question difficulty is higher than the ability of the student, we would say the student is < 50% likely to answer it correctly. However, if the student’s ability is higher than the question, the student has a > 50% probability of answering it correctly. If the ability and difficulty are exactly equal, the probability is 50%.

How are questions assigned difficulties?

Periodically, the responses to all questions in e-asTTle are extracted into a huge matrix. The raw scores (i.e. percent-correct) for each student and for each question are calculated.

For each question and each student, we work out the ratio between percent-correct and percent-incorrect. This is transformed using a natural logarithmic transformation. Question difficulties are adjusted for the spread of abilities, and person abilities are adjusted for the width of the test they sat. Then, Joint Maximum Likelihood Estimation is used to iterate through the matrix and fit it towards a logistic curve such as the above.

At the end of these transformations, all questions will be assigned a logit value – usually between -5 and +5 – to indicate their relative difficulty. These values are put back into e-asTTle and used to score students.

How are students assigned scores?

During scoring, students are assigned a value between -5 and +5 based on the difficulty values of the questions they answered.

A student with ability 0.0 has a 50% probability of correctly answering a question of 0.0 difficulty. The ability assigned is a best estimate of the student’s ability to correctly answer half of the questions at that difficulty level.

A similar procedure applies to deep, surface and strand scoring. However, here only a subset of questions is used in the calculation process. For example, only Algebra questions are included when generating an Algebra score.

The value between -5 and +5 is transformed to the e-asTTle scale (mean, 1500 and standard deviation, 100) to make it easier to interpret.

How do the scores map to curriculum levels?

Once the questions have assigned difficulty values, a sample of questions is taken and placed in difficulty order – that is, from easiest to hardest. A panel of teachers and curriculum experts decide the cut-off points for each curriculum level.

For reading and maths, an extra feature of e-asTTle scoring is a guessing correction that excludes unlikely responses from the scoring calculation.

When students correctly answer questions at a considerably higher curriculum level than expected given their overall response pattern, these responses are removed from the score calculation process. This results in a more realistic overall score for students.

For example, a student with an overall 2B score who answers two 4A questions correctly will have the two 4A questions removed from scoring, and their score re-calculated.

Another example is that a student who answers an entire test with a, a, a, a will either receive a dash or a very low e-asTTle score.

Nine types of questions are used in e-asTTle. 

Multiple choice

Students need to select the radio button (from the response panel) that they believe is correct. 

Students can select anywhere within the response panel. This will change the response panel from grey to blue, and it will fill in the empty radio button with a black radio button. 

To change the selection students can do either of the following. 

• Select the response panel again– this will remove that response. The response panel will change back to grey, and the radio button will be empty. 
• Select another response panel – this will change the existing response to the latest selected. 

Below is an example of a question. Note the black radio button on the selected answer.

Image Description: A multiple choice question example. The question is: Who is holding a card with an even number on it? Below the question is an image of 4 people holding cards with numbers: Ben – 24, Eru – 11, Aroha – 18, Davina – 35. Their names are listed as clickable answers underneath. Aroha’s name is selected as the answer.

Short answer

Enter a response in the available text boxes. There is no limit on the amount of text that can be entered, although most short answers require only a few words.

Image Description: A short answer question example. On the left of the screen are two steps to baking biscuits with images. Step 4. Scoop tablespoonfuls of the mixture onto a baking tray. Step 5. Bake for 10 minutes or until golden. On the right of the screen a question is displayed: 4) According to the recipe, how long do you bake the biscuits for? A blank box shows where to input the answer.

Matching

For each matching pair, choose the option from the dropdown which is believed to be correct. To bring up the dropdown options, select anywhere on the dropdown box. Some of the options may be pre-filled.

Image description: A matching question example. At the top of the screen is the question: 3) Match the sentence with the correct shape. Below on the left of the screen there are 2 drop-down menus with sentence beside them. The first reads: I have three sides. The second reads: I have 4 sides. On the right of the screen there are 3 images of shapes: a) square, b) triangle, and c) circle.

True/false

Select the radio button response panel beside the option which is believed to best fit the question or statement. 

Image description: A true or false question example: 6) Select whether the following statements are True or False. The first question reads: In the number 213, the value of 1 is ten. The second question reads: In the number 504, the value of 5 is fifty. There are two empty circles next to each question to select either true or false.

Multiple choice for several correct answers 

Select the checkbox response panel with the answer believed to be correct. You can select more than one response.

Image description: A multichoice question example: 4) Which numbers make this number sentence TRUE? Below that is 2 + [star image] > 5. There are 5 clickable answers to choose from: 1, 2, 3, 4, and 5. 4 and 5 are selected.

Sequence

Type in the numbers in the order believed to be the correct sequence of events.

Students on tablets may have to switch the virtual keyboard from alpha to numeric. Only numbers: 1, 2, 3, and so on are accepted. If students type out the numbers in word form – one, two, three – the system will mark the response as incorrect.

Image description: A sequence question example.

On the left, the text says: "Read Bird Nests and answer practice questions 5-8. Bird Nests: Some birds build their nests on the ground; others build them in trees or bushes. Nests keep birds safe and provide them with shelter, a place to lay their eggs and a place to raise their young. Some nests are made from twigs and feathers, which are woven together with strands from a spider’s web. Some are made from mud, grass or leaves."

In the centre of the page is cartoon image of a bird.

On the right-hand side, the question reads: “7) Write the numbers 1, 2 and 3 in the boxes to show the correct order for building a nest:

     3 – Weave the grass and twigs together

     1 – Find a good place to build a nest

     2 – Collect grass and twigs".

Inline

Enter the appropriate word or words into the boxes.:

Image description: An inline question reading example:

“Place ONE word in the space that best completes the passage. UMPIRE: Not everyone has what it takes to be a netball umpire. The level [1) blank box] fitness required is high and preparing for exams means hard work and hours of practice [2) blank box] a regular basis. Once you qualify, the work isn't easy.”

Shade

Select each bubble to highlight the correct option. You can select multiple bubbles.

Image description: A shade question example.

On the left, the text says: “Read Bird Nests and answer practice questions 5-8. Bird Nests. Some birds build their nests on the ground; others build them in trees or bushes. Nests keep birds safe and provide them with shelter, a place to lay their eggs and a place to raise their young. Some nests are made from twigs and feathers, which are woven together with strands from a spider’s web. Some are made from mud, grass or leaves."

In the centre of the page is cartoon image of a bird.

On the right-hand side, the question reads: “8) Shade the bubbles to show which words should have capital letters.” There are four bubbles to shade in, with the following words underneath: Different birds build their nests in different ways.

Word

These questions have a selection of word options available.

Students must choose the word that they believe is correct from the options available and type this word into the relevant box.

When the word has been entered (and spelt correctly), the equivalent word from the option box will be greyed out.

Teachers will need to mark these questions.

Image description: A word question example.

On the left the text reads: “Read Making Chocolate From Cocoa Beans and answer question 1. Making Chocolate from Cocoa Beans. An illustrated diagram shows the process of making chocolate from cocoa beans with the following headings: Cleaning, Roasting, Crushing, Grinding. Cocoa powder, Blending, Cocoa paste, Refining (conching), Tempering."

On the right, the question reads: “1) Put these words in the correct column to show whether they are ingredients or processes.” The words are: conching, cocoa beans, roast, crush, cocoa powder, milk, sugar. There are two empty lists with the headings Ingredient and Processes. Underneath each heading is the space for 4 words. Cocoa beans is already in the ingredients list."

e-asTTle writing is an online assessment tool designed to assess students’ progress in writing from years 1–10.

e-asTTle writing and the New Zealand Curriculum

e-asTTle supports teachers in assessing aspects of writing-to-communicate across the curriculum.

The e-asTTle writing tool has been informed by:

• The broad concept of using writing as a tool to support specific learning purposes across the curriculum.
• Students’ writing.
• The requirements for a rigorous, standardised assessment tool.
• The understanding that e-asTTle assesses using "writing to communicate" and general writing competence but does not assess using "writing to think about and record" across the curriculum.

Information used to support teacher decision making

Teachers use various methods, including learning conversations, observations, and standardised assessment tools like e-asTTle, to comprehensively understand students' strengths and areas for improvement. Standardised tools should be considered as just one component of the overall picture.

A rich picture of achievement for an individual student

Teachers create a detailed profile for each student that encompasses writing as a tool for achieving specific learning objectives across the curriculum. This profile includes the use of e-asTTle writing as a reliable and accurate assessment tool but also incorporates other methods like student tasks and conversations across various subject areas to provide a complete understanding of the student's abilities.

Using e-asTTle writing

Teachers choose a writing prompt that best fits their students' abilities to show their knowledge and skills within "writing as communication’" They assess their students' skills using a rubric and exemplars and through moderation processes. This helps them understand their students' current level and identify areas where they need to improve. Teachers and students can then discuss the writing together, set goals for improvement, and work together to achieve those goals.

e-asTTle writing manual and presentation

For more information about the e-asTTle writing tool, see Materials that come with this resource to download:

e-asTTle Writing Manual (2012) (.pdf)

The revised e-asTTle writing tool – tutorial (2012) (.ppt)

Customised test examples

Image description: Various examples of customised tests set at different levels.

Adaptive test examples

Image description: Various examples of adaptive tests settings for different levels.

Norms versus expectations

e-asTTle norms reflect the data collected in e-asTTle from 2007 until 2010 when the most recent re-calibration took place. For Reading and Maths, norms are available for Year 4 to Year 12.

The e-asTTle norms differ from the curriculum expectation.

The curriculum expected indicates how an average student in each year level should be progressing, according to the New Zealand curriculum documents.

The e-asTTle norms, however, indicate how the average student in New Zealand is progressing in the subject.

About generated e-asTTle scores

When e-asTTle scores are generated, they take into account the level of the questions asked. The possible combinations of e-asTTle questions are numerous and varied. You can create tests that vary widely in content, spread, and difficulty.

It should be noted that the norms are not specific to tests of a specific curriculum level or type. They are instead based on the combined data from all types of e-asTTle tests within an appropriate curriculum range.

What's included in the norming process 

Only tests that are considered within an appropriate curriculum range for each year level are included in the norming process.

For example, while it is possible to administer a Year 9 student any level of test, only tests between 3A and 5A are included in the norming process. This avoids skewing the data from tests where the minimum or maximum possible scores are very low or very high.

The average values are calculated using rolling means across the quarters. This ensures the e-asTTle norms will always increase across the year.

Note that scores need to be at least 44 points apart before we can assume a non-chance difference.

For example, if the "Number Knowledge" average is 1530 in Year 8, Quarter 2 and the corresponding "Probability" score is 1516, you cannot necessarily conclude the Year 8 students are performing better in Number Knowledge than in Statistics.

Insufficient numbers to generate justifiable norms

There are some year levels in e-asTTle where insufficient numbers of appropriate tests to generate justifiable norms exist.

This is particularly evident at Years 10-12. At these year levels, there is a potential selection bias, where the data is representative of particular ability range of students.

To mitigate the effects of non-representative sampling, norms have been adjusted to reflect appropriate score values. These norms will be updated in future years as e-asTTle is used further by students at the upper year levels.

See Materials that come with this resource to download e-asTTle norms and curriculum expected for reading and maths by year level and time of the year:

• Reading Norms (.doc)
• Maths Norms (.doc)

The Writing norms and curriculum expectations have been compiled in response to teacher requests. The mean curriculum levels come directly from e-asTTle writing. The curriculum expectations for Year 4-10 are the same as those used for Reading, which vary slightly from those used for Mathematics. The expectations for Years 1-3 have been extrapolated downwards from Year 4.

See Materials that come with this resource to download Writing Norms and curriculum expectations (July 2013) (.pdf).

The following table shows the cut-off points for each curriculum level as at October 2013. These are also referred to as cutscores. Levels 1B, 1P and 1A are relevant for writing only. Scores less than level 2 are reported as <2B for all other subjects.

Reading, Writing and Maths

Pānui, Tuhituhi and Pāngarau

Constructing the e-asTTle writing scale

The e-asTTle writing scale is based on an extension of the Rasch Measurement Model (RMM). Used widely in educational measurement, the RMM is a mathematical model that can be used to transform ordinal observations (such as rubric scores) into linear measures. The RMM predicts the probability of a test taker at a given proficiency level achieving success on a test item of known difficulty. Test-taker proficiency and item difficulty are assumed to be located on the same measurement scale and the probability of success on the item is a function of the difference between them. 

The Multifacet Rasch Model (MFRM) extends the RMM by taking into account additional facets besides student proficiency and item difficulty that might be associated with the measurement context. In the context of a writing assessment, these include marker severity and the difficulty of the prompt. 
To develop the e-asTTle writing scale, an MFRM was constructed that included: 

• Student writing proficiency.
• The difficulty of the prompts to which the students were writing.
• The difficulty of the elements against which the students’ written responses were being judged. 
• The thresholds or barriers to being observed in a scoring category for an element relative to the scoring category below.
• The harshness of the markers judging the students’ written responses. 

The model assumes that all these facets can be measured on a single continuum (measurement scale) and that their locations on this continuum are used to determine the probability that a student will score in the higher of two adjacent scoring categories. Statistical and graphical fit indicators are used to study the extent to which prompts, markers, students and marking rubrics fit the MFRM. 

To construct the measurement scale, student responses to 21 writing prompts were collected in a national trial involving approximately 5000 students from Years 1 to 10. The students involved were selected using a random sampling methodology, which is described in section 6.2. Care was taken so that all markers and prompts could be linked across the students involved. This meant that many of the students completed two prompts and that many of the responses were double-marked.

The markers involved in the trial were trained teachers or held relevant post-graduate degrees. Each marker attended a two-day training course at the start of the marking exercise. Marking was done in teams and moderation meetings were carried out daily. Data was entered and carefully validated before analysis of data was carried out using the computer program Facets (Linacre, 2010). 

The image above provides a graphical representation of the measurement scale constructed by the analysis process. The scale itself is presented on the left in e-asTTle writing scale units (aWs). The scale locations of students, prompts, markers, the elements of the rubric and the scale thresholds are displayed from left to right. As can be seen, these locations vary. Prompt 20, for instance, is located slightly higher on the scale than Prompt 27, indicating it was the more difficult of the two prompts. Similarly, some markers (indicated by asterisks) are higher on the scale than others, indicating they applied the rubric more harshly. 

The final model for e-asTTle takes into account the scale locations of the prompts, elements, and thresholds shown in the above image to transform students’ rubric scores to scale locations. Values for marker harshness are not included directly as there is no way to know how harshly a user may have marked. However, the variance in marker harshness exposed through the modelling process provides some idea of the imprecision markers introduce, and marker variance is included in the estimates of measurement error reported by the e-asTTle application for each scale score.

Box-and-whisker plots are based on five score points. Imagine the scores in your group or class ordered from lowest to highest.

Components of a box-and-whisker graph

Whiskers:

• Minimum - the lowest score in the group.
• Maximum - the highest score in the group.

Box:

• Median - the middle score (note, this is not necessarily the average).
• Lower quartile - the score halfway between the minimum and median.
• Upper quartile - the score halfway between the median and maximum.

Box-and-whisker plots provide information about the centre of group achievement, as well as the spread.

The blue box-and-whiskers represent the e-asTTle norms.

The red box-and-whiskers represent your group. Always check the number in brackets – this shows you how many students have sat the test. The more students have taken the test, the more you can be confident that the box-and-whisker is a good reflection of your group.

Image description: Labelled components of box and whisker plots. The blue box-and-whisper plot is labelled ‘NZ Norms'. The red box-and-whisper plot is labelled ‘Your class'. Arrows point to various components of the red box-and-whisper plot: Maximum Score, Upper Quartile, Median, Lower Quartile, Maximum Score, Number of results.

When they cannot fit on the graph, extremely high or low scores display as arrows. You may notice that the tails on the blue box-and-whisker are very long Reading and Maths – or represented as arrows. This is because e-asTTle tests have been sat by many, many students, so the highest and lowest possible score are very spread out. You may wish to refer to the Tabular Report to identify which students have the very high or low scores.

A score of any kind is always an estimate of true performance. Usually, if you were to re-test the same student again, their score would be at least slightly different. 

Even raw scores, such as 8/10 are often treated as if they are 100% precise, however even these are subject to error.

How can we know how precise a score is? 

We can estimate how widely spread out a student’s scores would be if they were re-tested repeatedly. This calculation is called the standard error of measurement (SEM), or the measurement error. 

A common analogy for the SEM is shooting arrows at a target. The SEM is how closely the arrows cluster together. According to the Rasch Model used in e-asTTle, on a given test, it is the very high and very low scores that have the highest SEM, and the scores in the middle with the lower SEM. The length of the test is also a factor – longer tests tend to have smaller SEM.

How precise are e-asTTle scores? 

On average, the measurement error of measurement is +/-22 points for e-asTTle Pānui, Pāngarau, Maths and Reading, and +/-40 points in e-asTTle Tuhituhi and Writing.

One way of interpreting this is: 

• For pānui, pāngarau, reading and maths, scores that differ by less than 44 points (22 x 2) tend not to be meaningfully different – they could be the same score, given chance factors involved in assessment.
• For tuhituhi and writing, scores that differ by less than 80 points tend not to be meaningfully different – they could be the same score, given chance factors involved in assessment.

However, of course, this is just an average. The measurement error of measurement can be calculated for each instance of a student sitting a test. 

For all domains, you can find the SEM for specific students by looking at their Individual Learning Pathway.

Image description: Screenshot showing student score of 1459 +/- 24 on a scale.

The SEM displays in two ways: 

• As a +/- value next to the student’s score.
• The circle on the score arrow represents this graphically. There is a good probability that the student’s true score falls within the red circle.

Measurement error differences across subjects 

The measurement error is calculated slightly differently depending on the subject. 

Maths and reading 

The measurement error is calculated based on the specific student’s score and test. 

• Longer tests will generally have smaller measurement errors than shorter tests. 
• Students achieving in the middle of the test (i.e. ~ 50% correct) will have smaller measurement errors than students who score high or low. 

Writing and Tuhituhi

Similarly to pānui, pāngarau, reading and maths, there is an estimate of the error associated with inconsistencies in how students respond to the writing or tuhituhi test. The measurement error will be higher for students with very low or high rubric scores. 

An additional error component is added to this estimate to represent the variability in how different markers apply the marking rubrics.