Studies and research

​​​The move to online assessment is informed by studies and research. Some of the current Queensland, national and international research is listed here, along with an overview of the findings.


Featured


This ACARA report, released in 2018, outlines the details of research designed to collect and evaluate empirical evidence on the feasibility and validity of Automated Essay Scoring in NAPLAN writing assessments.

Online Assessment Research: Item Trial 2016 (ACARA)

The NAPLAN Online Item Trial is a set of computer-based tests delivered through the online national assessment platform. The trial aims to evaluate test items under consideration for inclusion in future tests.

Online Assessment Research: Linking Study 2016 (ACARA)

Conducted in schools from 15 August to 16 September 2016, the linking study aims to compare and evaluate student performance on pen and paper tests against computer-based tests. This will be used to guide the development of NAPLAN's 2017 online tests. This study is an important element in ACARA's ongoing research into delivering NAPLAN Online.

Online Assessment Research: Scaling Study 2016 (ACARA)

The scaling study is a set of computer-based tests delivered using the online national assessment platform. The study aims to collect information about the performance of students in ACARA's multistage adaptive tests (also known as tailored tests) to guide the development of NAPLAN's 2017 online tests.


This ACARA study explores:

  • motivation as a factor in students' performance in NAPLAN
  • the effects of fixed and branched (that is, adapting to suit the abilities of students) online NAPLAN approaches
  • students' subjective experience of the online NAPLAN test.

The findings shed further light on the effects of adaptive online testing and extend current understanding beyond NAPLAN performance outcomes to the important area of student motivation.


The Australian Council for Educational Research (ACER) has established the Centre of Assessment Reform and Innovation (CARI) to research assessment theory and practice. CARI's website provides links to articles covering a range of assessment related topics, including the use of technology in assessment.


Prepared by the Foundation for Young Australians, The New Work Order report looks at the future of work for young Australians and the challenge​ of ensuring young Australians have skills and experience for the jobs of the future.


Assessing and reporting on student learning in general capabilities can be challenging, yet these are essential skills for 21st century learners. Fraillon says a way to address these challenges is to make students explicitly aware of their learning in the general capabilities. ACER, through the Centre for Assessment Reform and Innovation, is working on the development of scales for assessing and monitoring growth and improvement in general capabilities, as well as in fundamental skill areas such as reading and numeracy.


This ACER study compared the ICT capabilities of students in their 8th year of schooling in about 20 countries. While Australia compared well with the other countries, small percentage differences between states was reported. Interestingly, girls performed better than boys in all Australian states.


The opportunity to introduce more variables into the structure of assessment, such as offering computer-based tests at different times and locations, or tailoring tests to individual cohorts of students is a fundamental advantage of online assessment according to this University of Technology Sydney report.


School principals are critical to developing learning environments which incorporate student-centred approaches to teaching and learning with ICT. This paper outlines what Australian educational leaders see as factors that contribute to how leadership supports learning with ICT in Australian schools.


This report examines the extent of access to ICT in schools and homes and the confidence of students in making use of the technologies available to them. The report investigates the 'digital divide' and assesses access and use of ICT in Australia by state, gender, Indigenous background, socioeconomic background and geographic location.


The 2016 NAPLAN National Report has been published, confirming the majority of the initial findings of the NAPLAN summary information released in August 2016.


This (Foundation for Young Australians) report is about the changing future of work. It builds on a previous report, called Renewing Australia's Promise, which identified that Australia is not investing in young people to meet the challenge of maintaining current standards as the population ages. The New Work Order report shows there are more issues ahead for young people as they face the impact of the most significant disruption in the world of work since the industrial revolution.

Teachers


The findings from the 2014 ICT literacy sample assessment presented in this ACARA report provide valuable information on the ICT achievement levels of Years 6 and 10 students in Australia. Significantly, this was the first time NAP – ICT literacy was trialled and delivered to students in the online environment.


This University of Melbourne study investigated current knowledge about readability and layout in on-screen assessments, especially reading assessments. It also explored how this may impact on the development of the NAPLAN online tests in 2017.


In 2013, ACER investigated adjustment options to provide students with disability with access to NAPLAN Online. The research investigated the issues that proposed adjustments might raise for practical deployment, and whether there would be satisfactory replacements for allowable adjustments currently used with the paper-based version of the test. The research showed none of the proposed adjustments would violate the test construct by conferring disproportionate advantage on students with disability.

Online Assessment Research: Development Study 2014: Student engagement and motivation
This development study was commissioned by ACARA to examine the conditions and requirements for transitioning NAPLAN assessments from paper to online delivery. It comprised the following smaller studies.


  • This small study focused on the impact of factors (such as computer literacy, motor skills, reading ability, motivation and fatigue) on students' interactions with item layouts, item directions, and the technical enhancement types (features) contained in the items. Students' reading ability was also a variable in the study. This paper outlines the major benefits of online testing in English and a summary of the skills students require to successfully participate in such online tests.

  • This project examined design considerations of technology-enhanced test items that impact students' engagement with these items in a meaningful way.

  • This study looked at the effect of two types of technology-enhanced spelling items delivered via two modes—dictation via audio recording and proofreading—to students in Years 3, 5, 7 and 9.

  • This study focused on how students interacted with the challenging items on Testlet F of the tailored test. In particular how well Testlet F provided an adequate testing context for highly capable students, how these students applied knowledge, thinking skills and strategies.

  • This study focused on the perceived difficulty of challenging Reading items, conducted through individual interviews, was initiated in order to investigated the knowledge, thinking skills and strategies that 10 high performing students from each of the 4 NAPLAN (National Assessment Program Literacy and Numeracy) testing levels of schooling, (Years 3, 5, 7 and 9) used to answer items in Reading Testlet F in an online environment.


The Digital Technologies Hub has been designed by Education Services Australia (ESA) to assist educators in implementing the new Digital Technologies curriculum. A common question teachers have posed is, What's the difference between the ICT Capability and Digital Technologies? In response ESA has created an Info graphic that demonstrates how they differ using relevant examples for each. Schools can use these resources to advance the discussion and clarify the relationship between each.


In 2016 the ACCE conference, organised by the Queensland Society for Information Technology in Education (QSITE), explored the theme of Imagined Futures. "What IF schools were not encumbered by traditional norms and structures, and technology, social capital and pedagogies were used to their true realisation or potential?" The link provides access to the list of presentations and their corresponding papers.

Technology


This ACARA paper outlines the investigations undertaken into the validity and scalability of automated assessment of written tasks. ACARA undertook a study with four vendors, providing them with persuasive text essays for students in Years 3, 5, 7 and 9. Two human markers also scored each essay. Each essay was typical of its year level. The vendors used a combination of syntax, lexile and semantic scoring mechanisms against a rubric provided by ACARA. All vendors showed results that were similar to manual marking.


This study was conducted to ensure that NAPLAN can be equitably administered across all devices (such as laptops and tablets) that meet the minimum technical requirements. There were some limited device effects, which were small and centred on specific item types and features. It is anticipated that as student familiarity with devices improves between now and NAPLAN going online, these minor issues will be addressed.

Results from the device effect study informed the , available on the ACARA NAP website.


This ACER 2015 Australian study explores the impact of technology access from school and home use, on ICT skills of Year 9 students.


In August and September 2013, ACARA trialled the online delivery of a multistage computer adaptive test—also known as a tailored test. More than 250 schools throughout Australia volunteered to participate. The study included remote schools and a small number of home-based and geographically remote students. About 23,000 tests were delivered to students in Years 3, 5, 7 and 9, in reading, numeracy and writing. The study also investigated the cognitive and behavioural engagement of students with a tailored test and the effectiveness of the proposed test design in better targeting underperforming students.


Last updated 21 April 2020