Studies and research

​​The move to online assessment is informed by studies and research. Some of the current Queensland, national and international research is listed here, along with an overview of the findings.


Featured


Queensland Evidence Hub

Leaders

Evidence Hub case studies (DET login required)
Examples of evidence reports about what works best, where and in what context in Queensland state schools.

NAPLAN report guides and handbooks
These Queensland Curriculum and Assessment Authority (QCAA) reports provide teachers with comprehensive information on the NAPLAN tests and the performance of student groups in Queensland. They examine each learning area/test domain. They also provide a commentary on the results, including suggestions for classroom teachers on how to further investigate their students' performance.

NAPLAN test item analyses 2010 to 2015
QCAA staff developed these materials for teachers to use with the data in their NAPLAN class reports. Each item analysis includes a description of what is being tested, references to current state and national curriculum documents, and teaching strategies that support the development of the concepts and skills being tested.

They also include explanations of possible reasons for students' selections of incorrect options in multiple-choice items and of the most common incorrect responses to spelling and numeracy items.

Teachers

A Conceptual Framework for Emergent Digital Literacy 2016
This Griffith University paper proposes a framework that could guide research and practice by examining the relationships between emergent literacy skills, emergent digital literacy skills, and proficiency in reading and writing.

Beyond NAPLAN: How to read challenging texts 2015
This QCAA paper uses data from NAPLAN to examine the performance of Queensland students on more challenging texts. It examines what constitutes a challenging text and outlines the role of metacognitive conversation and cognitive tools to assist students in analysing challenging texts more effectively. It includes resources for classroom use.

Beyond NAPLAN: Using reading data to improve students' performance in higher-order questioning 2015
The NAPLAN reading test has regularly focused on reading with a sense of purpose, and on identifying purpose and main ideas. These items broadly assess higher-order comprehension, inference, synthesis and evaluation. This QCAA paper seeks to answer why students find these types of items difficult. It also gives ideas on how this difficulty can be addressed.

Queensland Students' Understanding of Fractions: Evidence from the NAPLAN test results
While fractions are widely used in everyday life the evidence suggests that students struggle with this important numerical concept. This QCAA paper refers to NAPLAN data and analyses specific link items in order to assist teachers in strengthening their students' understanding and application of fractional concepts.

Featured

Leaders

ACARA's research to inform moving NAPLAN online
ACARA is conducting a broad research and development program to inform work towards the transition from paper-based to computer-based assessments. ACARA's research program has delivered a number of studies designed to ensure technical and content readiness for NAPLAN online including:

  • Tailored tests
  • Tailored test development studies
  • Automated essay scoring
  • Device suitability
  • Student engagement and motivation
  • Readability and layout
  • Accessibility options
Online Assessment Research: Item Trial 2016 (ACARA)

The NAPLAN Online Item Trial is a set of computer-based tests delivered through the online national assessment platform. The trial aims to evaluate test items under consideration for inclusion in future tests.

Online Assessment Research: Linking Study 2016 (ACARA)

Conducted in schools from 15 August to 16 September 2016, the linking study aims to compare and evaluate student performance on pen and paper tests against computer-based tests. This will be used to guide the development of NAPLAN's 2017 online tests. This study is an important element in ACARA's ongoing research into delivering NAPLAN Online.

Online Assessment Research: Scaling Study 2016 (ACARA)

The scaling study is a set of computer-based tests delivered using the online national assessment platform. The study aims to collect information about the performance of students in ACARA's multistage adaptive tests (also known as tailored tests) to guide the development of NAPLAN's 2017 online tests.

Online Assessment Research: Development Study 2014: Student Motivation (PDF, 398KB)
This ACARA study explores:

  • motivation as a factor in students' performance in NAPLAN
  • the effects of fixed and branched (that is, adapting to suit the abilities of students) online NAPLAN approaches
  • students' subjective experience of the online NAPLAN test.

The findings shed further light on the effects of adaptive online testing and extend current understanding beyond NAPLAN performance outcomes to the important area of student motivation.

Centre for Assessment Reform and Innovation
The Australian Council for Educational Research (ACER) has established the Centre of Assessment Reform and Innovation (CARI) to research assessment theory and practice. CARI's website provides links to articles covering a range of assessment related topics, including the use of technology in assessment.

Assessing general capabilities
Assessing and reporting on student learning in general capabilities can be challenging, yet these are essential skills for 21st century learners. Fraillon says a way to address these challenges is to make students explicitly aware of their learning in the general capabilities. ACER, through the Centre for Assessment Reform and Innovation, is working on the development of scales for assessing and monitoring growth and improvement in general capabilities, as well as in fundamental skill areas such as reading and numeracy.

IEA International Computer and Information Literacy Study
This ACER study compared the ICT capabilities of students in their 8th year of schooling in about 20 countries. While Australia compared well with the other countries, small percentage differences between states was reported. Interestingly, girls performed better than boys in all Australian states.

Online examinations
The opportunity to introduce more variables into the structure of assessment, such as offering computer-based tests at different times and locations, or tailoring tests to individual cohorts of students is a fundamental advantage of online assessment according to this University of Technology Sydney report.

Leadership and Learning with ICT (PDF, 576KB)
School principals are critical to developing learning environments which incorporate student-centred approaches to teaching and learning with ICT. This paper outlines what Australian educational leaders see as factors that contribute to how leadership supports learning with ICT in Australian schools.

PISA 2003 Australia: ICT Use and Familiarity at School and Home (PDF, 861KB)
This report examines the extent of access to ICT in schools and homes and the confidence of students in making use of the technologies available to them. The report investigates the 'digital divide' and assesses access and use of ICT in Australia by state, gender, Indigenous background, socioeconomic background and geographic location.

2016 NAPLAN National Report released
The 2016 NAPLAN National Report has been published, confirming the majority of the initial findings of the NAPLAN summary information released in August 2016.

The new work order: Ensuring young Australians have skills and experience for the jobs of the future, not the past (PDF, 3.4MB)
This (Foundation for Young Australians) report is about the changing future of work. It builds on a previous report, called Renewing Australia's Promise, which identified that Australia is not investing in young people to meet the challenge of maintaining current standards as the population ages. The New Work Order report shows there are more issues ahead for young people as they face the impact of the most significant disruption in the world of work since the industrial revolution.

Teachers

National Assessment Program – ICT Literacy Years 6 & 10 Report 2014 (PDF, 9.4MB)
The findings from the 2014 ICT literacy sample assessment presented in this ACARA report provide valuable information on the ICT achievement levels of Years 6 and 10 students in Australia. Significantly, this was the first time NAP – ICT literacy was trialled and delivered to students in the online environment.

NAPLAN Online Readability and Layout Study - Literature Review: Major Reports and Working Papers (PDF, 9.9MB)
This University of Melbourne study investigated current knowledge about readability and layout in on-screen assessments, especially reading assessments. It also explored how this may impact on the development of the NAPLAN online tests in 2017.

National Assessment Program – Literacy and Numeracy: Online Assessment Research: Accessibility Options for Students with Disability 2014 (PDF, 846KB)
In 2013, ACER investigated adjustment options to provide students with disability with access to NAPLAN Online. The research investigated the issues that proposed adjustments might raise for practical deployment, and whether there would be satisfactory replacements for allowable adjustments currently used with the paper-based version of the test. The research showed none of the proposed adjustments would violate the test construct by conferring disproportionate advantage on students with disability.

Online Assessment Research: Development Study 2014: Student engagement and motivation
This development study was commissioned by ACARA to examine the conditions and requirements for transitioning NAPLAN assessments from paper to online delivery. It comprised the following smaller studies.

  • Technology Enhanced Items: Reading (PDF, 1.5MB)
    This small study focused on the impact of factors (such as computer literacy, motor skills, reading ability, motivation and fatigue) on students' interactions with item layouts, item directions, and the technical enhancement types (features) contained in the items. Students' reading ability was also a variable in the study. This paper outlines the major benefits of online testing in English and a summary of the skills students require to successfully participate in such online tests.
  • Technology Enhanced Items: Numeracy (PDF, 3.5MB)
    This project examined design considerations of technology-enhanced test items that impact students' engagement with these items in a meaningful way.
  • Cognitive Interviews: Spelling (PDF, 2.0MB)
    This study looked at the effect of two types of technology-enhanced spelling items delivered via two modes—dictation via audio recording and proofreading—to students in Years 3, 5, 7 and 9.

Digital Technologies Hub
The Digital Technologies Hub has been designed by Education Services Australia (ESA) to assist educators in implementing the new Digital Technologies curriculum. A common question teachers have posed is, What's the difference between the ICT Capability and Digital Technologies? In response ESA has created an Infographic that demonstrates how they differ using relevant examples for each. Schools can use these resources to advance the discussion and clarify the relationship between each.

2016 Australian Council for Computers in Education (ACCE) Conference
In 2016 the ACCE conference, organised by the Queensland Society for Information Technology in Education (QSITE), explored the theme of Imagined Futures. "What IF schools were not encumbered by traditional norms and structures, and technology, social capital and pedagogies were used to their true realisation or potential?" The link provides access to the list of presentations and their corresponding papers.

Technology

An evaluation of automated scoring of NAPLAN persuasive writing 2015 (PDF, 884KB)
This ACARA paper outlines the investigations undertaken into the validity and scalability of automated assessment of written tasks. ACARA undertook a study with four vendors, providing them with persuasive text essays for students in Years 3, 5, 7 and 9. Two human markers also scored each essay. Each essay was typical of its year level. The vendors used a combination of syntax, lexile and semantic scoring mechanisms against a rubric provided by ACARA. All vendors showed results that were similar to manual marking.

NAPLAN Online Research and Development: Report 1: Device Effect Study – Literature Review and Report 2: Device Effect Study – Field Trial
This study was conducted to ensure that NAPLAN can be equitably administered across all devices (such as laptops and tablets) that meet the minimum technical requirements. There were some limited device effects, which were small and centred on specific item types and features. It is anticipated that as student familiarity with devices improves between now and NAPLAN going online, these minor issues will be addressed.

Results from the device effect study informed the minimum technical specifications, available on the ACARA NAP website.

Home influence on the use of ICT at school
This ACER 2015 Australian study explores the impact of technology access from school and home use, on ICT skills of Year 9 students.

Tailored test design study 2013: Summary research report (PDF, 954KB)
In August and September 2013, ACARA trialled the online delivery of a multistage computer adaptive test—also known as a tailored test. More than 250 schools throughout Australia volunteered to participate. The study included remote schools and a small number of home-based and geographically remote students. About 23,000 tests were delivered to students in Years 3, 5, 7 and 9, in reading, numeracy and writing. The study also investigated the cognitive and behavioural engagement of students with a tailored test and the effectiveness of the proposed test design in better targeting underperforming students.

Featured


National centre for educational statistics

Leaders

National centre for educational statistics
The National Center for Education Statistics (NCES) is the primary federal entity for collecting and analysing data related to education in the U.S. and other nations. NCES is located within the U.S. Department of Education and the Institute of Education Sciences.

Programme for International Student Assessment Australia
The Australian Council for Educational Research (ACER) has just released the Australian report for the 2015 Programme for International Student Assessment (PISA). Managed by ACER at the national level for the Organisation for Economic Cooperation and Development (OECD), every three years PISA measures how well 15-year-olds from across the globe are able to use their knowledge and skills in science, reading and mathematics to meet real-life challenges.

Performance of fourth-grade students in the 2012 NAEP computer-based writing pilot assessment (PDF, 1.9MB)
This study (National Centre for Education Statistics, USA) begins with the policy question, 'Can fourth-graders fully display their writing ability on the computer?' There is no simple yes or no answer. The data shows that prior exposure to writing on the computer is itself associated with writing performance.

Review of International Large-Scale Assessments in Education: Assessing Component Skills and Collecting Contextual Data
The OECD has initiated the Programme for International Student Assessment (PISA) for Development in response to the rising need of developing countries to collect data about their education systems and the capacity of their student bodies.

Are California's Schools Ready for Online Testing and Learning?
Using survey data from the California Educational Technology Professionals Association (CETPA), this report examined school districts' technology infrastructure and assesses their readiness for online testing.

Are schools 'tech-ready' for the Common Core Standards?
In this Great Schools article, the author summarises the issues experienced during a trial run of the Common Core Standards online assessment, which involved 36 million students in 36 USA states.

Common core testing points out tech divide in rural, poorer schools
This Aljazeera America news article reports the challenges remote areas face in meeting the demands of online education.

Getting ready for online assessments (PDF, 7.4MB)
This is the third in a series of interactive papers from Digital Learning Now! that provide specific guidance on the adoption of Common Core Standards and the shift to personal digital learning. The paper documents important learnings from the USA experience. The recommendations are written for policy writers and school and district personnel as they work to bring school districts up to speed and make them ready for online assessments and learning.

NMC Horizon Report Preview 2016 K-12 Edition (PDF, 268KB)
This preview from New Media Consortium identifies six key trends, six significant challenges, and six important developments in education technology that will be featured in this year's full report.

Using computers widens the achievement gap in writing, a federal study finds
The research arm of the United States of America Department of Education found that fourth-grade students with low digital capability did poorly in writing tests given by computer (The Hechinger Report). Students with good ICT skills performed better. The research involved 10,000 fourth-grade students who completed two 30-minute writing assignments.

Arizona State University Professor Steve Graham says 'Your familiarity with the tool makes a difference.' 'They actually do better writing by hand if they're less experienced [with computers]. And if they're more experienced, then there may actually be an advantage toward writing on the computer.'

Teachers

Students, Computers and Learning: Making the Connection 2015
The Organisation for Economic Cooperation and Development (OECD) report highlights that students need to be equipped with basic literacy and numeracy skills before they can participate fully in the hyper-connected, digitised societies of the 21st century. The report highlights the importance of bolstering students' ability to navigate through digital texts. It also examines the relationship between computer access in schools, computer use in classrooms and performance in PISA assessment. As the report makes clear, all students need to be equipped with basic literacy and numeracy skills so they can participate fully in the hyper-connected, digitised societies of the 21st century.

Being a better online reader
This article explores the impact of digital texts on deep reading. As students learn how to read digital texts, do we need more immersive reading skills to be explicitly taught that deeper comprehension requires?

Raising literacy levels using digital learning: a design-based approach in New Zealand
This paper by The Curriculum Journal reports on a design-based intervention in urban primary and secondary schools serving culturally diverse students from low socio-economic status (SES) communities. The intervention capitalises on a partially implemented program in seven schools that use digital devices (netbooks) and applications to raise literacy levels.

Technology

An overview of Automated Scoring of Essays (PDF, 755KB)
This study by the Journal of Technology, Learning and Assessment is an overview of current approaches to automated essay scoring (AES). The study provides an overview of the main characteristics of AES programs and how they can be used in classroom assessment and standardised tests.

Contrasting Automated and Human Scoring of Essays (PDF, 780KB)
International research provides a balanced view of the contrasts between automated and human scoring of essays.

Teachers' Perceptions for How to Prepare Students for Success with the New Computer-Based Assessment Platform
“As technology becomes increasingly integrated into our daily lives, and standardised assessment moves to a computer-based testing platform, it is important to not only identify the possible effects of changing how student achievement is assessed, but also to examine whether the technology competency levels of teachers have an effect.” (Society for Information Technology & Teacher Education International Conference)

Last updated 22 March 2017