Evidence Hub case studies (department login required)
Examples of evidence reports about what works best, where and in what context in Queensland state schools.
NAPLAN report guides
The Student and School Summary Reports (SSSR) within the national online platform focusses on the students, classes and school results. The SSSR enables school leaders and teachers to identify areas of strength and development for students, linked to the Australian Curriculum. The SSSR provides specific features, such as:
- each test item linked to an exemplar and the specific ACARA curriculum content descriptor
- each test item degree of difficulty as expressed against the NAPLAN scale (e.g. 312, 415)
- student writing scripts for Yr 3, 5, 7 and 9
- the tailored testing pathway for each student.
The release of the SSSR occurs through the online assessment platform, following the completion of all the national scaling and equating procedures. A paper copy of the
final Individual Student Report (ISR) arrives in schools in late term 3, ready to be distributed to parents.
A Conceptual Framework for Emergent Digital Literacy 2016
This Griffith University paper proposes a framework that could guide research and practice by examining the relationships between emergent literacy skills, emergent digital literacy skills, and proficiency in reading and writing.
NAPLAN Online automated scoring research program: Research report (PDF, 1.1MB)
This Australian Curriculum, Assessment and Reporting Authority (ACARA) report, released in 2018, outlines the details of research designed to collect and evaluate empirical evidence on the feasibility and validity of automated essay scoring in NAPLAN writing assessments.
Online assessment research: Item trial 2016 (ACARA)
The NAPLAN Online item trial is a set of computer-based tests delivered through the online national assessment platform. The trial aims to evaluate test items under consideration for inclusion in future tests.
Online assessment research: Linking study 2016 (ACARA)
Conducted in schools from 15 August to 16 September 2016, the linking study aims to compare and evaluate student performance on pen and paper tests against computer-based tests. This will be used to guide the development of NAPLAN's 2017 online tests. This study is an important element in ACARA's ongoing research into delivering NAPLAN Online.
Online assessment research: Scaling study 2016 (ACARA)
The scaling study is a set of computer-based tests delivered using the online national assessment platform. The study aims to collect information about the performance of students in ACARA's multistage adaptive tests (also known as tailored tests) to guide the development of NAPLAN's 2017 online tests.
Online assessment research: Development study 2014—Student motivation (PDF, 398KB)
This ACARA study explores:
- motivation as a factor in students' performance in NAPLAN
- the effects of fixed and branched (that is, adapting to suit the abilities of students) online NAPLAN approaches
- students' subjective experience of the online NAPLAN test.
The findings shed further light on the effects of adaptive online testing and extend current understanding beyond NAPLAN performance outcomes to the important area of student motivation.
Centre for Assessment Reform and Innovation
The Australian Council for Educational Research (ACER) has established the Centre of Assessment Reform and Innovation (CARI) to research assessment theory and practice. CARI's website provides links to articles covering a range of assessment related topics, including the use of technology in assessment.
The New Work Order report by the Foundation for Young Australians (PDF, 3.4MB)
Prepared by the Foundation for Young Australians, The New Work Order report looks at the future of work for young Australians and the challenge of ensuring young Australians have skills and experience for the jobs of the future. It builds on a previous report, called
Renewing Australia's Promise, which identified that Australia is not investing in young people to meet the challenge of maintaining current standards as the population ages. The New Work Order report shows there are more issues ahead for young people as they face the impact of the most significant disruption in the world of work since the industrial revolution.
Assessing general capabilities
Assessing and reporting on student learning in general capabilities can be challenging, yet these are essential skills for 21st century learners. Fraillon says a way to address these challenges is to make students explicitly aware of their learning in the general capabilities. ACER, through the Centre for Assessment Reform and Innovation, is working on the development of scales for assessing and monitoring growth and improvement in general capabilities, as well as in fundamental skill areas such as reading and numeracy.
IEA International Computer and Information Literacy Study
This ACER study compared the ICT capabilities of students in their 8th year of schooling in about 20 countries. While Australia compared well with the other countries, small percentage differences between states was reported. Interestingly, girls performed better than boys in all Australian states.
The opportunity to introduce more variables into the structure of assessment, such as offering computer-based tests at different times and locations, or tailoring tests to individual cohorts of students is a fundamental advantage of online assessment according to this University of Technology Sydney report.
Leadership and learning with ICT
School principals are critical to developing learning environments which incorporate student-centred approaches to teaching and learning with ICT. This paper outlines what Australian educational leaders see as factors that contribute to how leadership supports learning with ICT in Australian schools.
PISA 2003 Australia: ICT use and familiarity at school and home (PDF, 860KB)
This report examines the extent of access to ICT in schools and homes and the confidence of students in making use of the technologies available to them. The report investigates the 'digital divide' and assesses access and use of ICT in Australia by state, gender, Indigenous background, socioeconomic background and geographic location.
NAPLAN national report released
The NAPLAN national report has been published, confirming the majority of the initial findings of the NAPLAN summary information released in August 2016.
National Assessment Program—ICT literacy Years 6 and 10 report 2014 (PDF, 10MB)
The findings from the 2014 ICT literacy sample assessment presented in this ACARA report provide valuable information on the ICT achievement levels of Years 6 and 10 students in Australia. Significantly, this was the first time NAP—ICT literacy was trialled and delivered to students in the online environment.
NAPLAN Online readability and layout study—Literature review: Major reports and working papers (PDF, 9.9MB)
This University of Melbourne study investigated current knowledge about readability and layout in on-screen assessments, especially reading assessments. It also explored how this may impact on the development of the NAPLAN online tests in 2017.
National Assessment Program—Literacy and Numeracy, Online assessment research: Accessibility options for students with disability 2014 (PDF, 845KB)
In 2013, ACER investigated adjustment options to provide students with disability with access to NAPLAN Online. The research investigated the issues that proposed adjustments might raise for practical deployment, and whether there would be satisfactory replacements for allowable adjustments currently used with the paper-based version of the test. The research showed none of the proposed adjustments would violate the test construct by conferring disproportionate advantage on students with disability.
Online assessment research: Development study 2014: Student engagement and motivation
This development study was commissioned by ACARA to examine the conditions and requirements for transitioning NAPLAN assessments from paper to online delivery. It comprised the following smaller studies.
Technology enhanced items: Reading (PDF, 1.1MB)
This small study focused on the impact of factors (such as computer literacy, motor skills, reading ability, motivation and fatigue) on students' interactions with item layouts, item directions, and the technical enhancement types (features) contained in the items. Students' reading ability was also a variable in the study. This paper outlines the major benefits of online testing in English and a summary of the skills students require to successfully participate in such online tests.
Technology enhanced items: Numeracy (PDF, 3.4MB)
This project examined design considerations of technology-enhanced test items that impact students' engagement with these items in a meaningful way.
Cognitive interviews: Spelling (PDF, 2MB)
This study looked at the effect of two types of technology-enhanced spelling items delivered via two modes—dictation via audio recording and proofreading—to students in Years 3, 5, 7 and 9.
Cognitive interviews: Challenging items (numeracy) (PDF, 1.8MB)
This study focused on how students interacted with the challenging items on Testlet F of the tailored test. In particular how well Testlet F provided an adequate testing context for highly capable students, how these students applied knowledge, thinking skills and strategies.
Cognitive interviews: Challenging items (reading) (PDF, 1.6MB)
This study focused on the perceived difficulty of challenging Reading items, conducted through individual interviews, was initiated in order to investigated the knowledge, thinking skills and strategies that 10 high performing students from each of the 4 NAPLAN (National Assessment Program Literacy and Numeracy) testing levels of schooling, (Years 3, 5, 7 and 9) used to answer items in Reading Testlet F in an online environment.
Digital Technologies Hub
The Digital Technologies Hub has been designed by Education Services Australia (ESA) to assist educators in implementing the new Digital Technologies curriculum. A common question teachers have posed is, What's the difference between the ICT Capability and Digital Technologies? In response ESA has created an Info graphic that demonstrates how they differ using relevant examples for each. Schools can use these resources to advance the discussion and clarify the relationship between each.
2016 Australian Council for Computers in Education (ACCE) conference
In 2016 the ACCE conference, organised by the Queensland Society for Information Technology in Education (QSITE), explored the theme of Imagined Futures. 'What
if schools were not encumbered by traditional norms and structures, and technology, social capital and pedagogies were used to their true realisation or potential?' The link provides access to the list of presentations and their corresponding papers.
An evaluation of automated scoring of NAPLAN persuasive writing 2015 (PDF, 976KB)
This ACARA paper outlines the investigations undertaken into the validity and scalability of automated assessment of written tasks. ACARA undertook a study with four vendors, providing them with persuasive text essays for students in Years 3, 5, 7 and 9. Two human markers also scored each essay. Each essay was typical of its year level. The vendors used a combination of syntax, lexile and semantic scoring mechanisms against a rubric provided by ACARA. All vendors showed results that were similar to manual marking.
NAPLAN Online research and development, Report 1: Device effect study—Literature review and Report 2: Device effect study—field trial
This study was conducted to ensure that NAPLAN can be equitably administered across all devices (such as laptops and tablets) that meet the minimum technical requirements. There were some limited device effects, which were small and centred on specific item types and features. It is anticipated that as student familiarity with devices improves between now and NAPLAN going online, these minor issues will be addressed.
Results from the device effect study informed the
minimum technical specifications, available on the ACARA NAP website.
Home influence on the use of ICT at school
This ACER 2015 Australian study explores the impact of technology access from school and home use, on ICT skills of Year 9 students.
Tailored test design study 2013: Summary research report (PDF, 955KB)
In August and September 2013, ACARA trialled the online delivery of a multistage computer adaptive test—also known as a tailored test. More than 250 schools throughout Australia volunteered to participate. The study included remote schools and a small number of home-based and geographically remote students. About 23,000 tests were delivered to students in Years 3, 5, 7 and 9, in reading, numeracy and writing. The study also investigated the cognitive and behavioural engagement of students with a tailored test and the effectiveness of the proposed test design in better targeting underperforming students.
National centre for educational statistics
The National Center for Education Statistics (NCES) is the primary federal entity for collecting and analysing data related to education in the U.S. and other nations. NCES is located within the U.S. Department of Education and the Institute of Education Sciences.
Programme for International Student Assessment Australia
The Australian Council for Educational Research (ACER) has just released the Australian report for the 2015 Programme for International Student Assessment (PISA). Managed by ACER at the national level for the Organisation for Economic Cooperation and Development (OECD), every three years PISA measures how well 15-year-olds from across the globe are able to use their knowledge and skills in science, reading and mathematics to meet real-life challenges.
Performance of fourth-grade students in the 2012 NAEP computer-based writing pilot assessment (PDF, 1.9MB)
This study (National Centre for Education Statistics, USA) begins with the policy question, 'Can fourth-graders fully display their writing ability on the computer?' There is no simple yes or no answer. The data shows that prior exposure to writing on the computer is itself associated with writing performance.
Review of international large-scale assessments in education: Assessing component skills and collecting contextual data
The OECD has initiated the Programme for International Student Assessment (PISA) for Development in response to the rising need of developing countries to collect data about their education systems and the capacity of their student bodies.
Are California's schools ready for online testing and learning?
Using survey data from the California Educational Technology Professionals Association (CETPA), this report examined school districts' technology infrastructure and assesses their readiness for online testing.
Are schools 'tech-ready' for the Common Core Standards?
In this Great Schools article, the author summarises the issues experienced during a trial run of the Common Core Standards online assessment, which involved 36 million students in 36 USA states.
Common core testing points out tech divide in rural, poorer schools
This Aljazeera America news article reports the challenges remote areas face in meeting the demands of online education.
Getting ready for online assessments (PDF, 7.4MB)
This is the third in a series of interactive papers from Digital Learning Now! that provide specific guidance on the adoption of Common Core Standards and the shift to personal digital learning. The paper documents important learnings from the USA experience. The recommendations are written for policy writers and school and district personnel as they work to bring school districts up to speed and make them ready for online assessments and learning.
NMC Horizon Report Preview 2016 K-12 edition (PDF, 1.2MB)
This preview from New Media Consortium identifies 6 key trends, 6 significant challenges, and 6 important developments in education technology that will be featured in this year's full report.
Using computers widens the achievement gap in writing, a federal study finds
The research arm of the United States of America Department of Education found that 4-grade students with low digital capability did poorly in writing tests given by computer (The Hechinger Report). Students with good ICT skills performed better. The research involved 10,000 4-grade students who completed two 30-minute writing assignments.
Arizona State University Professor Steve Graham says 'Your familiarity with the tool makes a difference.' 'They actually do better writing by hand if they're less experienced [with computers]. And if they're more experienced, then there may actually be an advantage toward writing on the computer.'
Students, computers and learning: Making the connection 2015
The Organisation for Economic Cooperation and Development (OECD) report highlights that students need to be equipped with basic literacy and numeracy skills before they can participate fully in the hyper-connected, digitised societies of the 21st century. The report highlights the importance of bolstering students' ability to navigate through digital texts. It also examines the relationship between computer access in schools, computer use in classrooms and performance in PISA assessment. As the report makes clear, all students need to be equipped with basic literacy and numeracy skills so they can participate fully in the hyper-connected, digitised societies of the 21st century.
Being a better online reader
This article explores the impact of digital texts on deep reading. As students learn how to read digital texts, do we need more immersive reading skills to be explicitly taught that deeper comprehension requires?
Raising literacy levels using digital learning: a design-based approach in New Zealand
This paper by The Curriculum Journal reports on a design-based intervention in urban primary and secondary schools serving culturally diverse students from low socio-economic status (SES) communities. The intervention capitalises on a partially implemented program in seven schools that use digital devices (netbooks) and applications to raise literacy levels.
An overview of automated scoring of essays (PDF, 754KB)
This study by the Journal of Technology, Learning and Assessment is an overview of current approaches to automated essay scoring (AES). The study provides an overview of the main characteristics of AES programs and how they can be used in classroom assessment and standardised tests.
Contrasting automated and human scoring of essays (PDF, 779KB)
International research provides a balanced view of the contrasts between automated and human scoring of essays.
Teachers' perceptions for how to prepare students for success with the new computer-based assessment platform
'As technology becomes increasingly integrated into our daily lives, and standardised assessment moves to a computer-based testing platform, it is important to not only identify the possible effects of changing how student achievement is assessed, but also to examine whether the technology competency levels of teachers have an effect.' (Society for Information Technology and Teacher Education International Conference).