Accepted Manuscript
The Contribution of Pupil, Classroom and School Level Characteristics to Primary
School Pupils’ ICT Competences: A Performance-based Approach
Koen Aesaert, Daniël Van Nijlen, Ruben Vanderlinde, Jo Tondeur, Ines Devlieger,
Johan van Braak
PII: S0360-1315(15)00092-5
DOI: 10.1016/j.compedu.2015.03.014
Reference: CAE 2822
To appear in:
Computers & Education
Received Date: 27 September 2014
Revised Date: 23 January 2015
Accepted Date: 4 March 2015
Please cite this article as: Aesaert K., Van Nijlen D., Vanderlinde R., Tondeur J., Devlieger I. & van
Braak J., The Contribution of Pupil, Classroom and School Level Characteristics to Primary School
Pupils’ ICT Competences: A Performance-based Approach, Computers & Education (2015), doi:
10.1016/j.compedu.2015.03.014.
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to
our customers we are providing this early version of the manuscript. The manuscript will undergo
copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please
note that during the production process errors may be discovered which could affect the content, and all
legal disclaimers that apply to the journal pertain.
MANUS CRIP T
ACCEP TED
ACCEPTED MANUSCRIPT
Title
The Contribution of Pupil, Classroom and School Level Characteristics to Primary School Pupils’ ICT
Competences: A Performance-based Approach
Authors
Koen Aesaert
a
*, Daniël Van Nijlen
b
, Ruben Vanderlinde
a
, Jo Tondeur
c
, Ines Devlieger
d
, Johan van
Braak
a
a
Department of Educational Studies, Ghent University, Henri Dunantlaan 2, B9000 Ghent, Belgium
b
Faculty of Psychology and Educational Sciences, Katholieke Universiteit Leuven, Dekenstraat 2,
B3000 Leuven, Belgium
c
Research Foundation Flanders (FWO), Ghent, Belgium
d
Department of Data Analysis, Ghent University, Henri Dunantlaan 1, B9000 Ghent, Belgium
* corresponding author: Department of Educational Studies, Ghent University, Henri Dunantlaan 2,
B9000 Ghent, Belgium. Tel.: +32 9 264 62 52; Fax: + 32 9 264 86 88. E-mail address:
koen.aesaert@ugent.be (K. Aesaert)
MANUS CRIP T
ACCEP TED
ACCEPTED MANUSCRIPT
1
The Contribution of Pupil, Classroom and School Level Characteristics to Primary School Pupils’ ICT
Competences: A Performance-based Approach
Abstract: The central aim of this study was to investigate which pupil, classroom and school level
characteristics are related to primary school pupils’ actual ICT competences. A sample of 378 pupils
in 58 schools in Flanders (the Dutch speaking part of Belgium) completed a performance-based ICT
competence test in order to measure their actual proficiency in retrieving and processing digital
information, and in communicating through a computer. To gather information on the factors at
each different level, questionnaires were administered to the pupils, their parents (n=378), their
teachers (n=83) and the ICT coordinators (n=58) of the schools. Pupils on average have a low to
medium score on the developed ICT competence test. The results of a hierarchical regression analysis
with multilevel design show that the differences in ICT competences can be mainly attributed to
differences in pupil level characteristics. The results indicate that especially non-ICT related pupil
characteristics are associated with differences in primary school pupils’ ICT competences, such as
introjected regulation, controlling learning style, analytic intelligence, sex and socioeconomic status.
Furthermore, the final model also indicates that parental ICT attitudes are related to primary school
pupils’ ICT competences. With regard to the classroom level characteristics, educational use of ICT as
an information tool is significantly related to pupils’ ICT competences.
1. Introduction
Within the context of 21
st
century skills and our information society, the importance of being digitally
competent is reflected in international and national policies for educational ICT use (European
Commission, 2007; ISTE, 2007; Kozma, 2008). These policies for educational ICT use have introduced
ICT competences in national and school curricula (Authors, 2013), i.e., the integration of ICT
competences in educational curricula or the development of ICT curricula has formalized the status
of ICT competences as educational outcomes. In this regard, Thomas and Knezek (2008) state that
ICT competence standards and attainment targets define the achievement expectations for students,
and as a consequence ICT competences are considered as educational outcomes.
Educational effectiveness research has shown that pupils’ educational outcomes are multilevel in
nature (Creemers & Kyriakides, 2008), i.e., differences in pupils’ educational outcomes are attributed
to factors at different levels, including the pupil, the classroom and the school level. However, with
regard to ICT competences as educational outcomes, few studies have taken into account this
multilevel aspect. In other words, very few studies have explicitly investigated whether the teacher
or the school matters in the development of pupils’ ICT competences. Claro et al. (2012) state that
besides elaborating on the traditionally used pupil level factors, such as SES, computer access, daily
use and confidence in performing ICT-related activities, future research should also focus on the
impact of pupils’ basic cognitive skills or teachers’ particular pedagogical practices that might foster
ICT competences.
MANUS CRIP T
ACCEP TED
ACCEPTED MANUSCRIPT
2
The purpose of this study is to investigate the degree to which certain factors at the pupil, classroom
and school level can explain differences in primary school pupils’ ICT competences. As such, we aim
to discover whether the teachers and schools play an important part in developing pupils’ ICT
competences. In order to measure primary school pupils’ actual ICT competences – the dependent
variable of this study - a performance-based digital test was used. As such, this study tackles the
problem of self-reported bias that indirect measures of ICT competence or ICT self-efficacy suffer
from. The pupil, classroom and school level characteristics that make up the independent variables of
this study were drawn from the Extensive Digital Competence (EDC)-model (Authors, 2014).
Measurements of ICT competences mostly target students from secondary and higher education
(Meelissen, 2008). Moreover, research in terms of national and international curricula for early
childhood and primary education indicates that ICT competences should already be taught at an
early age. As such, the focus of this study is on primary school pupils’ ICT competences.
2. Theoretical framework
2.1 ICT competences
In his analysis of literacies for the digital age, Martin (2006) explains that the concept of ICT literacy –
and the accompanied perception of ICT competences - has gone through a three-stage evolution of
mastery, application and reflection.
In the mastery phase (until the mid-1980s) ICT literacy was perceived as knowledge of how the
computer works (computer science) and skills on how to master and program it. ICT or computer
literacy emphasized learning about information technology rather than learning with or through
computers (Carleer, 1984). Tannenbaum and Rahn (1984) expressed this as having a fundamental
understanding of the components of the machine, of its history, of the principal application, and as
acquiring hands-on skill in programming language.
As operating systems and software applications became more user friendly and products of mass
usage, ICT literacy shifted into a more application oriented phase (until the late 1990s). Rather than
on specialist knowledge, ICT literacy focused on practical basic competences to apply common
software in education, work, leisure and home (Martin, 2006). Here it should be noted that skills
incorporated in both the mastery and application stage have a technical-procedural dimension. In
this context, Hakkarainen et al. (2000) combine the elements of both phases and describe technical
ICT skills as students’ mastery skills of ICT applications ranging from file management and text
processing to authoring tools and programming.
In the third and now dominant reflective phase, the focus of ICT literacy has shifted from basic skills
and use of applications to a more evaluative and critical use of computers. The acquisition of basic
ICT knowledge and skills is considered insufficient in terms of coping with the changes in our ever
evolving contemporary society (Voogt, 2008). For instance, retrieving data from the Internet not only
requires knowledge of search engines, but it also requires the ability to distinguish between relevant
and irrelevant data (Eshet, 2002). From this perspective, ICT competences can be situated in the 21
st
century skills movement. Rather than mastering basic ICT skills, ICT competence concerns problem
MANUS CRIP T
ACCEP TED
ACCEPTED MANUSCRIPT
3
solving, information processing, critical thinking, and creative and innovative ICT use (European
Commission, 2007). For example, ISTE’s National Educational Technology Standards for Students are
organized into the following six categories: 1) Creativity and Innovation; 2) Communication and
Collaboration; 3) Research and Information Fluency; 4) Critical Thinking, Problem Solving, and
Decision Making; 5) Digital Citizenship; and 6) Technology Operations and Concepts (ISTE, 2007).
According to Markauskaite (2007), ICT literacy refers to the interactive use of 1) general cognitive
capabilities, and 2) technical capabilities in order to successfully complete cognitive information and
ICT-based tasks. Definitions of ICT literacy in general cover both sets of capabilities in different areas
of problem solving and other generic activities, such as the ability to use technology and
communication tools to identify, access, manage, integrate, evaluate and create information, such
that individuals can function proficiently in our knowledge society (ETS, 2002; European Commission,
2007). Furthermore, Markauskaite’s (2007) description of ICT literacy is strongly related to the notion
that the mastery and application phases are subordinate to the reflective phase (Martin, 2006) i.e.,
the technical and application oriented skills need to be mastered in order to come to the more
critical, higher-order ICT competences. Within the context of the reflective phase, this study
perceives ICT competence as a multilayered and complex construct. An ICT competence refers to a
higher-order learning-process oriented competence used in complex, authentic and unpredictable
situations, and is underpinned by technical and application ICT knowledge and skills (Authors, 2013).
Research on the assessment of ICT competences can be divided into studies using self-reported
measures of ICT competence or ICT self-efficacy (indirect measurement) and studies using an
observation or performance-based approach (direct measurement) (Litt, 2013). The literature
indicates that most of the research is directed towards self-reported measures of ICT competences
or ICT self-efficacy. However, such indirect measures can suffer from validity problems as their
results are based on pupils’ own judgment and expectations of successfully performing computer
and internet related tasks (Hargittai, 2005; Meelissen, 2008; Merritt, Smith, & Di Renzo, 2005). As
self-report data do not always accurately reflect pupils’ actual ICT competences, conclusions drawn
from such studies can be misleading. On the other hand, direct measurement methods gather data
on pupils’ actual performance by analyzing observable, performance-based data, such as simulation-
based tasks or portfolios (Messick, 1994). Such tasks are more authentic and therefore considered as
more valid (Wirth, 2008). In order to tackle the validity problem of self-report bias, this study used a
direct measure to assess primary school pupils’ actual ICT competences. This direct measure is based
on an analysis of pupils’ performance on simulation-based hands-on tasks with a computer (Authors,
2014).
2.2 Digital information processing and communication
In order to measure the complexity of an ICT competence in a direct and valid way, a performance-
based test with authentic tasks was used in this study. Details on the development and validation of
the test can be found in Authors (2014). Because the administration of a performance-based test
takes time, it was not feasible to measure all of the competences included in the broad construct of
ICT competence. For example, the construct of ICT competence not only refers to the ability to
locate, manage or process digital information, but also refers to more creative and expressive forms
of digital media production and social online activities (Ito et al., 2009). Digital information
