data analytics and decision-making in education: towards the ... · data analytics and...

1

Agasisti & Bowers (2017)

Data Analytics and Decision-Making in Education: Towards the Educational Data Scientist as a Key Actor in Schools and Higher Education Institutions

Tommaso Agasisti Alex J. Bowers Politecnico di Milano School of Management Teachers College, Columbia University

[email protected] [email protected] ABSTRACT:1 In this chapter, we outline the importance of data usage for

improving policy-making (at the system level),

management of educational institutions and pedagogical

approaches in the classroom. We illustrate how traditional

data analyses are becoming gradually substituted by more

sophisticated forms of analytics, and we provide a

classification for these recent movements (in particular

learning analytics, academic analytics and educational data

mining). After having illustrated some examples of recent

applications, we warn against potential risks of inadequate

analytics in education, and list a number of barriers that

impede the widespread application of better data use. As

implications, we call for a development of a more robust

professional role of data scientists applied to education,

with the aim of sustaining and reinforcing a positive data-

driven approach to decision making in the educational

field.

Keywords: Data analysis, learning analytics, data-driven decision

making, educational management and policy, data scientist

1. Setting the stage: defining data mining, learning

analytics and academic analytics in the educational

field The use of data for decision-making in educational

institutions is neither a new topic nor an unknown practice.

Indeed, since a growing awareness dating back to the

1 This document is a pre-print of this manuscript, published in

the Handbook of Contemporary Education Economics.

Recommended Citation:

Agasisti, T., Bowers, A.J. (2017) Data Analytics and Decision-

Making in Education: Towards the Educational Data Scientist as

a Key Actor in Schools and Higher Education Institutions. In

Johnes, G., Johnes, J., Agasisti, T., López-Torres, L. (Eds.)

Handbook of Contemporary Education Economics (p.184-210).

Cheltenham, UK: Edward Elgar Publishing. ISBN: 978-1-

78536-906-3 http://www.e-elgar.com/shop/handbook-of-

contemporary-education-economics

1990s, school principals, teachers, parents, stakeholders

and policy-makers started looking at quantitative data as

an indispensable source for making decisions, formulating

diagnoses about strengths and weaknesses of institutions,

and assessing the effects of initiatives and policies, etc.

(Bowers, 2008; Mandinach, Honey, Light, & Brunner,

2008; Wayman, 2005; Wayman, Stringfield, &

Yakimowski, 2004). The diffusion of organizations and

initiatives such as Education for the Future

(http://eff.csuchico.edu) in the USA, or the DELECA –

Developing Leadership Capacity for data-informed school

improvement project (http://www.deleca.org) in Europe,

give testimony to a growing attention to the clever,

intensive and informed use of data for improving schools’

activities and results. The commitment to engage with a

stronger use of data became quite widespread across

schools, sustained by the evidence that “(…) the use of

data can make an enormous difference in school reform

efforts, by helping schools see how to improve school

processes and student learning” (Bernhardt, 2004; p. 3).

On policy grounds, the movement for adopting more

evidence-based educational policies and practices (Davies,

1999; Slavin, 2002) stems from the use of high-quality,

originally developed data about academic results obtained

through specific initiatives. The current debate of the

present chapter is focused on the potential of so-called

“big data” to transform education (Daniel, 2015; Cope &

Kalantzis, 2016), as it is rapidly doing in several aspects of

social life (Gandomi & Haider, 2015; Schutt & O'Neil,

2013).

Recently, one of the first developed areas of research that

has worked to systematically use the power of quantitative

analyses in the field of education has been the field of

Educational Data Mining (EDM) (Baker & Yacef, 2009;

Koedinger, D'Mello, McLaughlin, Pardos, & Rosé, 2015).

The baseline idea was quite simple in origin: applying the

techniques, methods and approaches of Data Mining and

Business Intelligence to another sector, after having

experimented with it in business, health care, genetics, etc.

In this sense, EDM “(…) seeks to use data repositories to

better understand learners and learning, and to develop

http://www.e-elgar.com/shop/handbook-of-contemporary-education-economics


http://eff.csuchico.edu)/

http://www.deleca.org)/

2


computational approaches that combine data and theory

to transform practice to benefit learners” (Romero &

Ventura, 2010, p.601). In such a perspective, the first aim

of an analyst who wants to apply data mining to education

is to collect raw data about the educational activities and

processes (at individual, organizational and the system

level) that have no sense if read without a theoretical

background as a lens (Bowers, Krumm, Feng, & Podkul,

2016). Such an effort has been traditionally very strong,

because of the way that data are stored in the educational

institutions – namely, following administrative rules and

procedures, and with little use of technology (Cho &

Wayman, 2015). Things are changing rapidly, and today

many schooling datasets are huge, well-classified, readily

available, and much diversified (Behrens & DiCerbo,

2014; Bowers, 2017; Bowers, Shoho, & Barnett, 2014;

Feng, Krumm, Bowers, & Podkul, 2016; Koedinger et al.,

2010). This evolution has been immediately caught by

researchers, and applications of sophisticated data mining-

style analyses to those datasets have generated a rich

stream of academic papers, reports, and practical

applications (Baker & Inventado, 2014). In a first stage,

then, the research aimed at mainly finding important and

frequent recurrences and patterns in available data,

something that users of this information usually do not

pursue (Bienkowski, Feng, & Means, 2012). However,

opportunities offered by developments in statistics,

operations research and information and communication

technologies (ICT) are today further empowering the

potential of data analysis, by moving towards more

sophisticated approaches that are classified under the name

of “analytics”. Using a definition of Chen et al. (2012,

p.1174): “(…) Data analytics refers to the BI&A (Business

Intelligence & Analytics) technologies that are grounded

mostly in data mining and statistical analysis. As

mentioned previously, most of these techniques rely on the

mature commercial technologies (…)”. The same

movement of analysing in more complex and structured

ways a large amount of quantitative information has been a

focus also of the public sector as a whole (Yiu, 2012;

Agasisti et al., forthcoming), and social services, within

which the educational sector is one of many, as well as

health care (Raghupathi & Raghupathi, 2014), social care,

etc.

Our contribution is focused on the more diffused uses of

analytics in the field of education, among which “Learning

Analytics” stands out as the recent main commonly

adopted definition and approach (Baker & Inventado,

2014). As recalled by Ferguson (2012, p. 305), the

application of analytics to the field of education can be

defined as follows: “Learning Analytics is the

measurement, collection, analysis and reporting of data

about learners and their contexts, for purposes of

understanding and optimising learning and the

environments in which it occurs”. In practical words,

Learning Analytics (hereafter, LA) embraces the

methodological and empirical efforts to collect data about

the determinants of the educational (learning) process, to

analyse it and understand its determinants (Gašević,

Dawson, & Siemens, 2015). A specific attention of this

stream of studies is focused on online learning; the amount

of information stored in the knowledge management

systems (KMS) that facilitate the interaction between the

students and the teachers is helpful for analyses – and it

permits going beyond the even larger availability of

administrative datasets (Andres, Baker, Siemens, Gašević,

& Spann, in press; Chung, 2014; Dawson, Gašević,

Siemens, & Joksimovic, 2014). In this perspective, the

promise of analytics evoked by Siemens (2010) can be

more easily satisfied by online learning tools: “Learning

analytics is the use of intelligent data, learner-produced

data, and analysis models to discover information and

social connections, and to predict and advise on learning”.

It should be stressed here that the growing extensive use of

digital learning tools and the wider development of

approaches for analysing educational data are an

intertwined phenomena. As explained by Siemens (2013;

p. 1381) “(…) Through the use of mobile devices, learning

management systems (LMS) and social media, a greater

portion of the learning process generates digital trails.

(…) and data trails offer an opportunity to explore

learning from new and multiple angles”.

Therefore, as acknowledged by Siemens himself, the

EDUCAUSE’s Next Generation learning initiative offers a

broader and cleaner definition, which is “the use of data

and models to predict student progress and performance,

and the ability to act on that information”2. In the last

definition, the explicit prevision of the “use” of (analysed

and elaborated) information introduces another crucial

feature of LA, which is the managerial and policy

implication derived from the analysis – that is to say,

school principals and policy makers put actions in practice

based on their judgments about the results of the analyses

(Bienkowski, et al., 2012; Bowers, 2017; Bowers, et al.,

2014; Feng, et al., 2016). In such a perspective, scholars

involved in LA tend to have a different approach from

academic educational scientists that focus solely on the

2 See the citation of this definition in the blog EdTech Digest,

May 10, 2012

(link available:

https://edtechdigest.wordpress.com/2012/05/10/learning-

analytics-the-future-is-now/)

https://edtechdigest.wordpress.com/2012/05/10/learning-analytics-the-future-is-now/)

https://edtechdigest.wordpress.com/2012/05/10/learning-analytics-the-future-is-now/)

3


results of studies (also suggesting implications) as LA

researchers tend to show very practical uses of the findings

for acting in the real life of school management, teaching

practices, or policy decisions to be taken. Seen from an

institutional perspective, one general characteristic of

educational data analytics (and LA in particular) is that the

effort traditionally made by individual teachers or specific

groups of teachers, i.e. the use of data for improving

educational experiences and results, is now made

systematic at the organizational level (Bowers, 2017;

Bowers, et al., 2016). In other terms, it is no more the

single responsibility of individuals in schools to use

quantitative evidence for rethinking and redesigning

didactic activities and interventions (Cosner, 2014; Farley-

Ripple & Buttram, 2015; Schildkamp, Poortman, &

Handelzalts, 2016), as the recent research argues that data

analytics is also an institution’s task – and, of course, the

tools that an institution can use for this aim are much more

powerful than those in the hands of volunteer teachers and

assistants (Bowers, et al., 2014).

The instruments that can be used for analytics, as well as

its objectives (i.e. the specific research and practical

questions to be answered), can be substantively different.

As a consequence, an attempt is recently emerging for

conceptually defining different fields and communities of

scholars and practitioners that engage with use of data for

educational improvement. We propose here a distinction

between three different approaches: educational data

mining (EDM), learning analytics (LA) and Academic

Analytics (AcAn). Such classification must be intended as

provisional, indicative and not prescriptive, and is based

on our critical reading of the current state-of-the-art of a

literature that is still in its evolution. As a first

approximation, the three approaches can be distinguished

on the basis of their purpose:

EDM uses data mining techniques applied to data

about the learning process, with the aim of

understanding patterns and recurrences (Romero &

Ventura, 2007; Baker & Inventado, 2014). As defined

by Scheuer & McLaren (2011, p. 1075): “[EDM aims

at] developing, researching, and applying

computerized methods to detect patterns in large

collections of educational data – patterns that would

otherwise be hard or impossible to analyse due to the

enormous volume of data they exist within”.

LA incorporates many of the models from EDM while

focusing on the teaching and learning activity; in this

sense, its main aim is to better inform teaching

practices (Baker, 2013). Bach (2010) states: “Learning

analytics is defined as the use of predictive modelling

and other advanced analytic techniques to help target

instructional, curricular and support resources to

support the achievement of specific learning goals”. A

feature of this definition is a major attention to the

output of the process, i.e. the achievement goal – LA

uses its tools to shed light on those factors that are

determinants of learning outputs.

AcAn, instead, focuses on organizational-level levers

that can be activated to change (improve) educational

activity and its results. Baepler & Murdoch (2010)

conceive that “Academic analytics combines select

institutional data, statistical analysis, and predictive

modelling to create intelligence upon which students,

instructors, or administrators can change academic

behaviour.” Beyond the specific context of higher

education, this type of focus in K-12 schools and

districts has recently also been termed “Organization-

level Data Analytics” (Bowers, 2017). As stressed by

Campbell & Oblinger (2007, p. 42) in their seminal

study on AcAn, this can be “(…) thought as the

practice of mining institutional data to produce

‘actionable intelligence’”. Thus, here the role of school

principals and managers – more than teachers – is at

the heart of the use of results. In one meaning

suggested by Goldstein (2005), Academic Analytics

can also be intended as the systematic use of data for

generating suggestions aimed at improving the internal

efficiency of educational institutions’ operations and

managerial processes (i.e. personnel management,

procurement, etc)3.

On practical grounds, the three approaches are hardly

separable, as tools, research questions, policy, and

managerial implications tend to be quite similar (as the

definitions above suggest). A simplified version of this

classification is the one proposed by Siemens & Long

(2011), who defined two broad areas (LA and AcAn) on

the basis of their different areas of focus: while LA

considers only the learning process, AcAn applies

techniques taken by business intelligence to education,

with the aim of generating ‘value’ for the institutions’

managers and policy makers. In particular, they suggest a

synthesis about how the various types of analytics can

differ by level of application (departmental, course-level,

institutional, etc.) and can benefit very different groups of

interest, such as students, faculty, school managers, etc.

This type of work is also analogous to recent writings on

the application of “data science” to organizational

improvement, where analysts work to translate data into

3 For additional details about the role of AcAn for facing

administrative and operational concerns (especially in the HE

field), see also Fritz (2011) and Goldstein & Katz (2005).

4


Table 1. Main differences between educational data mining, learning and academic analytics – a classification

Type of analytics Level or object of analysis Who benefits?

Educational Data Mining Course: learners’ profiles

Institution: patterns and recurrences across courses

Researchers and analysts, faculty, tutors

Learning Analytics

Course: social networks, conceptual development, discourse analysis, "intelligent curriculum"

Learners, faculty, tutors

Sub-organization (eg. Department): predictive modelling, patterns of success/failure

Learners, faculty

Academic Analytics

Institution: learners’ profiles, performance of academics, knowledge flow, institutions’ results

Administrators, funders, marketing

Regional (state/provincial): comparison between systems (performances, profiles, observable/administrative differences), benchmarking of institutions within the system

Funders, administrators

National and international: comparison between systems (performances, profiles, observable/administrative differences), benchmarking of institutions within the system

Governments, educational authorities, researchers and analysts

Source: Authors’ elaborations, originally inspired by Romero & Ventura (2010) and Siemens & Long (2011).

knowledge and action through data mining and

visualization, but also through interfacing with

organizational leaders and stakeholders to inform

evidence-based decisions (Bowers, 2017; Feng et al.,

2016; Schutt & O'Neil, 2013). Recalling the classification

presented above, EDM should not be regarded as an

approach in itself, but instead as a method (technique) that

can be indifferently used for LA or AcAn purposes. In our

own classification and proposal used in this chapter (Table

1), however, EDM must be considered as a distinct area of

research and practice, which is conceptually different from

the use of analytics techniques, and is helpful solely for

preliminary identification of patterns in data, that can be

subsequently used after additional analysis (for teaching,

in a LA perspective, or for organizational improvements,

in a more AcAn-oriented approach).

A common feature of the three approaches is that they

share the “loop of data” that is useful for using quantitative

information for supporting the decision (Figure 1)

(Bowers, et al., 2016; Mandinach, et al., 2008; Marsh,

2012; Schutt & O'Neil, 2013, Siemens, 2013). More

specifically, the first step (Collection & Acquisition)

consists of identifying the relevant datasets that are

potentially useful for answering the questions posed by

analysts and institutions’ managers. Then, through

“Storage”, datasets are systematically inserted in the

availability of official institutions’ registers. Experts in

these institutions should then revise the structured and

unstructured data, with the special purpose of creating new

datasets that can be effectively used for subsequent

analyses (“Cleaning”). Before starting with the empirical

modelling, datasets are subjected to a process of

“Integration”, which is characterized by the merging of

different sources –indeed often the analysis for answering

the question(s) necessitates various angles to study the

phenomenon under scrutiny. The next step is that of the

“Analysis”, where statistical, econometric and business

intelligence techniques are applied to the relevant datasets,

with the aim of detecting interesting patterns and results,

and of commenting and interpreting the main findings.

After this phase, such results should be synthesized and

explained. This is the objective of the step “Representation

and Visualization”, in which the experts that conducted the

analyses should work closely with educational managers to

understand the best way to illustrate the results, and to

support the awareness of the new knowledge generation

within the organization. As discussed in more detail below,

5


Figure 1. A data analytics model in education

Source: Authors’ elaborations, originally inspired by Siemens (2013).

this is where we see the most value for developing the

position of the data scientist who can work as a ‘bridge’

between data analysts and educational managers. Lastly,

the evidence from empirical analysis can be used for

realizing “Action(s)”, that is to say to design remedial

interventions, developing new (or modified) curriculum,

creating early warning systems and preventing phenomena

that harm student achievement, stimulating the

development of innovative educational strategies by

(groups of) teachers, etc. Of course, the process is

recursive, in the sense that new actions should be judged

through new data, which in turn require new cleaning,

integration, analysis and representation, and so on and so

forth.

The debate about the use of analytics in the educational

field is empowered by a set of policy and managerial

reasons. First of all, the discourse about the efficiency of

public spending (“produce more with less”), which is a

focus of education systems globally, requires an

understanding about the determinants of effective learning,

through making use of all the tools and opportunities that

new technologies allow. In one sense, the cost of education

cannot be considered anymore as granted, and should be

instead questioned by its efficiency (i.e. the ability of

generating the highest possible level of output) (Ingle &

Cramer, 2013; Levin et al., 2012). In this direction,

stagnation in graduation rates for high schools and

universities, as well as high levels of dropout from

education, in both the US and Europe (Murnane, 2013; de

Witte et al., 2013) represent a negative signal for

productivity of educational activities (Balfanz, Herzog, &

MacIver, 2007; Bowers, Sprott, & Taff, 2013; Riehl, 1999;

Rumberger & Palardy, 2005), that then should be further

explored with all the available instruments. Current global

trends in education accountability then suggest that the

policy makers’ perspective is to make managers of

educational institutions more and more responsible for

their results: performance-oriented legislations like the No

Child Left Behind (NCLB) Act and Every Student

Succeeds Act (ESSA) in the USA, and the

recommendations by the European Commission about the

efficiency of educational systems (European Commission,

2006) must be read in this light (Leithwood, 2013; Ni,

6


Bowers, & Essewein, 2016). Nevertheless, such policy

levers are problematic if knowledge about the learning

processes is not advanced. Indeed, this is the area where

analytics can be decisive in the future.

As economists of education, education sociologists, and

applied data miners and statisticians, we concur with the

work of this community of scholars with a particular

emphasis on advancing the application of analytics in this

domain, towards a closer link between results obtained

through analytics and the role of resources, managerial

processes, and incentives. Said another way, our goal is to

link the evidence from analytics with models of economic

behaviour of individuals and organizations through

building bridges in the research literature between these

domains. In this sense, when possible, we rely on some

results from the economics of education literature, to show

how they could be integrated with current debates in the

communities of learning and academic analytics. In a

broader sense, we aim to make our contribution to move

the debate from the technicalities behind modelling the

learning process towards the theory and empirical.

Specifically, we look to focus on econometric analyses of

educational processes within the framework of educational

production functions (EPFs), on the role of EPFs as a

model for studying education used by economists, in the

same spirit of Hanushek (1995). Therefore, the way for

integrating analytics objectives and the tradition of

education and economics studies is in the awareness that

“In education, the value of analytics and big data can be

found in (1) their role in guiding reform activities in (…)

education, and (2) how they can assist educators in

improving teaching and learning” (Siemens & Long,

2011: p. 38), as these are the ultimate goals of research in

the field when empirical analysis is used for investigating

the determinants of educational results (Coleman et al.,

1966).

This chapter aims at proposing a concrete step further in

the development of data analytics in education. Our feeling

is that, despite the rapid growth of attention towards the

role of data and quantitative information for exploring and

analysing educational patterns and results, there is still a

relevant separation between decision-makers (principals

and middle managers at the institution level, politicians at

the governmental level) and data analysts and researchers.

In a certain sense, the former actors are aware of the great

potential that resides in data, but consider their expertise as

technically inadequate to use the analyses. The data

analysts, on the other side, are satisfied with their

empirical (academic) work, and do not enter the practical

life and reality of school management and improvement.

The way we see as potentially innovative is to promote the

diffusion, within schools, of a new professional profile,

that of the educational data scientist, who owns the

technical skills to collect, analyse and use quantitative

data, and at the same time the managerial and

communication skills to interact with decision-makers and

managers at the school level to individuate good ways of

using the information in the practical way of improving

practices and initiatives.

The chapter is organized as follows. In the next section §2,

we review the literature about the use of data for informing

policy making (§2.1) and institution-level practices and

teaching (§2.2). Section §3 aims at presenting some

examples of recent tools used for data analyses and

analytics, ranging from traditional statistical explorations

to complex instruments of learning and academic

analytics. Section §4 describes the main challenges that

can constitute a barrier for the diffusion of analytics in

supporting educational decision-making processes. Lastly

section §5 concludes through the proposal of developing

the educational data-scientist as a key profession in

schools and universities.

2. Use of data analyses (and analytics) in the

educational arena

2.1 Data use by policy-makers (for implementing,

managing and evaluating policy interventions)

The interest of policy-makers in using quantitative

information for guiding policy interventions in education

is relatively recent (Jimerson & Childs, in press), and is

based on the awareness that, in the words of Andreas

Schleicher (Director of Education at OECD): “Data are

crucial to understanding the effect policies have on

education systems at a local level” (Schleicher, 2016). The

basic idea is that data – when collected and analysed in a

proper way – can be helpful for assessing if policies are

effective or not, in other words whether they reach the

objectives they pursued, and under which conditions this

happens.

A specific set of studies argue for the use of international

datasets for comparing differences in the educational

policies across countries – see Hanushek & Woessmann

(2010). The use of data taken from the most developed

assessment of students’ achievement in an international

fashion is particularly informative for policy-makers, as it

allows an understanding of which practices and policies

are working in different countries, holding other systemic

differences constant.

7


Behind this approach, the main assumption is that

“institutional structures” (like school autonomy and

accountability, competition between schools, and

assessment procedures) that are typically invariant at the

country level are actually playing a role in shaping

systematic differences in students’ results across countries.

Following this assumption, scholars and analysts have

applied analytics to data for suggesting policies that can be

adopted in various contexts to improve educational results,

by learning from data compared across different

contexts/settings. For example, Woessmann (2008) raises

the issue of designing output-oriented policies, ranging

from early childhood education and schools to higher

education, to efficiency improvement and equity of results;

Bishop (1997) assesses the effects of national curriculum

and testing on student achievement; Brunello & Checchi

(2007) find evidence of detrimental effects of early

tracking – i.e. students in countries where early tracking is

practised obtain lower test scores, all else equal. A

systematic review of policy evaluations that can be

conducted – with qualitative and quantitative methods –

taking advantage of international comparative assessments

can be found in Strietholt et al. (2014). Economists,

sociologists, and educational scientists moreover, have

illustrated the power of data to inform policy-makers about

the effects and prospective results of interventions at

country level. There are several examples of studies that

empirically assess the educational outputs of policies,

through the available administrative datasets or surveys at

the country level, and/or extracting country-specific data

from the international assessments cited above (see, for

example, Woessmann, 2008).

A trend that deserves specific attention is the growing

confidence in the evidence-based approach, inspired by

different disciplines (such as Medicine), and that consists

of conducting rigorous “field experiments” (see Slavin,

2002; Mosteller & Boruch, 2002). The intention of this

approach is to provide even more robust evidence about

“what works” in education to improve students’ and

schools’ performance, and this would be suggested as a

gold standard to policy-makers before undertaking

structural reforms (in a logic of “trial” before systemic

application) (Schneider, Carnoy, Kilpatrick, Schmidt, &

Shavelson, 2007). Honig & Coburn (2007) show that

policy officers in a United States school district central

office read empirical evidence-based analyses, but the

interpretation of results as well as the actual use is shaped

by the background and characteristics of the officers.

Additionally, the “hard” information and evidence is

complemented by officers using personal and local

knowledge of educational activities and results. These

elements remind us that the attitude of using data is not a

purely technical one, but requires human interaction and

proactivity for securing a meaningful policy-making

process (Marsh, 2012). When considering the use of data

by academic scholars, therefore, it should be kept in mind

that the primary goal of academic research is not

necessarily policy advice. As a consequence, there will

also be a “gap” between evidence provided by academic

research and implemented, subsequent policies – if any

(Whitty 2006). Such a gap is normal, and in a certain sense

is desirable as it helps in preserving research independence

and ensuring the work is not influenced by policy-making.

A completely different story comes out when considering

the influence that think-tanks, foundations or institutional

entities exert on politicians and policy-makers by working

with educational data. In Europe, for instance, two

networks funded by the European Commission (European

Expert Network on Economics of Education – EENEE,

and Network of Experts on Social Aspects of Education

and Training – NESET) are responsible for writing

Reports for the Commission, that then go on to influence

its official communications. OECD diffuses concise notes

with policy suggestions based on data analyses of its

Program for International Student Assessment (PISA),

named Pisa in Focus. In the US, various institutions issue

Reports that talk directly to policy actors, and aim at

influencing their subsequent actions – see, for instance, the

Reports from Brookings Institutions (e.g., Dynarski,

2016), Friedman Foundation for Educational Choice

(Forster, 2016) and Cato Institute (Bedrick et al., 2016).

All in all, the explicit aim of these Reports and activities is

to stimulate policy debates and sustain political ideas; and

policy-makers can selectively consider the results coming

out from these studies to justify their positions, or to

challenge political ideas sustained by their opponents. This

is also a use of data that should be considered; and this

attitude is important to be part of actors’ awareness – i.e. if

data and evidence are presented clearly (and

systematically) to policy actors, they reinforce their

intentions with the “power of data” – on the complex

nexus between data and governing educational systems

and policies (see crucial debates in Grek & Ozga (2009)

and Borer & Lawn (2013)).

2.2 Data use by the managers (for managerial purposes)

and by the teachers (for purposes of improving teaching

effectiveness)

Across the research literature on leadership and data use,

much of the research has focused on data-driven decision

making by leaders in elementary, middle and secondary

8


schools (Bowers et al., 2014; Cosner, 2014; Marsh &

Farrell, 2015; Schildkamp, Karbautzki, & Vanhoof, 2014;

Schildkamp, Poortman, Luyten, & Ebbeler, in press; Yoon,

in press). In this literature, researchers work to understand

the role of data in the everyday work of teachers and

school principals, with a focus on the role of leaders to set

the stage for productive data use. In their review of the

relevant literature, Bouwma-Gearhart & Collins (2015)

show how the educational institutions of all grades in both

Europe and US are subjected to a greater emphasis on the

“culture of evidence”, where the use of data can be

instrumental to orient interventions. In this vein, the role of

quantitative information and evaluation is becoming more

and more critical. Reviews of studies about how

institutions use data present two types of challenges: (i)

organizational (how decision-makers and data producers

are linked; which processes and incentives are established

for facilitating data use, etc.) and (ii) technical (that deal

with data integrity, validity, timeliness, etc.). On the

practice side, recommendations stem from normative case

studies of data use in practice, focusing on modified Plan-

Do-Study-Act improvement cycles, and concrete action

steps that leaders and teachers can take to use data as a

means to build capacity through honest and trusting

dialogue with and between teachers (Bambrick-Santoyo,

2010; Boudett et al., 2013; Piety, 2013). Much of this work

is focused on helping school practitioners move from a

culture of high inference and low evidence, to high

evidence with low inference (Bowers et al., 2014), with a

focus on the purposeful use of data and information

feedback cycles (Mandinach, et al., 2006; Marsh, 2012).

Indeed, research that delves deeply into the data use

culture in schools has shown that successful use of data by

management depends on teacher and leader data and

assessment literacy (Datnow & Hubbard, 2015; Demski &

Racherbäumer, 2015; Mandinach & Gummer, 2013).

However, while the research on data use articulates a

logical evidence-based improvement cycle that moves

from data, to information to knowledge, action and

feedback to inform each step of the cycle (Mandinach et

al., 2006; Marsh, 2012), much of this work devotes little

attention to the difficult task of turning data into

information and knowledge (Bowers, 2017; Bowers et al.,

2016; 2014). To date, the research as well as the normative

training literature for teachers and leaders has a strong

focus on “data” that mostly includes aligning teacher

interim assessments to state mandated standardized test

scores (Bambrick-Santoyo, 2010; Boudett, et al., 2013;

Coburn & Turner, 2011; Farley-Ripple & Buttram, 2014;

Piety, 2013; Turner & Coburn, 2012). Additionally,

analysis mostly focuses on descriptive statistics (Bowers et

al., 2014; Rutledge & Gale Neal, 2013), that can lead to an

over emphasis by school leaders on a deluge of bar graphs

and tables that serve mostly to disengage teachers from

data rather than create informative dialogue (Murray,

2013; Reeves & Burt, 2006).

To address this issue, recent research at the elementary,

middle and secondary school levels has encouraged an

increased role for data analytics in school decision making

(Bowers 2017; Bowers, et al., 2016). This work has been

mainly focused in the U.S., and has articulated a stronger

role of leaders in selecting broader definitions of data to

help inform decisions (Bernhardt, 2013; Bowers, 2009),

building teacher and leader capacity to create stronger data

use systems through more valid and informative

assessments (Gummer & Mandinach, 2015; Jacobs et al.,

2009; Popham, 2010) and purposeful data collection

systems (Wayman, et al., 2015; 2012; 2007). However,

missing from much of this research is a discussion of how

data are turned into information and knowledge to be used

for evidence-based decision-making (Bowers et al., 2014;

2016). As discussed in the introduction to the present

chapter, an emerging line of research has suggested that

the fields of EDM and LA may help inform data use in

schools (Bienkowski et al., 2012; Means et al., 2010). In

this work, EDM and LA are categorized under the larger

framework of data analytics in schools, as a means to bring

a deeper understanding to the data beyond descriptive

statistics, in which the data analytics process provides an

opportunity to “make visible data that have heretofore

gone unseen, unnoticed, and therefore unactionable”

(Bienkowski et al., 2012: p. ix).

A last point that is worth illustrating in this section is the

potential of technology in enhancing the understanding of

the educational processes, and this can in turn be

transformed by teachers as information to be used for

improving teaching practices and results (for a theoretical

and methodological essay on this point, see Laurillard,

2008). In practice, technology can help teachers to become

‘action researchers’ for their own teaching roles; by

tracking the activities done by students in their courses,

they can understand more about how students learn

(Behrens & DiCerbo, 2014; Chung, 2014), and can revise

their strategies for learning – also experimenting with new

mixes of tools, from more traditional lectures to the

introduction of videos, forums, blogs, online interactions,

etc (Wayman, et al., 2004). It must be clear, here, that the

systematic use of automated, collected information may

never substitute the role of teachers in understanding

which practices and strategies are better for their specific

courses, but IT can complement this continuous effort for

9


improvement by adding more evidence to the choices to be

made (Gašević, Dawson, & Siemens, 2015). Williamson

(2016) emphasizes this new set of opportunities offered by

IT, and proposes the expression “digital education

governance” where experts in collecting, managing,

analysing and visualizing data have the power to influence

decisions taken by various actors. Specifically, the author

indicates a trend where “digital data-based (…)

instruments are employed to perform a constant audit of

students’ actions” (p. 138)4.

3. Tools for data analyses and analytics: some examples

of existent experiences

3.1 Examples of data analyses and analytics: different

tools for different units of study

In this section, we critically discuss some examples of

using data analyses and analytics for improving students’

results and success. For the sake of classification, we

divide the examples into four categories:

Use of data analyses and analytics to inform

policy-makers about phenomena that are affecting

system-level educational results (at territorial,

regional and/or national or even international

levels);

School-level information that would help a school

principal and middle management to understand

the main patterns followed by students who are

attending that specific institution;

Course-specific data, that are useful for

instructors, with the aim of receiving timely

feedback on activities undertaken by students, the

results, and can activate concrete actions for

improving results given the features of the course

itself;

Interfaces for managing data about individual

students. In this case, the LA tool has the main

objective to propose the courses which students

should choose or to provide predictions about

prospective results based on individual

characteristics, previous grades and/or other

additional administrative information.

4 In particular, the author illustrates two major experiences. One is the “Learning Curve” developed by The Economist, at system level – i.e. more pertinent to describe the influence at policy level, as among the group of examples in the section 2.1. The second example is the activity realized by Pearson in its research about Digital Data, Analytics and Adaptive Learning for creating data anticipations and forecasts about individual students.

The choice of the experiences contained in this section has

been based on two major criteria: (i) the reference to case

studies that are published in international, peer-reviewed

academic journals, and (ii) the application of experience

for some years, as this allows the reporting of existing

assessments of these experiences. By no means should

these initiatives be considered the “best practices” in the

academic and educational world, as they are neither the

first implemented, nor the most complete. Nevertheless,

these examples are among the most cited in the current,

sparse literature that considers the role of data analyses

more systematically, and thus we use these examples, to

illustrate how the results provided by these tools can be

used for managerial purposes.

The examples illustrated in this section are:

For the analyses of system-level determinants of

instructional results:

o PIRLS (Progress in International Reading

Literacy Study) and TIMSS (Trends in

International Mathematics and Science

Study) studies governed by the

International Association for the

Evaluation of Educational Achievement

(IEA), as well as PISA (Programme for

International Student Assessment)

administered by Organisation for

Economic Co-operation and Development

(OECD) – with a specific focus on

international analyses of the effectiveness

of national educational systems.

o A study for developing an early warning

system at national level, based on the use

of data for Florida public schools (a

description is in Koon & Petscher, 2015).

As an example of data analyses for supporting

school-level interventions, we rely upon the

experience developed within the Strategic Data

Project at Harvard University

(http://sdp.cepr.harvard.edu). In addition to this

Project, we present some insights from best

practices among British Schools reviewed for a

Report written for the UK’s Government (Kirkup et

al., 2005)5.

For the LA applied to the course level, the tool

Course Signal developed by Purdue University,

which is based on an algorithm that considers

personal characteristics and online activities to

provide students with early warning on potential

problems with passing their course (Arnold, 2010;

5 Another analogous experience, that can be of interest for readers is that of AltSchools, as described by Horn (2015)

http://sdp.cepr.harvard.edu)/

10


Arnold & Pistilli, 2012). Early applications date

back to 2003.

For the interfaces that provide information to single

students (and their instructors), the instrument

Degree Compass, a course recommendation system

trialled at the Austin Peay State University, and

then extended to a group of universities and

community colleges in Tennessee6.

An example of how the data can be used for analyses of

educational results at the system-level is the discussion

around the important international assessments of student

achievement, those conducted by IEA and OECD as

defined above. These organizations release official

Reports, every time that a new edition of these studies is

available (usually every three years), and they form a

baseline for policy discussions worldwide about how

educational systems should be reformed or restructured to

gain improvements in students’ experiences and results.

The Reports themselves provide practical suggestions in

this vein; for instance, the comments contained in the

Report about TIMSS 2011 for mathematics indicates that:

“Students with the highest mathematics achievement

typically attend schools that emphasize academic success,

as indicated by rigorous curricular goals, effective

teachers, students that desire to do well, and parental

support.” (IEA, 2012; p. 16). Some academic studies

emphasize how politics have been influenced by that

attention to these data (see Grek, 2009; Bieber & Martins,

2011; and Meyer & Benavot, 2013 among others). Lastly,

several scholars conduct secondary analyses on these data,

with the aim of exploring a plurality of educational aspects

(and characteristics) that are likely to influence students’

results – with the specific intention of suggesting practices,

reforms and policy interventions7.

6 A similar example that can be interesting for the reader is the tool “Check My Activity”, implemented by the University of Maryland (Fritz, 2010), which consists of a self-service diagnosis of each student’s activity within the environment of the LMS supporting teaching at the University, to be compared against the peers’ average. The initiative has been launched in 2008. Also, for a wider discussion of the existent dashboard applications that can support learners’ learning (in a LA framework), see Verbert et al. (2013). 7 There is a huge number of studies of this kind; among others, Hanushek & Woessmann (2010) maybe stand out as a clear example of an attempt to derive theoretical and empirical economic considerations using data analyses with these datasets. But also see these other interesting examples; Woessman & West (2006) on the role of class size; Agasisti (2013) for the potential effects of competition between schools; Schmidt et al. (2001) on the curriculum structure.

There are several examples of the systematic use of data

for assessing students’ performance at a national level;

virtually all Ministries and agencies that are responsible

for standardized tests in single countries create reports in

which the results are summarized and noted, by means of

quantitative indicators, to stimulate reflections on the

policies and interventions that improve academic results8.

Less frequent are the cases of employing articulated

statistical methodologies to analyse those data. In this

chapter, we propose one of these cases, which has been

prepared for the Institute of Education Sciences (IES) by

the Regional Educational Laboratory Southeast (REL-SE).

Using data about Florida public school systems, Koon &

Petscher (2015) illustrate how alternative statistical

methods can be used to identify students who are likely to

struggle in their educational outputs, and how developing

an early warning system at the system level may be useful.

Specially, the authors employ two approaches: a set of

logistic regressions and classification trees (CART). In

both cases, a dichotmous variable is used as the output of

the educational process, and is based on a standardized test

score (SAT-10), where scores at or above the 40th

percentile were coded as 1 for “not at risk”, and scores

below the 40th percentile were coded as 0 for “at risk”,

and a battery of test scores are used to predict the student

falling in one of the two categories. Then the two

approaches (logistic regressions and CARTs) are used for

estimating whether students “at risk” actually fail to meet

the expected educational standard. Although the methods

differ in their theoretical and conceptual backgrounds, the

results provided are quite similar (indices of classification

accuracy were used to assess differences in the results

between the approaches). The authors concluded that:

“The CART results were found to be comparable with

those of logistic regression, with the results of both

methods yielding negative predictive power greater than

the recommended standard of .90.” (p. 19). In light of

these similarities, the authors indicate their preference

8 Another example of instruments which are available at system level for supplying quantitative information to parents and relevant stakeholders are the publicly accessible websites, that benchmark information about test scores and other administrative features of single schools/institutions. Instances of this kind are the website MySchool.Edu, administered by the national government of Australia (see Mockler, 2013) or the various privately-run websites in the US context, as for instance Greatshools.org which effects are being studied by Lovenheim & Walsh (2014). Although we do not describe these specific examples in further detail here, it is important to be aware of the role that such internet-based information providers are playing in affecting data use and analytics in the educational arena.

11


towards a more extensive use of the CART approach, by

providing an interesting explanation on policy grounds that

stems from a higher level of clarity of how results are

used. In other words, while the tables containing the

logistic regressions’ main findings are adequate only for

technical readers, figures and tables produced from

CARTs are easily interpretable also for stakeholders

without a specific technical literacy on statistics. This is

crucial in the area of data analyses and data-driven

decision making. Indeed, the scope of analysts is not only

that of developing robust (and often sophisticated)

empirical models, but also to present the obtained results

in an understandable way for policy-makers (in this case,

at state/national level) so that they can be convinced to

take actions (Bowers, 2017; Schutt & O'Neil, 2013). In our

view, the area of elaborating methods for making the

findings clearer for key political actors is one where the

different competencies of analysts must develop, and when

the role of economists of education should be questioned

in terms of effectiveness.

Turning to the school level, the idea prompted by the

Strategic Data Project (SDP) at Harvard University is of

particular interest, given the explicit aim of training people

with a given mix of competencies to help school decision-

makers (and principals in particular) to make decisions that

are informed by evidence and quantitative data (Hallgren,

Pickens Jewell, Kamler, Hartog, & Gothro, 2013). As

indicated in the Project’s website, “SDP was formed on

two fundamental premises: (i) Policy and management

decisions can directly influence schools' and teachers'

ability to improve student achievement, and (ii) valid and

reliable data analysis significantly improves the quality of

decision making.”( http://sdp.cepr.harvard.edu) Of

specific value is the experience of the fellowship program

in that Project, that provides education for developing

skills in three main areas: measurement and analysis,

leadership/management and effective communication, and

research findings in education policy. The Fellows trained

in the project then have experience on real cases of use of

data, and are invited to write reports summarizing the

experience they have in schools with using data for

making decisions. For instance, Holt et al. (2015) wrote a

Capstone Report about their experience in Rochester

schools, in which they described how they use a plurality

of analytical methods (simple regressions, Lasso

Regressions9, CART, etc.) to describe the patterns of

students who do not succeed at Grade 3, by systematically

using key information about pre-schooling and results at

9 For an understanding of the way in which LASSO regressions do work, see Tibshirani (1996).

Grades 1 and 2. Based on their analysis, school district

policy-makers and principals developed a dashboard that

can now be used by the teachers in schools to monitor the

progress of their students against a predicted pattern of

educational results over time.

Kirkup et al. (2005) conducted a study for the National

Foundation for Educational Research (NFER),

commissioned by the Department for Education and Skills

(DfES). The research aims at identifying how British

primary and secondary schools use data for improving

teaching and learning activities10. The main results of the

study was that data usage is not useful per se; instead, “A

recurrent theme was that data only becomes effective if it

stimulates questions about the actual learning that is

taking place and how it can be developed further.” The

wider findings reported by the authors evidence that school

leaders are not searching necessarily for a “tool” that helps

them in using data, but instead a way of proceeding that

can involve different actors at the school level (teachers,

middle-management, etc.) in open dialogue about practices

that can be adopted for the improving results, and how

data can help in assessing the results of these practices.

Course Signal is one of the most famous examples of LA

tools presented by academics and practitioners, and

discussed in the scientific community. Course Signal is

defined by its developers at Purdue University (where it

was also used in experimental trials) as an “early

intervention solution”: in simple words, its main objective

is to data mine large amounts of data about students’

characteristics and their use of educational facilities to

provide real-time feedback to them (see Arnold, 2010;

Arnold & Pistilli, 2012). The target for the intervention is

instructors more than students; indeed, the output of the

predictions of academic success in a given course is

provided to the former, not the latter. The system is based

on the estimation (through data mining techniques) of a

“risk level” for each student in each course. The algorithm

that calculates such risk level is using four groups of

variables: (i) current performance (points earned in a

course until a precise moment); (ii) effort, that is

monitored through the interactions of students in the

Learning Management System; (iii) prior academic

10 Also on the use of data for school improvement in the UK case, see Demie (2013). Earl & Katz (2006) debate how school leaders should use data in a more conceptual way; Yang et al. (1999) created a guide for communicating the results to school heads, while Wayman & Stringfield (2006) claim for data usage to bringing improvements. Marsh et al. (2006) illustrated how data-driven decision making can be beneficial for schools.

http://sdp.cepr.harvard.edu)/

12


performance, through proxies like high school GPA,

grades in previous exams, standardized test scores, etc. and

(iv) students’ characteristics, such as residency, age, and

other administrative information. Once the risk-level for

the i-th student in the k-th course is produced, this is

transmitted to the instructor; he/she at that point can define

an intervention plan for critical cases, and such a plan can

range from simple e-mail, to a help-desk, to designing

individualized remedial lessons for the student. At a higher

level of decision-making, results are also used to

restructure curricula and the orientation programs. The

results obtained in the first application of Course Signal, as

indicated by Arnold & Pistilli (2012) suggest positive

effects on retention – the primary goal of the project.

Students that attended a class with Course Signal persist at

higher rates than their peers who did not attend classes

with Course Signal support. When interviewed, both

faculty and students report the behavioural effect of

knowing data and receiving feedback, respectively, as the

key channel through which the tool exerts its effects. Such

clues are a crucial feature of data analytics, moving actors

of the educational process (teachers, students and

principals) through the provision of actionable

information.

Degree Compass is a course recommendation system that

suggests the best patterns of courses that a higher

education student should take to maximize his/her

probability of success (see Denley, 2012; 2013; 2014). The

system is based on an algorithm, that is not dependent on

students’ preferences and willingness to take specific

courses (as recommendation systems usually are), but

instead on grade and enrolment data. By mining all records

about students’ characteristics, choices of enrolment and

grade obtained, the algorithm ranks courses according to

the probability that a student with a certain array of

features will succeed in passing it. Also, the system

designs the best pattern (i.e. the sequence of courses)

following the same approach; and it predicts grades that

would be obtained in each of the exams. All this

information is then transmitted to each student, through a

very simplified web-based interface, in which the strength

of recommendation of various course combinations is

expressed through the assignment of a number of stars

(from 1 to 5). In selecting the courses that are more

strongly suggested, the algorithm also considers the

choices already made by students (especially, the major

and previous exams). In this way “(…) the system most

strongly recommends those courses which are necessary

for a student to graduate, core to the institution’s

curriculum and their major, and in which the student is

expected to succeed academically” (Denley, 2014; p. 64).

An add-on feature of the system is also the software called

MyFuture, which provides information about degree

pathways and the transition between higher education and

the job market, by mining large datasets to obtain

predictions about those courses that are more likely to be

conducive to student success. Lastly, the characteristics of

the system are periodically discussed with faculty, to

obtain their suggestions about which elements should be

included in the calculations and predictions made by the

algorithm; through such interactions, a by-product is also

to obtain higher levels of involvement of faculty members

in the project.

Some preliminary results obtained with the use of Degree

Compass at both Austin Peay State and Tennessee’s

institutions, as reported by Denley (2014), seem

encouraging. On one side, the average grade obtained by

students after the implementation of the tool increased

substantially (although it is not clear whether this could be

due to grade inflation and/or through the steering of

students towards ‘easier’ courses). This benefit for

academic performance appears as uniformly distributed

across student subpopulations including minorities and

socioeconomically disadvantaged students; this is very

important because informative gaps for these groups has

been among the major motivations for the project. On the

other side, the system is very accurate in predicting

success – more than 80% of students who were predicted

to pass a given exam, actually did so. Lastly, there is a

significant gain in number of credits acquired by students

who choose the suggested courses, as compared with their

counterparts who did not choose them. Overall, the

positive experience with the instrument contributed to its

widespread and integrated use into the day-by-day

academic planning and monitoring activity at the

University. As students and their specific instructors are

the main target of the information provided by the tool,

one key aspect of the initiative’s success resides in the

simple usage of the interface for both actors. Future

developments of this experience refer to the ability of

collecting more granular data about how to best

communicate the information to students, and about how it

actually influences students’ choices (see Denley, 2013). A

last point to be illustrated here is the (indirect) positive

effect of the initiative on the economic challenges faced by

Austin Peay State University (APSU). Indeed, the State of

Tennessee introduced a major change in the formula used

for funding public colleges, by introducing graduation and

success rates – replacing the previous input-based system.

Given the substantial positive effect of Degree Compass in

increasing passing rates across students, APSU received a

substantial increase of institutional funding. This side-

13


effect of the introduction of an effective LA tool highlights

the potential synergy between the use of information for

students’ utility and the positive impact on institutional

management and performance.

3.2 Tools for Data Analyses and Analytics: summarizing

considerations

The presentation of the experiences in the previous section

highlight a couple of crucial points that are worth

particular attention in depicting the state-of-the-art. First,

most of the recent experiences of LA and academic

analytics are US-based stories. Although there are

potentially interesting cases also in Europe (see, for

example, Kickmeier-Rust & Albert, 2013), it is evident

that the practical use of tools for leveraging the value of

data in education is much more developed in the North-

American context. It is likely that European researchers in

various fields (computer science, education, economics,

sociology, etc.) will find a significant space to develop and

stimulate real-usage of new available techniques and

instruments for LA purposes. Therefore, diffusing good

practices in formal and informal settings will also help a

more systematic approach towards data-driven (evidence-

based) decision making in European educational

institutions. Second, many of the practical cases cited

above are focused on higher education and, overall, on the

use of informative dashboards by students and instructors.

In our opinion, two developments are desirable here:

larger amounts of data are becoming available also

for primary and secondary education. As a

consequence, researchers and practitioners should

start developing new types of tools for supporting

teachers in clarifying more the educational

strategies that they use, and the results that they

are able to obtain, by linking various sources of

data about the experience of students that may

result also in computer-based suggestions for

choices in subsequent educational steps and

grades;

most of the potential use of data is still unexplored

by the decision-makers (principals, deans,

managers, presidents, etc.) who can instead

establish and develop a real evidence-based

approach for management of educational

institutions, orientated towards greater efficiency

and student success. Formative initiatives,

benchmarking exercises and sharing of positive

experiences can be instruments for developing

awareness among decision-makers.

The emerging role of LA and AcAn as distinctive fields of

exploration and research is providing support for spurring

innovation and experiments in the discipline. For instance,

data analytics applications in elementary, middle and

secondary school research have most recently focused on

the issue of early warning systems and indicators

(EWS/EWI) for students at-risk of failure in school

(Bowers et al., 2013; Mac Iver, 2013). The study by

Knowles (2015) reports the case of a Wisconsin early

warning indicator data mining study, which is a good

example of how statistical modelling can be coupled with

big administrative datasets for scaling an early warning

system for high school completion to a state-wide

longitudinal data system. Our intuition – that is based on

cases presented, and on the recent patterns of the literature

about the use of data for supporting decision-making in

education – is that LA and AcAn are constituting the next

step of evolution for the data analyses in the educational

context. Although convinced by this dynamic, we are

aware of the severe limitations that the use of analytics in

education can encounter: the next section is devoted to

describing the main issues among them.

4. Barriers and impediments to the use of analytics in

education

The previous sections have introduced the motivation for a

more extensive use of analyses based on quantitative data

as improving performance of schools/HEIs (also with the

introduction of more aggressive analytics). We next

illustrate the potential drawbacks and obstacles that a

strong use of evidence-based decision making can

experience in the field of education.

The first problem is ethical, and resides in the intrusive use

of data that can threaten privacy and the question about

how the use of personal data should be limited (the so call

‘big brother’ risk) (Ifenthaler & Tracey, 2016; Slade &

Prinsloo, 2013; Spector, 2016). Many tools that can help

improve students’ performance are based on tracking

previous test scores, and this is embedded in normal

screening of student results. Therefore, for working in a

more effective way (i.e. for being more predictive of future

performance) the systems should also triangulate the

information with personal students’ characteristics. The

identification of individual students in the automated

algorithms raise serious privacy concerns, even though the

analytical tool is built in the students’ interest (see a

discussion in Fritz, 2011 – presenting the specific case of a

tool called Check My Activity).

In addition, the premise of the most recent, sophisticated

tools for analytics is that the analysts should be able to

collect together a massive amount of quantitative

14


information in their models, for coupling them with

learning theories and derive practical information to

support policy-making, institutional management and

pedagogical initiatives. This approach requires use of data

for purposes that are often different from the one for which

they were collected – i.e. for research and analyses, rather

than for administrative tasks. Also, there are general

ethical issues surrounding how acceptable it is to collect

and utilize personal information for researchers. Indeed, if

the models are employed to trace “profiles” of students

and instructors (Lawson, Beer, Rossi, Moore, & Fleming,

2016; Scholes, 2016; Willis, Slade, & Prinsloo, 2016),

unintended consequences can arise – such as the building

of automated algorithms to signal potential outcomes (i.e.

dropout or ineffective teaching) and/or the definition of

desirable learning patterns dependent upon personal

characteristics. Overall, these problems underline that it is

necessary to develop a series of guidelines that indicate

how data analyses and analytics, originally intended for

improving students’ satisfaction and results, do not result

in a set of practices that harm students’ and faculty’s rights

and privacy (see Willis et al., 2013). Williamson (2016)

warns against the risk of a presumed “techno-scientific

objectivity”.

As one potential guideline in this domain, as has been

argued recently across the “Open Access Science” domain

in general (Fecher & Friesike, 2014; Masum et al., 2013;

Molloy, 2011; Strasser, 2015), and in the machine learning

domain specifically (Braun & Ong, 2014), we concur with

recent authors that open access and open research

standards must be used in educational processes, especially

when machine learning algorithms are used to make high

stakes decisions on students, teachers or schooling

systems. As generally noted by Molloy (2011, p. 1),

“open” in a research and practice context is defined as

“The work shall be available as a whole and at no more

than a reasonable reproduction cost, preferably

downloading via the Internet without charge. The work

must also be available in a convenient and modifiable

form”. While open access data standards may not apply in

education, due to issues of student, parent and school

privacy (Willis, Slade, & Prinsloo, 2016), we argue here

that open code and open access standards (Stodden, Guo,

& Ma, 2013) must be used when data analytic or machine

learning algorithms are used to inform evidence-based

improvement cycles in schools, or to the extent that LA

algorithms make recommendations for content and

instruction for student learning. Fecher and Friesike

(2014), describe five schools of thought in the discourse on

open access including “The infrastructure school (which is

concerned with the technological architecture), the public

school (which is concerned with the accessibility of

knowledge creation), the measurement school (which is

concerned with alternative impact measurement), the

democratic school (which is concerned with access to

knowledge) and the pragmatic school (which is concerned

with collaborative research)” [italics original] (p.17).

Additionally, as a matter of education system efficiency

and ethical use of taxpayer resources, while student data

should not be open, the algorithms used in schooling

organizations should be published under open access

standards so that any student, parent, or concerned citizen

may examine the code that may recommend decisions on

instruction for their children, teachers and schools across

their community and context.

A second problem is more technical in nature, and is about

the complexity of data and data integration. Since

Bernhardt (2004) it has been clear how data analyses

require an adequate system of support to organize data and

use them properly; her call of extensive and sapient use of

DataWareHouses11 must be read in this light. The issue is

not only related to the informative system supporting data

analyses (which, however, is always a major issue for

single educational institutions) but also to the quality and

scope of available data. To assure that data uses in the

analysis effectively capture the phenomena to be

evaluated, data and indicators themselves should have

demonstrable robustness, clarity, and pertinence. In other

words, even when moving from simple data analyses to

more sophisticated analytics, the fundamental basic rules

about performance indicators still hold, such as reliability,

validity, accuracy and accessibility (Popham, 2010). A

certain angle of this perspective is related to the problem

of data veracity: it is without a doubt that maintaining data

accuracy when collecting and integrating millions of data

points from various sources is indeed very challenging.

Third, some authors underline how the creation of an

adequate platform for classifying, analysing data, and

creating support for decision-making is a costly investment

(see Goldstein & Katz, 2005). Williamson (2016) explains

11 As indicated by the Business Dictionary online, a Data Warehouse is (…) “(…) a massive database (typically housed on a cluster of servers, or a mini or mainframe computer) serving as a centralized repository of all data generated by all departments and units of a large organization. Advanced data mining software is required to extract meaningful information from a data warehouse”. (see weblink at http://www.businessdictionary.com/definition/data-warehouse.html)

http://www.businessdictionary.com/definition/data-warehouse.html)

http://www.businessdictionary.com/definition/data-warehouse.html)

15


how the implementation of a sophisticated data analytics

approach requires the establishment of a socio-technical

data infrastructure, composed by experts and data

scientists, as well as by modern hardware and software

tools; this of course needs substantial investment. The

huge impact of fixed costs for these tasks also explains

why some big corporations (like Pearson) and important

institutions (like OECD) are developing products and

services that can be used by institutions and individual

stakeholders. In the spirit of learning/academic analytics,

such investments should be sustained by the

institutional/central level, somehow taking advantage of

scale effects. In the next phases of development of

academic analytics, therefore, adopters should also be

aware of the necessity to present evidence of cost/benefits

for these investments. That is to say that the results

favoured by analytics (for instance: higher graduation

rates, higher grades, more satisfied students, etc.) must be

big enough to justify the amount of money currently

invested in creating the data infrastructure and expertise

behind the work of analyses. Nevertheless, these issues are

completely compatible with requirements for open access

code. Not only is open access code publication ethical,

especially when the research and development is paid for

through taxpayer funds; further development is stimulated

through the availability of the code globally. This allowed

the development communities to work to standardize and

share processes, replicate and extend innovations in

multiple contexts, and work collaboratively to increase the

usefulness and accuracy of data analytic tools.

Fourth and finally, it is necessary to verify whether the

various institutions that are interested in developing

sophisticated analyses actually have the necessary

competences, in terms of technical (analytical) skills of the

personnel or, at least, they create conditions for developing

them. As indicated by Arnold et al. (2014), such skills deal

with some major areas of competencies/abilities: data

expertise (understanding how data are collected, stored and

can be questioned and discussed), analytics expertise (to

develop and validate statistical and operational modelling),

evaluation competences (to assess how analytics is used,

and which effects it produces) and teachers/learners’

support, with the aim of transferring the knowledge

created using data to people engaged in the day-by-day

educational practice. Recently Bowers (2017) has detailed

specifics for training education research professionals in

these skills and competencies within higher education

schools of education. Among these capacities, a special

place is reserved for the ability of communicating the

results of the analyses in an effective manner both to upper

management and administration as well as broadly to

system stakeholders and participants, which is a

determinant for being sure that outputs of analyses are

used by decision-makers (see section §2). This is also

important for building consensus in the educational

community about the utility of diffusing analytics practices

and tools. In this perspective, the methodological

challenge is difficult: presenting the results without

excessive simplification (providing an awareness of the

complexities of the learning process) but with enough

clarity to make the information understandable, and thus

usable.

5. A way forward: systemic change and the role of

educational data scientists

The objective of introducing a more systematic and

substantial way to the daily use of quantitative information

in school and university activities requires a decisive shift

in the paradigm of operations that should be accompanied

by the introduction of new professional figures. Among

them, a key actor that we have in mind is the educational

data scientist, whose work is to facilitate the

communication between three worlds: (i) one of technical

experts in data analyses and analytics, (ii) that of decision-

makers at various levels (policy analysts, school

principals, institutions’ managers) and (iii) the community

of teachers, engaged in frontline instruction. The work of

the educational data scientist would be extremely delicate,

because it necessitates simultaneously the technical

capacity for sophisticated data analysis and the sensibility

for the specificities of the educational field –typically

qualitative. Mirroring the emerging definitions of data

science (Schutt & O'Neil, 2013), especially as applied to

education (Bowers 2017; Bowers, et al., 2016), this work

involves the data analytics process noted above in which

data are first processed, which also then leads to

descriptive statistics, but then current data analytic

analyses are applied in an effort to understand and

articulate significant and useful previously unidentified

patterns in the data as the analyst works to communicate

this new information to management.

Practically, such an approach implies a different set of

skills to be developed for prospective teachers and school

heads. Recently, Bowers (2017), in writing about the role

of quantitative methods training for school and school

system leaders, articulated four different interconnected

roles for school system data leadership. First is the

practicing administrator, who requires training in how to

use data and analytics to solve problems and inform

evidence-based improvement cycles. Second, is the

16


quantitative analyst, who in a school system focuses on

efficient service management analytics, such as course

scheduling, budgets, student transportation and cost-

benefit analysis for new and current initiatives. Third, is

the research specialist, who focuses on assessment

construction and validation, surveys and program

evaluation with an eye towards psychometrics, testing and

inferential statistics. And fourth is the district data

scientist, who integrates education data mining, LA,

technology and instruction as well as design-based

research (Coburn et al., 2013) into school system decision

making cycles. Currently, there is a lack of training and

capacity building in all four roles, as programs such as

Harvard University’s recent Strategic Data Project

(Hallgren et al., 2013) are relatively new. Complete

formative experiences of this kind are virtually absent in

Europe, although some universities are starting to work on

the topic, more on the research and institutional side – see

for instance, the investments made by the University of

Edinburgh for using LA12, and/or the project SHEILA

(Supporting Higher Education Institutes with Learning

Analytics)13 also developed at that University. All in all,

training and developing this new expertise requires an

adequate level of commitment and resources; this scope

should suggest a way to prioritize funding allocations in

the future.

To conclude, our opinion is that the development of the

new figure of data scientists should be part of a wider

effort to value the potential of data-driven decision making

in education (see, in the same direction, Cope & Kalantzis,

12 As indicated in the university’s website

(http://www.ed.ac.uk/information-services/learning-

technology/learning-analytics): “The University of Edinburgh

has a wide range of activities in the field of learning analytics.

As shown in the diagram below, these activities cross many

disciplinary, organisational, practice, and research boundaries.

Led by the Vice-Principal Digital Education, Centre for

Research in Digital Education, School of Informatics,

Information Services, Student Systems, and the Institute for

Academic Development, activities in learning analytics include

University leaders, researchers, and practitioners from support,

research, and academic units of the University collaborating on

a variety of projects funded through both internal and external

sources.” 13 From the Project’s website

(http://www.de.ed.ac.uk/project/supporting-higher-education-

integrate-learning-analytics-sheila): “SHEILA aims to develop a

framework that will guide policy development for learning

analytics adoption in higher education in Europe at the levels of

ministries of higher education, quality assurance bodies, and

institutions. It is in close cooperation with the Society of

Learning Analytics and the FP7 project LACE”.

2016). As predicted by MacFayden et al. (2014), the

implementation and effective use of analytics at all levels

of the organization requires a structural change of culture,

as well as a systemic set of modifications in the way in

which the various activities are conducted and assessed. In

this perspective, there is not a one-size-fits-all solution,

and each organization should find the set of data and

processes that best fits its objectives and features. In this

sense, it is also useless to conduct single steps of change,

while a more coordinated group of actions (from data

collection, storing, analyses and use for planning and

feedback) seems more adequate for institutional purposes.

Recommended Citation: Agasisti, T., Bowers, A.J. (2017) Data Analytics and

Decision-Making in Education: Towards the Educational

Data Scientist as a Key Actor in Schools and Higher

Education Institutions. In Johnes, G., Johnes, J., Agasisti,

T., López-Torres, L. (Eds.) Handbook of Contemporary

Education Economics (p.184-210). Cheltenham, UK:

Edward Elgar Publishing. ISBN: 978-1-78536-906-3

http://www.e-elgar.com/shop/handbook-of-contemporary-

education-economics

REFERENCES: Agasisti T., de Witte K. & Rogge N. (forthcoming).

‘Big data and its use in the measurement of public

organizations’ performance and efficiency: state-of-the-art

and perspectives’. Public Policy and Administration.

Agasisti, T. (2013). ‘The efficiency of Italian secondary

schools and the potential role of competition: a data

envelopment analysis using OECD-PISA2006 data’.

Education Economics, 21(5), 520-544.

Andres, J. M. L., Baker, R. S., Siemens, G., Gašević,

D., & Spann, C. A. (in press). Replicating 21 Findings on

Student Success in Online Learning. Technology,

Instruction, Cognition, and Learning.

Arnold, K. E. (2010). ‘Signals: Applying Academic

Analytics’. EDUCAUSE Quarterly, 33(1), n1.

Arnold, K. E., & Pistilli, M. D. (2012). ‘Course signals

at Purdue: using learning analytics to increase student

success’. In Proceedings of the 2nd international

conference on learning analytics and knowledge (pp. 267-

270). ACM.

Arnold, K. E., Lynch, G., Huston, D., Wong, L., Jorn,

L., & Olsen, C. W. (2014). ‘Building institutional

capacities and competencies for systemic learning

analytics initiatives’. In Proceedings of the Fourth

International Conference on Learning Analytics and

Knowledge (pp. 257-260). ACM.

http://www.de.ed.ac.uk/project/supporting-higher-education-integrate-learning-analytics-sheila)

http://www.de.ed.ac.uk/project/supporting-higher-education-integrate-learning-analytics-sheila)



17


Bach, C. (2010). ‘Learning analytics: Targeting

instruction, curricula and student support’. Office of the

Provost, Drexel University.

Baepler, P., & Murdoch, C. J. (2010). ‘Academic

analytics and data mining in higher education’.

International Journal for the Scholarship of Teaching and

Learning, 4(2), 17.

Baker, R. S. (2013). ‘Learning, Schooling, and Data

Analytics’. In M. Murphy, S. Redding & J. Twyman

(Eds.), Handbook on innovations in learning Philadelphia,

PA: Center on Innovations in Learning, Temple

University; Charlotte, NC: Information Age Publishing.

Baker, R. S., & Inventado, P. S. (2014). ‘Educational

data mining and learning analytics’. In J. A. Larusson & B.

White (Eds.), Learning Analytics (pp. 61-75). New York:

Springer.

Baker, R. S., & Yacef, K. (2009). ‘The State of

Educational Data Mining in 2009: A Review and Future

Visions’. Journal of Educational Data Mining, 1(1), 3-16.

Balfanz, R., Herzog, L., & MacIver, D. J. (2007).

‘Preventing student disengagement and keeping students

on the graduation path in urban middle-grades schools:

Early identification and effective interventions’.

Educational Psychologist, 42(4), 223-235.

Bambrick-Santoyo, P. (2010). Driven by Data: A

Practical Guide to Improve Instruction. San Francisco,

CA: Jossey-Bass.

Bedrick, J., Butcher, J., & Bolick, C. (2016). ‘Taking

Credit for Education: How to Fund Education Savings

Accounts through Tax Credits’, Cato Institute Policy

Analysis, #785.

Behrens, J. T., & DiCerbo, K. E. (2014). Technological

Implications for Assessment Ecosystems: Opportunities

for Digital Technology to Advance Assessment. Teachers

College Record, 116(11), 1-22.

Bernhardt, V. (2013). Data analysis for continuous

school improvement (3 ed.). New York: Routledge.

Bernhardt, V. L. (2004). Data Analysis for Continuous

School Improvement (2 ed.). Eye On Education.

Bieber, T., & Martens, K. (2011). ‘The OECD PISA

study as a soft power in education? Lessons from

Switzerland and the US’. European Journal of Education,

46(1), 101-116.

Bienkowski, M., Feng, M., & Means, B. (2012).

‘Enhancing Teaching and Learning Through Educational

Data Mining and Learning Analytics: An Issue Brief’.

Washington, DC: U.S. Department of Education, Office of

Educational Technology.

Bishop, J. H. (1997). ‘The effect of national standards

and curriculum-based exams on achievement’. The

American Economic Review, 87(2), 260-264.

Borer, V. L., & Lawn, M. (2013). ‘Governing

Education Systems by Shaping Data: from the past to the

present, from national to international perspectives’.

European Educational Research Journal, 12(1), 48-52.

Boudett, K. P., City, E. A., & Murnane, R. J. (2013).

Data Wise: Revised and Expanded Edition: A Step-by-Step

Guide to Using Assessment Results to Improve Teaching

and Learning. Revised and Expanded Edition. Cambridge,

MA: Harvard Education Press.

Bouwma-Gearhart, J., & Collins, J. (2015, October).

‘What We Know About Data-Driven Decision Making In

Higher Education: Informing Educational Policy and

Practice’. In Proceedings of International Academic

Conferences (No. 2805154). International Institute of

Social and Economic Sciences.

Bowers, A. J. (2008). ‘Promoting Excellence: Good to

great, NYC's district 2, and the case of a high performing

school district’. Leadership and Policy in Schools, 7(2),

154-177.

Bowers, A. J. (2009). ‘Reconsidering grades as data for

decision making: More than just academic knowledge’.

Journal of Educational Administration, 47(5), 609-629.

Bowers, A. J. (2017). ‘Quantitative Research Methods

Training in Education Leadership and Administration

Preparation Programs as Disciplined Inquiry for Building

School Improvement Capacity. Journal of Research on

Leadership Education, 12(1), p.72-96.

http://doi.org/10.1177/1942775116659462.

Bowers, A. J., Krumm, A. E., Feng, M., & Podkul, T.

(2016). ‘Building a Data Analytics Partnership to Inform

School Leadership Evidence-Based Improvement Cycles’.

Paper presented at the Annual meeting of the American

Educational Research Association, Washington, DC.

Bowers, A. J., Shoho, A. R., & Barnett, B. G. (2014).

‘Considering the Use of Data by School Leaders for

Decision Making’. In A. J. Bowers, A. R. Shoho & B. G.

Barnett (Eds.), Using Data in Schools to Inform

Leadership and Decision Making (pp. 1-16). Charlotte,

NC: Information Age Publishing.

Bowers, A. J., Sprott, R., & Taff, S. (2013). ‘Do we

know who will drop out? A review of the predictors of

dropping out of high school: Precision, sensitivity and

specificity’. The High School Journal, 96(2), 77-100.

http://doi.org/10.1177/1942775116659462

18


Braun, M. L., & Ong, C. S. (2014). ‘Open Science in

Machine Learning’. In V. Stodde, F. Leisch & R. D. Peng

(Eds.), Implementing Reproducible Research (pp. 343-

365): Chapman and Hall/CRC.

Brunello, G., & Checchi, D. (2007). ‘Does school

tracking affect equality of opportunity? New international

evidence’. Economic Policy, 22(52), 782-861.

Campbell, J. P., & Oblinger, D. G. (2007). ‘Academic

analytics’. EDUCAUSE Review, 42(4), 40-57.

Chen, H., Chiang, R. H., & Storey, V. C. (2012).

‘Business Intelligence and Analytics: From Big Data to

Big Impact’. MIS Quarterly, 36(4), 1165-1188.

Cho, V., & Wayman, J. C. (2015). ‘Districts’ Efforts

for Data Use and Computer Data Systems: The Role of

Sensemaking in System Use and Implementation’.

Teachers College Record, 116(2), 1-45.

Coburn, C. E., & Turner, E. O. (2011). ‘Research on

Data Use: A Framework and Analysis’. Measurement:

Interdisciplinary Research and Perspectives, 9(4), 173-

206.

Coburn, C. E., Penuel, W. R., & Geil, K. E. (2013).

Research-Practice Partnerships: A Strategy for

Leveraging Research for Educational Improvement in

School Districts. New York, NY: William T. Grant

Foundation.

Coleman, J. S. (1966). Equality of educational

opportunity. Washington, DC: U.S. Department of Health,

Education, and Welfare, Office of Education.

Cope, B., & Kalantzis, M. (2016). ‘Big Data Comes to

School’. AERA Open, 2(2), 1-19.

Cosner, S. (2014). ‘Strengthening Collaborative

Practices in Schools: The Need to Cultivate Development

Perspectives and Diagnostic Approaches’. In A. J. Bowers,

A. R. Shoho & B. G. Barnett (Eds.), Using Data in

Schools to Inform Leadership and Decision Making.

Charlotte, NC: Information Age Publishing.

Chung, G. K. W. K. (2014). Toward the Relational

Management of Educational Measurement Data. Teachers


Daniel, B. (2015). ‘Big Data and analytics in higher

education: Opportunities and challenges’. British Journal

of Educational Technology, 46(5), 904-920.

Datnow, A., & Hubbard, L. (2015). ‘Teachers’ Use of

Assessment Data to Inform Instruction: Lessons From the

Past and Prospects for the Future’. Teachers College

Record, 117(4), 1-26.

Dawson, S., Gašević, D., Siemens, G., & Joksimovic,

S. (2014). ‘Current state and future trends: a citation

network analysis of the learning analytics field’. Paper

presented at the Proceedings of the Fourth International

Conference on Learning Analytics And Knowledge,

Indianapolis, Indiana, USA.

Davies, P. (1999). ‘What is evidence‐based

education?’, British Journal of Educational Studies, 47(2),

108-121.

De Witte, K., Cabus, S., Thyssen, G., Groot, W., & van

den Brink, H. M. (2013). ‘A critical review of the literature

on school dropout’. Educational Research Review, 10, 13-

28.

Demie, F. (2013). Using Data to Raise Achievement

Good Practice in Schools. Lambeth Research and Statistics

Unit.

Demski, D., & Racherbäumer, K. (2015). ‘Principals'

Evidence-Based Practice--Findings from German

Schools’. International Journal of Educational

Management, 29(6), 735-748.

Denley, T. (2012). ‘Austin Peay State University:

Degree Compass’. In D.G. Oblinger (ed.), Gamer

changers: education and information technologies (pp.

263-267), EDUCAUSE.

Denley, T. (2013). ‘Degree Compass: A Course

Recommendation System’. EDUCAUSE Review Online.

http://er.educause.edu/articles/2013/9/degree-compass-a-

course-recommendation-system

Denley, T. (2014). ‘How predictive analytics and

choice architecture can improve student success’.

Research & Practice in Assessment, 9.

Dynarski, M. (2016). ‘On negative effects of vouchers’.

Brookings Institution. Evidence Speaks Reports, Vol 1,

#18.

Earl, L. M., & Katz, S. (Eds.). (2006). Leading schools

in a data-rich world: Harnessing data for school

improvement. Corwin Press.

European Commission. (2016). Communication from

the Commission to the Council and to the EU Parliament,

8/9/2006, on efficiency and equity in European education

and training systems, COM(2006)481 final.

Farley-Ripple, E. N., & Buttram, J. L. (2014).

‘Schools’ Use of Interim Data: Practices in Classrooms,

Teams, and Schools’. In A. J. Bowers, A. R. Shoho & B.

G. Barnett (Eds.), Using Data in Schools to Inform

Leadership and Decision Making (pp. 39-66). Charlotte,

NC: Information Age Publishing.

http://er.educause.edu/articles/2013/9/degree-compass-a-course-recommendation-system

http://er.educause.edu/articles/2013/9/degree-compass-a-course-recommendation-system

19


Farley-Ripple, E. N., & Buttram, J. L. (2015). ‘The

Development of Capacity for Data Use: The Role of

Teacher Networks in an Elementary School’. Teachers


Fecher, B., & Friesike, S. (2014). ‘Open Science: One

Term, Five Schools of Thought’. In S. Bartling & S.

Friesike (Eds.), Opening Science: The Evolving Guide on

How the Internet is Changing Research, Collaboration

and Scholarly Publishing (pp. 17-47). Cham: Springer

International Publishing.

Feng, M., Krumm, A. E., Bowers, A. J., & Podkul, T.

(2016). ‘Elaborating data intensive research methods

through researcher-practitioner partnerships’. Paper

presented at the Proceedings of the Sixth International

Conference on Learning Analytics & Knowledge,

Edinburgh, United Kingdom.

Ferguson, R. (2012). ‘Learning analytics: drivers,

developments and challenges’. International Journal of

Technology Enhanced Learning, 4(5-6), 304-317.

Forster, G. (2009). ‘A Win-Win Solution: The

Empirical Evidence on How Vouchers Affect Public

Schools. School Choice Issues in Depth’. Friedman

Foundation for Educational Choice.

Fritz, J. (2010). ‘Video Demo of UMBC's" Check My

Activity" Tool for Students’. Educause Quarterly, 33(4).

Fritz, J. (2011). ‘Classroom walls that talk: Using

online course activity data of successful students to raise

self-awareness of underperforming peers’. The Internet

and Higher Education, 14(2), 89-97.

Gandomi, A., & Haider, M. (2015). ‘Beyond the hype:

Big data concepts, methods, and analytics’. International

Journal of Information Management, 35(2), 137-144.

Gašević, D., Dawson, S., & Siemens, G. (2015). ‘Let’s

not forget: Learning analytics are about learning’.

TechTrends, 59(1), 64-71.

Goldstein, P. J., & Katz, R. N. (2005). ‘Academic

analytics: The uses of management information and

technology in higher education’, EDUcause.edu.

Grek, S. (2009). ‘Governing by numbers: The PISA

‘effect’ in Europe’. Journal of Education Policy, 24(1),

23-37.

Grek, S., & Ozga, J. (2009). ‘Governing education

through data: Scotland, England and the European

education policy space’. British Educational Research

Journal, 36(6), 937-952.

Gummer, E. S., & Mandinach, E. (2015). ‘Building a

Conceptual Framework for Data Literacy’. Teachers


Hallgren, K., Pickens Jewell, C., Kamler, C., Hartog, J.,

& Gothro, A. (2013). Strategic data project and education

pioneers year 1 report: Laying the groundwork for data-

driven decision making. Princenton, NJ: Mathematica

Policy Research.

Halverson, R. (2010). ‘School formative feedback

systems’. Peabody Journal of Education, 85(2), 130-146.

Hanushek, E. A. (1995). ‘Education production

functions’. In International Encyclopedia of Economics of

Education, 277-282.

Hanushek, E. A., & Woessmann, L. (2010). ‘The

economics of international differences in educational

achievement’ (No. w15949). National Bureau of Economic

Research.

Holt, A., & Schools, T. P. (2015). Early Elementary

On-Track Indicators Leading to Third-Grade Reading

Proficiency. SDP Capstone Report.

Honig, M. I., & Coburn, C. (2007). ‘Evidence-based

decision making in school district central offices: Toward

a policy and research agenda’. Educational Policy.

Horn, M. B. (2015). ‘The Rise of AltSchool and Other

Micro-schools’. Education Next, 15(3).

IEA (2012). ‘TIMSS 2011 International Results in

Mathematics – Executive Summary’, Chestnut Hill, MA:

TIMSS & PIRLS International Study Center, Boston

College.

Ifenthaler, D., & Tracey, M. W. (2016). Exploring the

relationship of ethics and privacy in learning analytics and

design: implications for the field of educational

technology. Educational Technology Research and

Development, 64(5), 877-880.

Ingle, W. K., & Cramer, T. (2013). ‘Not Just

Accountability: A Cost-Effectiveness Analysis of Third-

Grade Reading Diagnostic Tools’. In B. G. Barnett, A. R.

Shoho & A. J. Bowers (Eds.), School and district

leadership in an era of accountability (pp. 143-171).

Charlotte, NC: Information Age Publishing Inc.

Jacobs, J., Gregory, A., Hoppey, D., & Yendol-

Hoppey, D. (2009). ‘Data Literacy: Understanding

Teachers' Data Use in a Context of Accountability and

Response to Intervention’. Action in Teacher Education,

31(3), 41-55.

20


Jimerson, J. B., & Childs, J. (in press). Signal and

Symbol: How State and Local Policies Address Data-

Informed Practice. Educational Policy.

Jimerson, J. B., & Wayman, J. C. (2015). ‘Professional

Learning for Using Data: Examining Teacher Needs and

Supports’. Teachers College Record, 117(4), 1-36.

Jürges, H., Schneider, K., & Büchel, F. (2005). ‘The

effect of central exit examinations on student achievement:

Quasi-experimental evidence from TIMSS Germany’.

Journal of the European Economic Association, 3(5),

1134-1155.

Kickmeier-Rust, M. D., & Albert, D. (2013). ‘Learning

analytics to support the use of virtual worlds in the

classroom’. In Human-Computer Interaction and

Knowledge Discovery in Complex, Unstructured, Big Data

(pp. 358-365). Springer Berlin Heidelberg.

Kirkup, C., Sizmur, J., Sturman, L., & Lewis, K.

(2005). ‘Schools' Use of Data in Teaching and Learning’.

Research Report RR671. National Foundation for

Educational Research.

Knowles, J. E. (2015). ‘Of Needles and Haystacks:

Building an Accurate Statewide Dropout Early Warning

System in Wisconsin’. Journal of Educational Data

Mining, 7(3), 18-67.

Koedinger, K. R., Baker, R. S., Cunningham, K.,

Skogsholm, A., Leber, B., & Stamper, J. (2010). ‘A data

repository for the EDM community: The PSLC DataShop’.

In C. Romero, S. Ventura, M. Penchenizkiy & R. S. Baker

(Eds.), Handbook of Educational Data Mining (pp. 43-55).

Koedinger, K. R., D'Mello, S., McLaughlin, E. A.,

Pardos, Z. A., & Rosé, C. P. (2015). ‘Data mining and

education’. Wiley Interdisciplinary Reviews: Cognitive

Science, 6(4), 333-353.

Koon, S., & Petscher, Y. (2015). ‘Comparing

Methodologies for Developing an Early Warning System:

Classification and Regression Tree Model versus Logistic

Regression’. REL 2015-077. Regional Educational

Laboratory Southeast.

Laurillard, D. (2008). ‘The teacher as action researcher:

Using technology to capture pedagogic form’. Studies in

Higher Education, 33(2), 139-154.

Lawson, C., Beer, C., Rossi, D., Moore, T., & Fleming,

J. (2016). Identification of ‘at risk’ students using learning

analytics: the ethical dilemmas of intervention strategies in

a higher education institution. Educational Technology

Research and Development, 64(5), 957-968.

Leithwood, K. (2013). ‘Concluding synthesis and

commentary’. In B. G. Barnett, A. R. Shoho & A. J.

Bowers (Eds.), School and district leadership in an era of

accountability (pp. 255-269). Charlotte, NC: Information

Age Publishing Inc.

Levin, H. M., Belfield, C. R., Hollands, F., Bowden, A.

B., Cheng, H., Shand, R., Hanisch-Cerda, B. (2012).

‘Cost-effectiveness analysis of interventions that improve

high school completion’. New York, NY: Center for

Benefit-Cost Studies of Education, Teachers College,

Columbia University.

Lovenheim, M. F., & Walsh, P. (2014). ‘Does Choice

Increase Information? Evidence from Online School

Search Behavior’, paper presented at the 2016 AEFP

Conference.

Mac Iver, M. A. (2013). ‘Early Warning Indicators of

High School Outcomes’. Journal of Education for Students

Placed at Risk (JESPAR), 18(1), 1-6.

Macfadyen, L. P., & Dawson, S. (2012). ‘Numbers Are

Not Enough. Why e-Learning Analytics Failed to Inform

an Institutional Strategic Plan’. Educational Technology &

Society, 15(3), 149-163.

Macfadyen, L. P., Dawson, S., Pardo, A., & Gasevic,

D. (2014). ‘Embracing big data in complex educational

systems: The learning analytics imperative and the policy

challenge’. Research & Practice in Assessment, 9.

Mandinach, E. B., & Gummer, E. S. (2013). ‘A

Systemic View of Implementing Data Literacy in Educator

Preparation’. Educational Researcher, 42(1), 30-37.

Mandinach, E. B., Honey, M., & Light, D. (2006). ‘A

Theoretical Framework for Data-Driven Decision

Making’, Paper presented at the American Educational

Research Association, San Francisco, CA

Mandinach, E. B., Honey, M., Light, D., & Brunner, C.

(2008). ‘A Conceptual Framework for Data-Driven

Decision Making’. In E. B. Mandinach & M. Honey

(Eds.), Data-Driven School Improvement: Linking Data

and Learning (pp. 13-31). New York: Teachers College

Press.

Marsh, J. A. (2012). ‘Interventions Promoting

Educators’ Use of Data: Research Insights and Gaps’.

Teachers College Record, 114(11), 1-48.

Marsh, J. A., & Farrell, C. C. (2015). How leaders can

support teachers with data-driven decision making: A

framework for understanding capacity building.

Educational Management Administration & Leadership,

43(2), 269-289.

21


Marsh, J. A., Pane, J. F., & Hamilton, L. S. (2006).

‘Making sense of data-driven decision making in

education’, RAND Corporation Occasional Paper.

Masum, H., Rao, A., Good, B. M., Todd, M. H.,

Edwards, A. M., Chan, L., Bourne, P. E. (2013). ‘Ten

Simple Rules for Cultivating Open Science and

Collaborative R&D’. PLos Computational Biology, 9(9).

Means, B., Padilla, C., & Gallagher, L. (2010). ‘Use of

Education Data at the Local Level From Accountability to

Instructional Improvement’. Washington, DC: U.S.

Department of Education, Office of Planning, Evaluation

and Policy Development.

Meyer, H. D., & Benavot, A. (Eds.). (2013). PISA,

power, and policy: The emergence of global educational

governance. Symposium Books Ltd.

Mockler, N. (2013). ‘Reporting the ‘education

revolution’: MySchool. edu. au in the print media’.

Discourse: Studies in the Cultural Politics of Education,

34(1), 1-16.

Molloy, J. C. (2011). ‘The Open Knowledge

Foundation: Open Data Means Better Science’. PLos

Biology, 9(12).

Mosteller, F., & Boruch, R. F. (Eds.). (2002). Evidence

matters: Randomized trials in education research.

Brookings Institution Press.

Murnane, R. J. (2013). ‘US high school graduation

rates: Patterns and explanations’ (No. w18701). National

Bureau of Economic Research.

Murray, J. (2013). ‘Critical issues facing school leaders

concerning data-informed decision-making’. School

Leadership & Management, 33(2), 169-177.

Ni, X., Bowers, A. J., & Essewein, J. (2016). ‘What

Counts in Calculating School and District Level

Performance Index Scores: A Summary and Analysis of

Academic Performance Index Metrics across the 50

States’. New York: Teachers College, Columbia

University.

Pardo, A., & Siemens, G. (2014). ‘Ethical and privacy

principles for learning analytics’. British Journal of

Educational Technology, 45(3), 438-450.

Piety, P. J. (2013). Assessing the educational data

movement. New York, NY: Teachers College Press.

Popham, W. J. (2010). Everything school leaders need

to know about assessment. Thousand Oaks, CA: Corwin.

Raghupathi, W., & Raghupathi, V. (2014). ‘Big data

analytics in healthcare: promise and potential’. Health

Information Science and Systems, 2(1), 3.

Reeves, P. L., & Burt, W. L. (2006). ‘Challenges in

Data-Based Decision-Making: Voices from Principals’.

Educational Horizons, 85(1), 65-71.

Riehl, C. (1999). ‘Labeling and letting go: An

organizational analysis of how high school students are

discharged as dropouts’. In A. M. Pallas (Ed.), Research in

sociology of education and socialization (pp. 231-268).

New York: JAI Press.

Romero, C., & Ventura, S. (2007). ‘Educational data

mining: A survey from 1995 to 2005’. Expert systems with

applications, 33(1), 135-146.

Romero, C., & Ventura, S. (2010). ‘Educational data

mining: a review of the state of the art’. Systems, Man, and

Cybernetics, Part C: Applications and Reviews, IEEE

Transactions on, 40(6), 601-618.

Rumberger, R. W., & Palardy, G. J. (2005). ‘Test

scores, dropout rates, and transfer rates as alternative

indicators of high school performance’. American

Educational Research Journal, 42(1), 3-42.

Rutledge, S., & Gale Neal, B. (2013). ‘"The numbers

speak for themselves": Data use and organization of

schooling in two Florida elementary schools’ in The

infrastructure of accountability: Data use and the

transformation of American education (pp. 113-128).

Cambridge, Mass: Harvard Education Press.

Scheuer, O., & McLaren, B. M. (2012). ‘Educational

data mining’. In Encyclopedia of the Sciences of Learning

(pp. 1075-1079). Springer US.

Schildkamp, K., Karbautzki, L., & Vanhoof, J. (2014).

Exploring data use practices around Europe: Identifying

enablers and barriers. Studies in Educational Evaluation,

42, 15-24.

Schildkamp, K., Poortman, C. L., & Handelzalts, A.

(2016). ‘Data teams for school improvement’. School

Effectiveness and School Improvement, 27(2), 228-254.

Schildkamp, K., Poortman, C., Luyten, H., & Ebbeler,

J. (in press). Factors promoting and hindering data-based

decision making in schools. School Effectiveness and

School Improvement, 1-17.

Schleicher, A. (2016). ‘Going beyond education

policies – how can PISA help turn policy into practice?’

Education & Skills Today, OECD.

Schneider, B., Carnoy, M., Kilpatrick, J., Schmidt, W.

H., & Shavelson, R. J. (2007). ‘Estimating casual effects:

22


Using experimental and observational designs’ (report

from the Governing Board of the American Educational

Research Association Grants Program). Washington, DC:

American Educational Research Association.

Schmidt, W. H., McKnight, C. C., Houang, R. T.,

Wang, H., Wiley, D. E., Cogan, L. S., & Wolfe, R. G.

(2001). Why Schools Matter: A Cross-National

Comparison of Curriculum and Learning. The Jossey-Bass

Education Series. Jossey-Bass, 989 Market Street, San

Francisco, CA 94103-1741.

Scholes, V. (2016). The ethics of using learning

analytics to categorize students on risk. Educational

Technology Research and Development, 64(5), 939-955.

Schutt, R., & O'Neil, C. (2013). Doing Data Science:

Straight Talk from the Frontline. Cambridge, MA:

O'Reilly.

Siemens, G. (2010). ‘What are learning analytics’.

ELEARNSPACE: Learning, networks, knowledge,

technology, community.

Siemens, G. (2013). ‘Learning Analytics: the

emergence of a Discipline’. American Behavioral

Scientist, 57(10), 1380-1400.

Siemens, G., & Long, P. (2011). ‘Penetrating the Fog:

Analytics in Learning and Education’. EDUCAUSE

review, 46(5), 30.

Slade, S., & Prinsloo, P. (2013). ‘Learning analytics

ethical issues and dilemmas’. American Behavioral

Scientist, 57(10), 1510-1529.

Slavin, R. E. (2002). ‘Evidence-based education

policies: Transforming educational practice and research’.

Educational Researcher, 31(7), 15-21.

Stodden, V., Guo, P., & Ma, Z. (2013). Toward

Reproducible Computational Research: An Empirical

Analysis of Data and Code Policy Adoption by Journals.

PloS one, 8(6), e67111.

Spector, J. M. (2016). Ethics in educational technology:

towards a framework for ethical decision making in and

for the discipline. Educational Technology Research and

Development, 64(5), 1003-1011

Strasser, C. (2015). Research Data Management.

Baltimore, MD: National Information Standards

Organization.

Strietholt, R., Bos, W., Gustafsson, J. E., & Rosén, M.

(Eds.). (2014). Educational Policy Evaluation through

International Comparative Assessments. Waxmann

Verlag.

Tibshirani, R. (1996). ‘Regression shrinkage and

selection via the lasso’. Journal of the Royal Statistical

Society. Series B (Methodological), 267-288.

Turner, E. O., & Coburn, C. E. (2012). ‘Interventions to

promote data use: An introduction’. Teachers College

Record, 114(11), 1-13.

Verbert, K., Duval, E., Klerkx, J., Govaerts, S., &

Santos, J. L. (2013). ‘Learning analytics dashboard

applications’. American Behavioral Scientist, 57(10),

1500-1509.

Wayman, J. C. (2005). ‘Involving teachers in data-

driven decision making: Using computer data systems to

support teacher inquiry and reflection’. Journal of

Education for Students Placed at Risk, 10(3), 295-308.

Wayman, J. C., Stringfield, S., & Yakimowski, M.

(2004). ‘Software enabling school improvement through

the analysis of student data’ (Report No. 67). Baltimore,

MD: Johns Hopkins University, Center for Researc on the

Education of Students Placed At Risk.

Wayman, J. C., & Stringfield, S. (2006). ‘Data use for

school improvement: School practices and research

perspectives’. American Journal of Education, 112(4),

463-468.

Wayman, J. C., Cho, V., & Johnston, M. T. (2007).

‘The data-informed district: A district-wide evaluation of

data use in the Natrona county school district’. Austin: The

University of Texas at Austin.

Wayman, J. C., Cho, V., Jimerson, J. B., & Snodgrass

Rangel, V. W. (2015). ‘A Look into the Workings of Data

Use in a Mid-Sized District’. In I. E. Sutherland, K. L.

Sanzo & J. P. Scribner (Eds.), Leading Small and Mid-

Sized Urban School Districts (Vol. 22, pp. 241-275):

Emerald.

Wayman, J. C., Cho, V., Jimerson, J. B., & Spikes, D.

D. (2012). ‘District-wide effects on data use in the

classroom’. Education Policy Analysis Archives, 20(25), 1-

28.

Whitty, G. (2006). ‘Education(al) research and

education policy making: is conflict inevitable?’. British

Educational Research Journal, 32(2), 159-176.

Williamson, B. (2016). ‘Digital education governance:

data visualization, predictive analytics, and ‘real-

time’policy instruments’. Journal of Education Policy,

31(2), 123-141.

Willis III, J. E., Campbell, J., & Pistilli, M. (2013).

‘Ethics, Big Data, and Analytics: A Model for

Application’, EDUCAUSE Review Online.

23


Willis, J. E., Slade, S., & Prinsloo, P. (2016). Ethical

oversight of student data in learning analytics: a typology

derived from a cross-continental, cross-institutional

perspective. Educational Technology Research and

Development, 64(5), 881-901.

Woessmann, L. (2008). ‘Efficiency and equity of

European education and training policies’. International

Tax and Public Finance, 15(2), 199-230.

Woessmann, L., & West, M. (2006). ‘Class-size effects

in school systems around the world: Evidence from

between-grade variation in TIMSS’. European Economic

Review, 50(3), 695-736.

Yang, M., Goldstein, H., Rath, T., & Hill, N. (1999).

‘The use of assessment data for school improvement

purposes’. Oxford Review of Education, 25(4), 469-483.

Yiu, C. (2012). ‘The big data opportunity: Making

government faster, smarter and more personal’. Policy

Exchange.

Yoon, S. Y. (in press). Principals’ Data-Driven Practice

and Its Influences on Teacher Buy-in and Student

Achievement in Comprehensive School Reform Models.

Leadership and Policy in Schools, 1-24.

data analytics and decision-making in education: towards the ... · data analytics and...

Documents