do mechanical turks dream of big data?

Lehrstuhl Informatik 5 (Information Systems)

Prof. Dr. M. Jarke 1

Learning Layers

This slide deck is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Do Mechanical Turks Dream of Big Data?

Advanced Community Information Systems (ACIS) RWTH Aachen University, Germany

[email protected]



Learning Layers

Responsive Open

Community Information

Systems

Community Visualization

and Simulation

Community Analytics

Community

Support

Web Analytics W

eb E

ngin

eerin

g

Advanced Community Information Systems (ACIS)

Requirements Engineering



Learning Layers

Abstract With the advent of data collections on a planetary level, also the role of researchers producing, processing and analysing such data sets is debated as heated as in the early days of nuclear research. It seems that the Dr. Strangelove image of scientists has turned into a faceless mass of Mechanical Turks hiding behind agencies and large research networks. So, it is time to peek behind the curtain to disclose the network nature of modern science. A basic ethical obligation is to get enough knowledge to make informed decisions. So, we visit some recent incidents of big data debates in higher education and mass surveillance. In particular, we are questioning the role of computer science as producer of dual use weapons of mass surveillance. Ironically, computer science is not only part of the problem but also part of the solution. We discuss some interesting socio-technical approaches of giving back the power of data transparently into the hands of the owners.



Learning Layers

Agenda

The N

etwor

ked

Scien

tist

Lear

ning A

nalyt

ics

Less

ons L

earn

t

Conc

lusion

s &

Outlo

ok



Learning Layers

THE NETWORKED SCIENTIST



Learning Layers

Iconographic Images of Science



Learning Layers




Learning Layers

Computer Science Knowledge Network



Learning Layers The Knowledge Map of Computer Science

Map of computer science in 2010 [Pham, Klamma & Jarke, SNAM 2010]

HCI

Networks and Communication

Software Engineering

Artificial Intelligence Theory

Database

Computer Graphics

Computer Vision

Security and Privacy

Distributed and Parallel Computing

Machine Learning

Data Mining

Map of computer science in 1990 Map of computer science in 1995

Map of computer science in 2000 Map of computer science in 2005



Learning Layers

Mechanical Turks



Learning Layers

Consequences for Scientometrics ■  The great „iron fence“ has been replaced by many fences

around research communities –  Dr. Strangelove is a faceless community now –  The long tail of research communities –  Many research communities under public pressure (e.g.

environmental sciences - http://www.pangaea.de/) –  It will get worse! (open access/data, public funding cuts)

■  Big Data Research for Understanding Science –  Social Network Analysis, Machine Learning –  Mechanical Turks?

■  Where is the research ethics? –  Menlo Report (2012)



Learning Layers

LEARNING ANALYTICS



Learning Layers

What is Learning Analytics?

"Field associated with deciphering trends and patterns from educational big data, or huge sets of student-related data, to further the advancement of a personal ized, suppor t ive system of h igher education." (2013 Horizon Report)



Learning Layers

Leaking



Learning Layers Recent News about

Lea(rn|k)ing Analytics L ■  BIFIE-Leak - 400.000 confidential tests of pupils and 37.000

E- mail addresses of Austrian teachers have been found on Romanian servers accessible from the Internet (Die Presse)

■  UMD-Leak - 300.000 personal record data were compromised by a hack at the University of Maryland (UMD)

■  FSU-Leak: 47.000 teachers in training data leaked at Florida State University (FSU)

■  Oxford-Leak: University of Oxford Leaks List of Its 50 Worst-Performing Students (The Chronicle of Higher Education)

This list is really endless



Learning Layers TeLLNet - SNA for European

Teachers‘ Life Long Learning ■  How to manage and handle large scale data on

social networks? ■  How to analyse social network data in order to

develop teachers’ competence, e.g. to facilitate a better project collaboration?

■  How to make the network visualization useful for teachers’ lifelong learning?

Song, Petrushyna, Cao, Klamma: Learning Analytics at Large: The Lifelong Learning Network of 160, 000 European Teachers. EC-TEL 2011



Learning Layers Analysis and Visualization of

Lifelong Learner Data



Learning Layers

Ethical Concerns During the Project

■  The eTwinning platform data should be protected as much as possible –  No live access, access only in anonymous dumps –  Better: Privacy preserving technologies

■  Teacher Workshops –  Identification of teachers only with consent

■  Learning Analytics Tools –  Tool was available on the Web –  Data accessible only for teacher in the networks



Learning Layers

Still, Technology is a Powerful Tool



Learning Layers

LESSONS LEARNT



Learning Layers

Datability ■  A clear identification of benefits, risks and harms for

collecting ICT data ■  Ethical guidelines, approval routines and best

practices for data sharing in science and education ■  Transparency and accountability without the loss of

privacy ■  Academic freedom



Learning Layers

What we get ■  Will happen J Big Data by Digital Eco Systems (Quantitative Analysis)

–  A plethora of targets (Small Birds) –  Professional Communities are distributed in a long tail –  Professional Communities use a digital eco system

–  An arsenal of weapons (Big Guns) –  A growing number of community learning analytics methods –  Combined methods from machine intelligence and knowledge representation

■  May not happen L Deep Involvment with community (Qualitative Analysis) –  Domain knowledge for sense making –  Passion for community and sense of belonging –  Community learns as a whole

→ Community Learning Analytics for the Community by the Community



Learning Layers Learning Analytics vs. Community

Learning Analytics Formal Learning Learning Analytics Community

Regulated Learning

Community Learning Analytics

Environment LMS EDM/VA CIS/ROLE DM/VA/SNA/Role Mining

Tools Fixed LMS Specific Eco-System Tool Recommender

Activities Fixed Content Recommender

Dynamic Content Recommender / Expert Recommender

Goals Fixed Progress Dynamic Progess / Goal Mining / Refinement

Communities Fixed Not applicable Dynamic (Overlapping) Community Detection

Use Cases Courses Learning Paths Peer Production / Scaffolding

Semantic Networks of Learners / Annotations



Learning Layers Reflective Open

Community Information Systems

•  Network Models

•  Network Analysis

•  Actor Network Theory

•  Communities of Practice

•  Expert Identification

•  Community Detection

•  Web Mining •  Recommender

Systems •  Multi Agent

Simulation

Web

Ana

lytics

•  Advanced Web & Multimedia Technologies •  XMPP • HTML5 • MPEG-7

• Web Services • REST •  LAS

• Cloud Computing

• Mobile Computing

Web

Eng

ineer

ing

• MediaBase • MobSOS • D-VITA

• Requirements Bazaar • Direwolf • AERCS/CAMRS

• yFiles • Repast • AERCS

• LAS & LAS2peer • youTell • SeViAnno 2.0

Responsive Open

Community Information

Systems

Community Visualization &

Simulation

Community Analytics

Community Support

Requirements Engineering

•  Large-Scale Web-Based Social Requirements Engineering •  Agent and Goal Oriented i* Modeling •  Participatory Community Design