crowdsourcing for educational evaluation in e-learning · 2014-06-23 · crowdsourcing for...

Crowdsourcing for Educational Evaluation ine-Learning

Raj Agrawal

M.Tech ProjectUnder the guidance of

Prof. Deepak B. Phatak

Computer Science & EngineeringIndian Institute of Technology, Bombay

June 23, 2014

Outline Introduction Crowdsourcing in Educational Evaluation Problem Scope Proposed Solution Implementation Details Conclusion References

1 IntroductionCrowdsourcingClassification of CrowdsourcingMotivation

2 Crowdsourcing in Educational EvaluationNeed of Crowdsourcing in Educational EvaluationPeer to Peer Evaluation System

3 Problem Scope

4 Proposed SolutionMeasuring Knowledge Level of the ParticipantsGroup Formation in Peer EvaluationAllocation Strategies of Projects for Evaluation Based on KnowledgeLevelCalculating Final Grade

5 Implementation Details

6 Conclusion

7 References

Outline Crowdsourcing for Educational Evaluation in e-Learning 2

Introduction

Introduction”Crowdsourcing”

• What is Crowdsourcing?

• Distributive Problem solving Approach

• Crowdsourcing Users• Requestor• Workers

• Classification:• Applications• Performance

Introduction Crowdsourcing for Educational Evaluation in e-Learning 4

Classification of Crowdsourcing

Figure: A Classification of Crowdsourcing

Introduction”Motivation”

• The main motivation behind crowdsourcing workers are:• Financial Incentives, e.g., Amazon Mechanical Turk

• Entertainment, e.g., the ESP Game• Create a community, e.g., Wikipedia and Stack Overflow.

• The main motivation behind crowdsourcing workers are:• Financial Incentives, e.g., Amazon Mechanical Turk• Entertainment, e.g., the ESP Game

• Create a community, e.g., Wikipedia and Stack Overflow.

• The main motivation behind crowdsourcing workers are:• Financial Incentives, e.g., Amazon Mechanical Turk• Entertainment, e.g., the ESP Game• Create a community, e.g., Wikipedia and Stack Overflow.

Crowdsourcing in Educational Evaluation

Need of Crowdsourcing in Educational Evaluation

• MOOC Courses• Flexible self paced method of learning• Attracts large number of participants

• Traditional approaches of evaluation

• Use the concept of Crowdsourcing• Motivational issues

• Peer to peer evaluation system

Crowdsourcing in Educational Evaluation Crowdsourcing for Educational Evaluation in e-Learning 8

Peer to Peer Evaluation System

Peer Evaluation

Peer evaluation is the evaluation of work by one or morepeople of similar competence to the creator of the workcalled peer

• Evaluation can be completed in time

• Learning advantage

Peer Evaluation

Challenges

• Diversity in Intelligence

• Reliability

• Poor judgment

• Time Consuming

• Lack of motivation

Challenges

• Reliability

• Poor judgment

• Time Consuming

Challenges

• Reliability

• Poor judgment

• Time Consuming

Challenges

• Reliability

• Poor judgment

• Time Consuming

Challenges

• Reliability

• Poor judgment

• Time Consuming

Problem Scope

Problem

Diversity in intelligence and domain knowledge, amongparticipants, creates an unavoidable problem of ensuringfairness in evaluation. Fairness here refers to the ability ofparticipants to evaluates their peers assignments

Proposed Solution

The problem of ensuring fairness in peer evaluation systemmay be resolved by exploiting the knowledge levels of theparticipants

Problem Scope Crowdsourcing for Educational Evaluation in e-Learning 11

Problem Scope

Problem

Diversity in intelligence and domain knowledge, amongparticipants, creates an unavoidable problem of ensuringfairness in evaluation. Fairness here refers to the ability ofparticipants to evaluates their peers assignments

Proposed Solution

The problem of ensuring fairness in peer evaluation systemmay be resolved by exploiting the knowledge levels of theparticipants

Problem Scope Crowdsourcing for Educational Evaluation in e-Learning 11

Proposed Solution to Ensure Fairness in Peer

Evaluation

Proposed System

Figure: Proposed System Workflow

Proposed Solution Crowdsourcing for Educational Evaluation in e-Learning 13

Measuring Knowledge Level of the Participants

• Use performance history of participants• Assumption: Prerecorded data is available for all participants in the

course.• In practical scenario this assumption may not always be correct .

• Take a quiz• Assumption: An expert is available who can set the quality quiz to

test knowledge• Delay in workflow• Performance of the system is highly dependent on the quality of the

• Use a combination of above approaches.• Assumption: An expert is available to

• Ensure that performance data is available of all participants• To set the quality quiz to test participant knowledge

Group Formation

• Number of evaluations per assignment

• Number of groups

• Number of particiants in each group

Our Approach

If we want that each peer should be evaluated by M peers,then we should form M groups. If the total number ofparticipants are N, then we should have ( N

M ) participantsin each group, and each participant will evaluate Mparticipants.

Group Formation

• Number of evaluations per assignment

• Number of groups

• Number of particiants in each group

Our Approach

If we want that each peer should be evaluated by M peers,then we should form M groups. If the total number ofparticipants are N, then we should have ( N

M ) participantsin each group, and each participant will evaluate Mparticipants.

Allocation Strategies of Projects for Evaluation Based onKnowledge Level

Classification

Participants are classified in classes C1,C2, ...Cn based ontheir knowledge level, whereKnowledge level(Ci ) > Knowledge level(Ci+1)

• Allocate project of participant from class Ci to participant of classCi+1

• Limitation: Does not guarantee fair evaluation

• Allocate project of participant from class Ci to participant of classCi−1

• Limitation: Participants may not have any significant learningadvantage by the process of evaluation.

• Allocate projects for evaluation of participant Pi of class Ck ,to exactly 1 participant from each class C1,C2.....,Cn

Classification

Calculating Final Grade

• Normalizing the grades from each class• Normalizing factor of a class [normalizing factor(Class(Ci))]• Normalized score by participant Pi of class Ci =

normalized score(Pi ) = score(Pi ) ∗ normalizing factor(Class(Ci ))

• Calculating final grade: Average of normalized scores from allparticipants is the final grade of project.

Final grade(project(pj)) =

n∑i=0

normalized score(partcipantij )

Implementation Details

RequirementsParticipant’s Requirements

• He should be able to register themselves in a new course

• He should be able to submit any pending assignment

• He should be able to appear in any quiz available

• He should have an interface to download his peer’s submission forevaluation

• He should have an interface to submit the evaluation results

Implementation Details Crowdsourcing for Educational Evaluation in e-Learning 19

RequirementsInstructor’s Requirements

• He should be able to create a new assignment for a course• He should be able to add new evaluation criteria to the system, for

assignment evaluation

• He should be able to add questions to the pool• He must have an interface to add/update/delete questions

• He should be able to create quiz with available questions for aspecific assignment

• He should be able to add a new course to the system

Use Case Diagram

Database Design

Participant’s Side InterfaceView Courses

Participant’s Side InterfaceSubmit Assignment

Participant’s Side InterfaceAttempt Quiz

Participant’s Side InterfaceEvaluate Assignments

Participant’s Side InterfaceSubmit Evaluation Result

Instructor’s Side InterfaceCourse Details

Instructor’s Side InterfaceAdd Question

Instructor’s Side InterfaceCreate Quiz

Instructor’s Side InterfaceCreate Assignment

Conclusion

• Large number of participants in MOOC

• Traditional approach of evaluation is not suitable

• Use peer evaluation

• Challenge: Diversity in intelligence and knowledge among peers

• Exploit knowledge level information to distribute task of evaluationto ensure fairness

Conclusion Crowdsourcing for Educational Evaluation in e-Learning 33

Conclusion

Future Work

• Implement other strategies, for measuring knowledge level

• Perform experiments with large number of participants

• Integration of developed system with edX platform

Future Work

References I

Roy Ballantyne, Karen Hughes, and Aliisa Mylonas.

Developing procedures for implementing peer assessment in large classes using an action research process.Assessment & Evaluation in Higher Education, 27(5):427–441, 2002.

Luis von Ahn and Laura Dabbish.

Labeling images with a computer game.In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’04, pages 319–326, New York, NY,USA, 2004. ACM.

M.E.D. van den Bogaard and G.N. Saunders-Smits.

Peer #x00026; self evaluations as means to improve the assessment of project based learning.In Frontiers In Education Conference - Global Engineering: Knowledge Without Borders, Opportunities Without Passports, 2007.FIE ’07. 37th Annual, pages S1G–12–S1G–18, 2007.

Daniel Weld, Eytan Adar, Lydia Chilton, Raphael Hoffmann, Eric Horvitz, Mitchell Koch, James Landay, Christopher Lin, and

Mausam Mausam.Personalized online education a crowdsourcing challenge.AAAI Workshops, 2012.

Man-Ching Yuen, I. King, and Kwong-Sak Leung.

A survey of crowdsourcing systems.In Privacy, security, risk and trust (passat), 2011 ieee third international conference on and 2011 ieee third internationalconference on social computing (socialcom), pages 766–773, 2011.

References Crowdsourcing for Educational Evaluation in e-Learning 35

crowdsourcing for educational evaluation in e-learning · 2014-06-23 · crowdsourcing for...

Documents