competency, compatibility and preferences in reciprocal peer recommendation · ·...

Competency, Compatibility and Preferences in Reciprocal Peer Recommendation

Boyd A. PottsThe University of Queensland

[email protected]

Carl ReidsemaThe University of Queensland

[email protected]

Hassan Khosravi The University of Queensland

[email protected]

With thanks to:Aneesha Bakharia, Mark Belonogoff and Melanie Fleming

1

Potts, Khosravi & Reidsema, 2017. Competency, Compatibility and Preferences in Reciprocal Peer Recommendation

Outline

1. Introduction2. Platform Description3. Problem Formulation & Notation4. Competency Preference Model & Compatibility Function5. Reciprocal Peer Recommendation6. Evaluation7. Future Work

2


1. Introduction

3

0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

Chart 1: Higher education full year student data, commencing students by year

Commencing student enrolments

Commencing student enrolments Commonwealth supported

(Department of Education and Training, 2017)


1. Introduction

4

• Opportunity for new approaches to improving students’ social and peer networks

• Engagement and networks contribute to student success (Wilcox et al., 2005)

• Social isolation contributes to student failure (McInnerney & Roberts, 2004)

• Support development of networks with fit-for-purpose technology


1. Introduction

5

• Recommender systems help people find resources in an otherwise overwhelming environment.

• Satisfy user preferences to a higher degree than competing items (e.g. online content, shopping)

Kluver, D (2015)


1. Introduction

6

• Reciprocal recommendation seeks to connect two users such that both sets of preferences are satisfied (e.g. online dating)

• Key to recommendation is the user preference model. We propose a reciprocal recommender for peer study sessions with compatibility and preferences based on competencies

1. Competent users recommended to less competent for support• By how much should users’ competencies differ?• How to model this as a recommendation?

2. Users of similar competencies recommended for study partnerships


2. Platform Description

7

• Recommendation in Personalised Peer Learning Environments (RiPPLE)• https://hkhosrav.github.io/RiPPLE-Core/

• Open-source, course-level, student-facing platform where learners can provide learning support, seek support, find study partners



8

• Individuals nominate weekly availability and learning support preferences for course-relevant topics (e.g. SQL, Loops)



9

• Indicator of competency (derived initially from MCQ responses in RiPPLE, updated with cumulative assessment over learning period)


3. Problem Formulation & Notation

10

Element Form DescriptionUsers U Number of learners using RiPPLE, u denotes an arbitrary user

Topics L Number of course topics, l denotes arbitrary topic

Times T Number of weekly time slots for scheduling a session, t denotes arbitrary time slot

Roles Q Number of roles, q=1 for providing support, q=2 for seeking support, q=3 to seek study partners


3. Problem Formulation & Notation

11

Element Form Description

Requests RUxLxQA three-dimensional array where Rulq

= 1 indicates that user u has indicated interest in participating in a study session on topic l with role q

Availability AUxTA two-dimensional array in which Aut

= 1 shows that user u is available at time t

Competencies CUxLA two-dimensional array in which Cul

shows the competency of user u in topic l on a 100-point scale

Preferences PUxQA two-dimensional array in which P

uqshows the competency preference of

a user u in role q


4. Competency Preference Model & Compatibility Function

12

• Compatibility between two users ultimately a function of –

1. Individual competencies

2. Joint competency threshold

3. Competency preferences –• i.e. how competent do you prefer a partner to be?• Role-driven



13

• Define joint competency as the magnitude of the vector of two competencies in Cartesian space

• Propose that peers’ joint competency (J) should be above a certain threshold (τ < J) for effective sessions

0

20

40

60

80

100

0 20 40 60 80 100

Co

mp

ete

ncy

, use

r1

Competency, user2

Topic l

user1 user2 J



14

• Use joint competency (J) in a logistic function (H) to compute the extent to which a partnership meets the desirable threshold (τ), with leniency parameter (α)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.6 0.8 1

H s

core

Joint competency

more strict more lenient



15

• Users set preferences (puq) for the competencies in their peers

• E.g. pu11 = -10 means u1 is comfortable providing support to peers whose competency is 10 points below their own competency (Cu11)

• Compatibility w.r.t. puq is calculated as the height of a Gaussian function (G) with centre Cu11 + pu11 and standard deviation σ

• σ models the leniency for matching peers that do not fit their exact preference puq



16

• pu11 = -10 , Cu11 = 75

• Constrain eligibility ‖ users (1) provide support to less competent users, (2) receive support from more competent users (3) find partners of similar competency.

0

20

40

60

80

100

120

0 20 40 60 80 100 120

G s

core

Competency



17

• Putting it together:

• The compatibility score (s) between two users is the product of H and G, summed over matched topics l and related role preference q


5. Reciprocal Peer Recommendation

18

• The algorithm takes input requests, availabilities and competencies, and generates a list of up to k recommendations for each user in the entire cohort.



19

• The harmonic mean guarantees to provide smaller reciprocal scores for users whose compatibilities differ considerably, so as to prioritise recommendations that benefit both users

1. Select a user u1, then for each other user (u2) uses A to find a mutually convenient time slot

2. R is used to find a set of matching roles and associated topics

3. Users not satisfying constraints A and R receive score ε

4. Reciprocal score Score[u2] is calculated as the harmonic mean of the compatibilities from u1→u2 and u2→u1



20

• u1 is providing support and prefers users who have competency 10 points lower)

• u2 is seeking peer support with a preference for those who have competency 80 points higher

Preference of the peer

providing supporterReciprocal score


receiving supporter



21

• The extent to which both users are recommended to each other is defined by the harmonic mean distribution shown in the third frame


providing supporterReciprocal score


receiving supporter


6. Evaluation

22

• Synthetic data generated for R, C, A, P (see paper for details)

• Evaluated on scalability (runtime with number of users and recommendations)

• Evaluated on reciprocality:Learner u1 is a successful (reciprocal) recommendation for (out of the K-total) for learner u2, iff u1 is also in the top k recommendations of u2 (Prabhakar, 2017)


6. Evaluation

23

Runtime Reciprocality


7. Future Work

24

• Primary results indicate that RiPPLE can provide recommendations across a range of competency levels and cohort sizes

• Planned implementation in large introductory courses

• Subsequent empirical evaluation –• Designed A/B testing in RiPPLE; evaluate with control group receiving

random recommendations• Scalability, reciprocality, peer feedback, course performance (grade)• Behavioural – how learners choose among recommendations,

conditions of accepting recommendations


Comments, feedback

25

[email protected]

ReferencesHigher education student enrolment summary time series statistics full year, 2006-2016. Department of Education and Training. Downloaded from https://docs.education.gov.au/node/45146

JoanneMMcInnerney and Tim S Roberts. 2004. Online learning: Social interaction and the creation of a sense of community. Educational Technology & Society 7, 3(2004), 73–81.

Paula Wilcox, Sandra Winn, and Marylynn Fyvie-Gauld. 2005. “It was nothing to do with the university, it was just the people”: the role of social support in the first-year experience of higher education. Studies in higher education 30, 6 (2005), 707–722.

Kluver, D. (2015). What is a recommender system? https://www-users.cs.umn.edu/~kluve018/what-is-a-recommender.html

mailto:[email protected]

https://docs.education.gov.au/node/45146

competency, compatibility and preferences in reciprocal peer recommendation · ·...

Documents