Example-Based Problem Solving Support Using Concept Analysis of Programming

Content Roya Hosseini1 and Peter Brusilovsky1,2

1 Intelligent Systems Program (ISP)2 School of Information Sciences

University of Pittsburgh

12th International Conference on Intelligent Tutoring Systems

Young Researchers TrackJune 8, 2014

University of Pittsburgh Intelligent Systems Program

2

Purpose

Select relevant examples for:Supporting problem solving in Java

Programming

Introduce two concept-based approach for:Finding similar examples to a question

3

Outline

Literature Review

Concept-based Similarity Approach

Evaluation: Lab Study

Example-based Problem Solving

Helps students in solving problems

Finds most relevant examples for problems

4

5

ELM-ART

ELM-ART: an ITS for LISP programming (Weber 1996)

Required advance analysis for:Task DescriptionDomain KnowledgeLearner Model

6

Outline

Literature Review

Concept-based Similarity Approach

Evaluation: Lab study

7

Concept-based Similarity

Uses a Standard Ontology

Extracts concepts in programming content

(Hosseini & Brusilovsky 2013)

Measures similarity of concepts in contents

is-a

Abstraction

Inheritance

Encapsulation

OverridingMethod

InheritanceField

Inheritance

OverridingEquals

OverridingHash Code

is-a is-ais-a

is-a

is-a is-a

8

Example

public class Tester { public static void main (String[] args) { int x = 0; for (int i = 0; i < 10; i++){ for (int j = 0; j < 10; j++){ … } } }}

for (int i = 0; i < 10; i++){ …}for (int j = 0; j < 10; j++){ …}

Global

for (int i = 0; i < 10; i++){ for (int j = 0; j < 10; j++){

… } }

Local

9

Cosine Similarity

Question vector

Example vector

Global Similarity

Wn-1

Wn

W2

...W1

Cn-1

Cn

C2

...C1

Wn-1

Wn

W2

...W1

Cn-1

Cn

C2

...C1

10

Local Similarity

#1: Forms a subtree from concepts in the same block

if

++

<

If ( x < 2 ) {

x++;}

11

Local Similarity

#2: Compares subtrees of question and exampleTree Edit Distance (TED)

Example

a

cb

e

if

Question

e

gf

a

cb d

a

cb d

a

cb

e

if

1 2 TED: 3

e

gf

12

Local Similarity

#3: Measures similarity based on TED

Example

a

cb

e

if

a

hd

e

gh i

i

lk m n

Example

Question

e

gf

a

cb d TED: 3

TED: 6

13

Global vs. Local

We Are Globally Similar!

14

Outline

Literature Review

Concept-based Similarity Approach

Evaluation: Lab study

15

Study Design

Time Rating #Example

Approach

Failing in question Optional 5 Random

End of question

Mandatory 4 Both

16

Date: January, 2014 12 students

Java Contents:6 topics83 annotated examples24 parametric questions

17

Task

Pretest Solving 4 questions in 3 Java topics

Rating helpfulness of examples‘Not helpful at all’(0) - ‘Very helpful’(3)

Post-test

18

Solving question

19

Examples & Rating

20

Results

GLOBAL LOCAL0

1

2

31.95

1.49

Average Users Rat-ings

21

Rating EvaluationWork in progress

RMSE Precision 2+

Precision 3 MRR

Global Local 0

0.20.40.60.8

1

0.29 0.32

Average RMSE

22

Rating EvaluationWork in progress

RMSE Precision 2+

Precision 3 MRR

Global Local 0

0.5

10.69 0.62

Average Precision 2+

23

Rating EvaluationWork in progress

RMSE Precision 2+ Precision 3 MRR

Global Local 0

0.5

1

0.28 0.23

Average Precision 3

24

Rating EvaluationWork in progress

RMSE Precision 2+ Precision 3 MRR

Global Local 0

0.5

10.7

0.61

Average MRR

25

Work in progress

RMSE Precision 2+

Precision 3 MRR0

0.20.40.60.8

1

Ratings over Difficulty Level of Question

Global Easy Local EasyGlobal Moderate Local Moderate

26

HOW-TOsWork in progress

#1: Define structure of the content?

#2: Do personalized example selection?

27

Work in progress

Concepts that appear together in a:Block/line

#1: Define structure of the content?

if

++

<

If ( x < 2 ) {

x++;}

28

Work in progress

Consider user knowledge information Select examples with:

least unknown partsenough new parts

#2: Do personalized example selection?

29

Work in progress

Lab study : 30 subjectsPersonalized & non-personalized Global-Local approach

Personalized example selection did not work out! Hard question: Local approach has the least RMSE Easy-Medium question: Global approach has the least RMSE

Concept-based Similarity

User Knowledge Level

Personalized Example-Selection

30

Discussion

Global & Local Concept-based approach:Generalizability across other programming

domains

Limitations:Few contentsFew subjects

31

Next Steps

Investigating:

learning gains of students in the studyother approaches for capturing content

structurepersonalized example selection○ user knowledge, …

Adaptive visualization of problem-example space

Thank You! Personalized Adaptive Web Systems School of Information Sciences University of Pittsburgh

Roya Hosseini (roh38@pitt.edu)http://people.cs.pitt.edu/~hosseini/