csse 2014 hmm presentation_ta_ed

Machine Learning Algorithms: Applications to Educational Data

Saad Chahine, PhD May 26, 2014

Machine Learning “Field of study that gives computers the ability to learn without being explicitly programmed.” (Arthur Samuel, 1959)

“A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.” ( Tom Mitchel (1997), Machine Learning, McGraw Hill | Web Page http://www.cs.cmu.edu/~tom/ )

Two Main Types of Algorithms

Supervised learning:• What we are commonly used to in educational research• We know the data and outputs • We have an idea of the kids of analysis we plan to run (e.g., Linear

Regression)

Unsupervised learning:• Used less often in educational research • We try to find a hidden structure to data that may not be labeled • We have more of an intuition of what we are trying to find (e.g. K-

Means Cluster)

My Interest in Machine Learning

Q: Can we begin to build software programs that learn who we are and can then provide individual learning supports through the use of assessment and feedback?

Markov Models “The future is independent of the past, given the present.” (translation Andrey Markov, 1856-1922)

Limitation – Only takes into account current state and the most recent prior state

Hidden Markov Models - A method(s) of finding a hidden

(latent) structure with a sequential data set

Ghahramani, Z.(2001) An introduction to hidden Markov models and Bayesian networks. International Journal of Pattern Recognition and Artificial Intelligence, 15(1): 9-42.

Piaget Developmental Data

• Visser & SpeekenBrink (2010). depmixS4: An R package for hidden Markov Models. International Journal of Statistical Software, 36(7). http://dare.uva.nl/document/361939

• Data from: Jansen, B.R.J., & van der Maas, H.L.J. (2002). The development of children’s rule use on the balance scale task. Journal of Experimental Child Psychology, 81(4), 383–416.

• Siegler, R.S. (1981). Developmental sequences within and between concepts. Number 46 in Monographs of the Society for Research in Child Development. SRCD.

depmixS4 – Balance Data > data(balance)- 779 participants - Ages from 5-19 years - 4 distance items

CAPP - OSCE• Clinical Assessment for Practice Program

(CAPP)• A program of the College of Physicians

and Surgeons of Nova Scotia (CPSNS)• Objective Structured Clinical Exam (OSCE)• Multiple stations with sequences &

competencies

CAPP OSCE Dataset • 434 observations • 31 participants • 14 stations • 9 measures of competency (Coded

P/F) • 13 different case IDs

My Learning • I conducted the balance data

analysis first• Then I began to examine the OSCE

data• The next slides compare the two as

preliminary analysis

Balance • “Used Age as a

covariate on class membership”

• 3 State Model best• Converged in 77

iterations • loglink = -917.50• AIC = 1867• BIC = 1942

OSCE• Used CASE ID as a

covariate on class membership

• 2 state model best• Converged in 55

iterations • loglink = -1757.81• AIC = 3555• BIC = 3637

Balance • Probabilities at

zero values of the covariates

• 0.001, 0.988, 0.009

OSCE• Probabilities at

zero values of the covariates

• 0.606, 0.394

Balance

OSCE

Balance

OSCE

Balance

OSCE

What's Next • OSCE data did not fit as well as the

Balance data – More Years may help • Learning HMM further and potential

application to performance assessments

• Experiment with different covariates in the datasets

Acknowledgements • Acknowledge the use code by Visser

& SpeekenBrink (2010) in depmixS4 package

• Thank you to CAPP & Bruce Holmes

for the use of OSCE data

Thank You

saad.chahine@msvu.ca