Learn how to make your drawings come alive…

Lecture 3:SKETCH RECOGNITIONAnalysis, implementation, and comparison of sketch recognition

algorithms, including feature-based, vision-based, geometry-based, and timing-based recognition algorithms; examination of methods to combine results from various algorithms to improve recognition using

AI techniques, such as graphical models.

Class Overview

• Paper discussion (lead by Joshua Peschel)

• Discussion of previous homework and implementation issues

• Lecture on classification details of Rubine

• Introduction to Long

• Homework

Read Paper Discussion: Rubine (lead by Joshua Peschel)

• Thoughts?

Class Overview

• Paper discussion (lead by Joshua Peschel)

• Discussion of previous homework and implementation issues

• Lecture on classification details of Rubine

• Introduction to Long

• Homework

Homework Implementation Discussion

• Compare average feature values for each gesture type.

• What did people get?

Implementation Issues (Repeat Slide from Lecture 2)

• Tablets are faster now than they were• Rubine paper was based on slower data (mouse?)• But, for correct feel, pen needs to be fast• Issues you may have:

– Duplicate location (2 consecutive points in same place)– Duplicate time (2 consecutive points at the same time)– Divide by zero (because of above problems)– Make sure your to convert to double before dividing in Java– Remove the second point not the first for duplicate points

Performing Recognition on these Features

• What do you think about the features?

• Can you foresee any recognition problems?

• Which features will be more helpful?

Rubine Classification

• Evaluate each gesture 0 <= c <= C.

• Vc = value = goodness of fit for that gesture c.

• Pick the largest Vc , and return gesture c

Class Overview

• Paper discussion (lead by Joshua Peschel)

• Discussion of previous homework and implementation issues

• Lecture on classification details of Rubine

• Introduction to Long

• Homework

Rubine Classification

• Wc0 = initial weight of gesture

• Wci = weight for the I’th feature

• Fi = ith feature value

• Sum the features together

Compute gesture covariance matrix

• How are the features of the shape related to each other?

• Look at one example - look at two features – how much does each feature differ from the mean – take the average for all examples – that is one spot in the matrix

• http://mathworld.wolfram.com/Covariance.html• Is there a dependency (umbrellas/raining)

Normalize

• cov(X) or cov(X,Y) normalizes by N-1, if N>1, where N is the number of observations. This makes cov(X) the best unbiased estimate of the covariance matrix if the observations are from a normal distribution.For N=1, cov normalizes by N

• They don’t normalize for ease of next step (so just sum, not average)

What does it mean to “Normalize”

• Taking the average• But… we want to find the true variance.• Note that our sample mean is not exactly

the true mean.• By definition, our data is closer to the

sample mean than the true mean• Thus the numerator is too small• So we reduce the denominator to

compensate

Common Covariance Matrix

• How are the features related between all the examples?• Top = non normalize total covariance• Bottom = normalization factor = total number of

examples – total number of gesture types (classes)

Weights

• Wcj = weight for the jth feature of the cth shape

• Sum for each feature– Common Covariance Matrix inverted* ij

– Average feature value for the ith feature for the cth gesture

Initial Weight

• Initial gesture weight =

• Sum for each feature in class:– Feature weight * average feature value

Rubine Classification

• Evaluate each gesture 0 <= c <= C.

• Vc = value = goodness of fit for that gesture c.

• Pick the largest Vc , and return gesture c

Rubine Classification

• Wc0 = initial weight of gesture

• Wci = weight for the I’th feature

• Fi = ith feature value

• Sum the features together

Rubine Adjustments: Eliminate Jiggle

• Any input point within 3 pixels of the previous point is discarded

Rubine Adjustments:Rejection Technique 1

• If the top two gestures are near to each other, reject.

• Vi > Vj for all j != i

• Reject if less than .95

Rubine Adjustments:Rejection Technique 2

• Mahalanobis distance

• Number of standard deviations g is from the mean of its chosen class i.

Implementation Issues

• Java matrix class• JAMA for Java: http://

math.nist.gov/javanumerics/jama/ • or unused for C++: http://

www.techsoftpl.com/matrix/download.htm or unused for C++: http://www.programmersheaven.com/download/30784/download.aspx

Class Overview

• Paper discussion (lead by Joshua Peschel)

• Discussion of previous homework and implementation issues

• Lecture on classification details of Rubine

• Introduction to Long

• Homework

Long features (from Rubine)• 1. Cosine of initial angle• 2. Sine of initial angle• 3. Size of bounding box• 4. Angle of bounding box• 5. Distance between first• and last points• 6. Cosine of angle between• first and last points• 7. Sine of angle between first• and last points• 8. Total length• 9. Total angle• 10. Total absolute angle• 11. Sharpness

Long Features: New• 12. Aspect [abs( – #4)]• 13. Curviness• 14. Total angle traversed / total• length• 15. Density metric 1 [#8 / #5]• 16. Density metric 2 [#8 / #3]• 17. Non-subjective• “openness” [#5 / #3]• 18. Area of bounding box• 19. Log(area)• 20. Total angle / total absolute• angle• 21. Log(total length)• 22. Log(aspect)

Class Overview

• Paper discussion (lead by Joshua Peschel)

• Discussion of previous homework and implementation issues

• Lecture on classification details of Rubine

• Introduction to Long

• Homework

Homework

• Read Long paper – who wants to present?• Implement Classifier • Apply Rubine features to your classifier

(w/out adjustments – save for weekend). Computer accuracy on Math data.

• Implement Long Features• Apply Long features to your classifier

(compute accuracy on Math data)• Bring in Classification results