Evaluating Classifiers Reading: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)An introduction to ROC analysis Evaluating Classifiers What we want: classifier that best predicts unseen data Assumption: Data is iid (independently and identically distributed) Accuracy and Error Rate Accuracy = fraction of correct classifications on unseen data (test set) Error rate = 1 Accuracy How to use available data to best measure accuracy? Split data into training and test sets. But how to split? Too little training data: Dont get optimal classifier Too little test data: Measured accuracy is not correct One solution: k-fold cross validation Each example is used both as a training instance and as a test instance. Split data into k disjoint parts: S 1, S 2,..., S k. For i = 1 to k Select S i to be the test set. Train on the remaining data, test on S i, to obtain accuracy A i. Report as the final accuracy. Confusion matrix for a given class c Actual Predicted (or classified) PositiveNegative (in class c)(not in class c) Positive (in class c) TruePositiveFalseNegative Negative (not in class c)FalsePositiveTrueNegative Evaluating classification algorithms Example: A vs. B Results from Perceptron: Instance Class Perception Output 1 A1 2 A+1 3 A+1 4 A1 5 B+1 6 B1 7 B1 1 B1 Accuracy: ActualPredicted PositiveNegative Positive Negative Assume A is positive class Confusion Matrix Evaluating classification algorithms Confusion matrix for a given class c Actual Predicted (or classified) PositiveNegative (in class c)(not in class c) Positive (in class c) TruePositiveFalseNegative Negative (not in class c)FalsePositiveTrueNegative Type 2 error Type 1 error Precision and Recall All instances in test set Positive Instances Negative Instances Precision and Recall All instances in test set Instances classified as positive Positive Instances Negative Instances Instances classified as negative Precision and Recall All instances in test set Instances classified as positive Positive Instances Negative Instances Recall = Sensitivity = True Positive Rate Example: A vs. B Results from Perceptron: Instance Class Perception Output 1 A1 2 A+1 3 A+1 4 A1 5 B+1 6 B1 7 B1 1 B1 Assume A is positive class Results of classifier ThresholdAccuracyPrecisionRecall Creating a Precision vs. Recall Curve 14 (sensitivity) (1 specificity) 15 Results of classifier ThresholdAccuracyTPRFPR Creating a ROC Curve Precision/Recall versus ROC curves 17 18 19 20 There are various algorithms for computing AOC (often built into statistical software) we wont cover this in this class. How to create a ROC curve for a perceptron Run your perceptron on each instance in the test data, without doing the sgn step: Get the range [min, max] of your scores Divide the range into about 200 thresholds, including and max For each threshold, calculate TPR and FPR Plot TPR (y-axis) vs. FPR (x-axis)