2008 nvo summer school11 basic concepts in data mining kirk borne george mason university t he us n...

2008 NVO Summer School 11

Basic Concepts in Data Mining

Kirk BorneGeorge Mason University

THE US NATIONAL VIRTUAL OBSERVATORY


OUTLINE

• The New Face of Science• Scientific Knowledge Discovery• Data Mining Examples and Techniques• Basic Concepts in Data Mining• What’s next?


OUTLINE



The Scientific Data FloodThe Scientific Data Flood

PipelinePipeline

Scientific Data FloodScientific Data Flood

Large ScienceProjectLarge ScienceProject


The New Face of Science – 1

• Big Data (usually geographically distributed)– High-Energy Particle Physics– Astronomy and Space Physics– Earth Observing System (Remote Sensing)– Human Genome and Bioinformatics– Numerical Simulations of any kind– Digital Libraries (electronic publication repositories)

• e-Science– Built on Web Services (e-Gov, e-Biz) paradigm– Distributed heterogeneous data are the norm– Data integration across projects & institutions– One-stop shopping: “The right data, right now.”


• Databases enable scientific discovery– Data Handling and Archiving (management of massive

data resources)– Data Discovery (finding data wherever they exist)– Data Access (WWW-Database interfaces)– Data/Metadata Browsing (serendipity)– Data Sharing and Reuse (within project teams; and by

other scientists – scientific validation)– Data Integration (from multiple sources)– Data Fusion (across multiple modalities & domains)– Data Mining (KDD = Knowledge Discovery in

Databases)

The New Face of Science – 2


OUTLINE



So what is Data Mining?

• Data Mining is Knowledge Discovery in Databases (KDD)

• Data mining is defined as “an information extraction activity whose goal is to discover hidden facts contained in (large) databases.”

• Note: Machine Learning is the field of Computer Science research that focuses on algorithms that learn from data.

• Data Mining is the application of Machine Learning algorithms to large databases.


Scientific Data MiningData Mining is the Killer App for

Scientific Databases.• Scientific Data Mining References:

– http://voneural.na.infn.it/ – http://astroweka.sourceforge.net/ – http://www.itsc.uah.edu/f-mass/

• Framework for Mining and Analysis of Space Science data (F-MASS)

• Data mining is used to find patterns and relationships in data. (EDA = Exploratory Data Analysis)

• Patterns can be analyzed via 2 types of models:– Descriptive : Describe patterns and create meaningful subgroups or

clusters. (Unsupervised Learning, Clustering)– Predictive : Forecast explicit values, based upon patterns in known

results. (Supervised Learning, Classification)

• How does this apply to Scientific Research? … – through KNOWLEDGE DISCOVERY

Data Information Knowledge Understanding / Wisdom!


Understanding: the Universe is expanding!!

Astronomy Example

Data:

Information (catalogs / databases):– Measure brightness of galaxies from image (e.g., 14.2 or 21.7)– Measure redshift of galaxies from spectrum (e.g., 0.0167 or

0.346)

Knowledge:Hubble Diagram Redshift-Brightness

Correlation Redshift = Distance

(a) Imaging data (ones & zeroes)

(b) Spectral data (ones & zeroes)


Astronomers have been doing Data Mining for centuries

“The data are mine, and you can’t have them!”

• Seriously ... • Astronomers love to classify things ...

(Supervised Learning. e.g., classification)

• Astronomers love to characterize things ... (Unsupervised Learning. e.g., clustering)

• And we love to discover new things ... (Semi-supervised Learning. e.g., outlier detection)


This sums it up ...

• Characterize the new (clustering)

• Assign the known (classification)

• Discover the unknown (outlier detection)

• 2 benefits of very large data sets within a scientific domain:

•best statistical analysis of “typical” events

•automated search for “rare” events

Graphic from S. G. Djorgovski


OUTLINE



Database Systems and Data Mining

• Data mining brings novel non-traditional (Machine Learning) concepts to large DBMS (e.g., association mining; neural networks; decision trees; link analysis; pattern recognition; classification; regression; self-organizing maps). For example:

– Clustering Analysis = group together similar items, and separate the dissimilar items

– Classification = predict the class label

– Regression = predict a numeric attribute value

– Association Analysis = detect attribute-value conditions that occur frequently together


ClusteringClassificationAssociationsNeural NetsDecision TreesPattern RecognitionCorrelation/Trend AnalysisPrincipal Component

AnalysisIndependent Component

AnalysisRegression AnalysisOutlier/Glitch IdentificationVisualizationAutonomous AgentsSelf-Organizing Maps (SOM)Link (Affinity Analysis)

Data Mining Methods and Some Examples

Classify new data items usingthe known classes & groups

Find unusual co-occurring associationsof attribute values among DB items

Organize information in the database based on relationships among key data descriptors

Identify linkages between data items based on features shared in common

Group together similar items andseparate dissimilar items in DB

Predict a numeric attribute value


Some Data Mining Techniques Graphically Represented

Self-Organizing Map (SOM)

Outlier (Anomaly) Detection

Clustering

Link Analysis Decision Tree

Neural Network


• Supervised Learning– Classification

• Unsupervised Learning– Clustering– Link Analysis– Association Analysis

• Semisupervised Learning– Outlier Detection– Class Discovery

Categories of Machine Learningand some Examples


Some Classification AlgorithmsClassification = the process of learning and then applying a function that classifies the data into a set of predefined classes.

• Bayes Theorem • Support Vector

Machines (SVM)• Decision Trees• Regression• Neural Networks• Markov Modeling• K-Nearest Neighbors


Classification - a 2-Step Process

1. Model Construction (Description): describing a set of predetermined classes = Build the Model.

– Each data element/tuple/sample is assumed to belong to a predefined class, as determined by the class label attribute

– The set of tuples used for model construction = the training set– The model is represented by classification rules, decision trees, or

mathematical formulae

2. Model Usage (Prediction): for classifying future or unknown objects, or for predicting missing values = Apply the Model.

– It is important to estimate the accuracy of the model:• The known labels of the test sample are compared with the

classification results from the model• Accuracy rate is the percentage of test set samples that are

correctly classified by the model• Test set is chosen completely independent of the training set,

otherwise overfitting will occur – overfitting is a bad thing!


Classification Methods:Decision Trees, Neural Networks, SVM (Support Vector Machines)

There are 2 Classes!

How do you ...-Separate them?-Distinguish them?-Learn the rules?-Classify them?

ApplyKernel

(SVM)


Some Clustering AlgorithmsClustering = the process of partitioning a set of data into subsets or clusters such that a data element belonging to a cluster is more similar to data elements belonging to that same cluster than to the data elements belonging to other clusters.

– Squared Error– Nearest Neighbor– K-Means (most

popular)– Mixture Models

(statistical)


Clustering is used to discover the different unique groupings (classes) of attribute values.The case shown below is not obvious: one or two groups?


This case is easier: there are two groups.(in fact, this is the same set of data elements as shown on the previous slide, but plotted here using a different attribute.)


Semi-supervised Learning: Outlier Detection and Class Discovery

Figure: The clustering of data clouds (dc#) within a multidimensional parameter space (p#).

Such a mapping can be used to search for and identify clusters, voids, outliers, one-of-kinds, relationships, and associations among arbitrary parameters in a database (or among various parameters in geographically distributed databases).

• statistical analysis of “typical” events

• automated search for “rare” events


Outlier Detection: Serendipitous Discovery of Rare or New Objects & Events


Principal Components Analysis &Independent Components Analysis

Cepheid Variables:Cosmic Yardsticks-- One Correlation-- Two Classes!

... Class Discovery!


Why use Data Mining?Here are 6 reasons...

1. Most projects now collect massive quantities of data.2. Because of the enormous potential for new discoveries in

existing huge databases.3. Data mining moves beyond the analysis of past events to

predicting future trends and behaviors that may be missed because they lie outside experts’ expectations.

4. Data mining tools can answer complex questions that traditionally were too time- consuming to resolve.

5. Data mining tools can explore the intricate interdependencies within databases in order to discover hidden patterns and relationships.

6. Data mining allows decision-makers to make proactive, knowledge-driven decisions.


OUTLINE



Basic Concepts = Key Steps

• The key steps in a data mining project usually invoke and/or follow these basic concepts:– Data browse, preview, and selection– Data cleaning and preparation– Feature selection– Data normalization and transformation– Similarity/Distance metric selection– ... Select the data mining method– ... Apply the data mining method– ... Gather and analyze data mining results– Accuracy estimation– Avoiding overfitting


Key Concept for Data Mining:Data Previewing

• Data Previewing allows you to get a sense of the good, bad, and ugly parts of the database

• This includes:– Histograms of attribute distributions– Scatter plots of attribute combinations– Max-Min value checks (versus expectations)– Summarizations, aggregations (GROUP BY)– SELECT UNIQUE values (versus expectations)– Checking physical units (and scale factors)– External checks (cross-DB comparisons)– Verify with input DB


Key Concept for Data Mining:Data Preparation = Cleaning the Data

• Data Preparation can take 40-80% (or more) of the effort in a data mining project

• This includes:– Dealing with NULL (missing) values– Dealing with errors– Dealing with noise– Dealing with outliers (unless that is your

science!)– Transformations: units, scale, projections– Data normalization– Relevance analysis: Feature Selection– Remove redundant attributes– Dimensionality Reduction


Key Concept for Data Mining:Feature Selection – the Feature Vector

• A feature vector is the attribute vector for a database record (tuple).

• The feature vector’s components are database attributes: v = {w,x,y,z}

• It contains the set of database attributes that you have chosen to represent (describe) uniquely each data element (tuple).– This is only a subset of all possible attributes in the DB.

• Example: Sky Survey database object feature vector:– Generic: {RA, Dec, mag, redshift, color, size}– Specific: {ra2000, dec2000, r, z, g-r, R_eff }

→


Key Concept for Data Mining:Data Types

• Different data types:– Continuous:

• Numeric (e.g., salaries, ages, temperatures, rainfall, sales)– Discrete:

• Binary (0 or 1; Yes/No; Male/Female)• Boolean (True/False)• Specific list of allowed values (e.g., zip codes; country names;

chemical elements; amino acids; planets)– Categorical:

• Non-numeric (character/text data) (e.g., people’s names)• Can be Ordinal (ordered) or Nominal (not ordered)• Reference: http://www.twocrows.com/glossary.htm#anchor311516

• Examples of Data Mining Classification Techniques:– Regression for continuous numeric data– Logistic Regression for discrete data– Bayesian Classification for categorical data


Key Concept for Data Mining:Data Normalization & Data Transformation

• Data Normalization transforms data values for different database attributes into a uniform set of units or into a uniform scale (i.e., to a common min-max range).

• Data Normalization assigns the correct numerical weighting to the values of different attributes.

• For example:– Transform all numerical values from min to

max on a 0 to 1 scale (or 0 to Weight ; or -1 to 1; or 0 to 100; …).

– Convert discrete or character (categorical) data into numeric values.

– Transform ordinal data to a ranked list (numeric).

– Discretize continuous data into bins.


Key Concept for Data Mining:Similarity and Distance Metrics

• Similarity between complex data objects is one of the central notions in data mining.

• The fundamental problem is to determine whether any selected pair of data objects exhibit similar characteristics.

• The problem is both interesting and difficult because the similarity measures should allow for imprecise matches.

• Similarity and its inverse – Distance – provide the basis for all of the fundamental data mining clustering techniques and for many data mining classification techniques.


Similarity and Distance Measures (metrics)


Similarity and Distance Measures

• Most clustering algorithms depend on a distance or similarity measure, to determine (a) the closeness or “alikeness” of cluster members, and (b) the distance or “unlikeness” of members from different clusters.

• General requirements for any similarity or distance metric:– Non-negative: dist(A,B) > 0 and sim(A,B) > 0– Symmetric: dist(A,B)=dist(B,A) and sim(A,B)=sim(B,A)

• In order to calculate the “distance” between different attribute values, those attributes must be transformed or normalized (either to the same units, or else normalized to a similar scale).

• The normalization of both categorical (non-numeric) data and numerical data with units generally requires domain expertise. This is part of the pre-processing (data preparation) step in any data mining activity.


Popular Similarity and Distance Measures

• General Lp distance = ||x-y||p = [sum{|x-y|p}]1/p

• Euclidean distance: p=2

– DE = sqrt[(x1-y1)2 + (x2-y2)2 + (x3-y3)2 + … ]

• Manhattan distance: p=1 (# of city blocks walked)

– DM = |x1-y1| + |x2-y2| + |x3-y3| + …

• Cosine distance = angle between two feature vectors:

– d(X,Y) = arccos [ X ٠ Y / ||X|| . ||Y|| ]

– d(X,Y) = arccos [ (x1y1+x2y2+x3y3) / ||X|| . ||Y|| ]

• Similarity function: s(x,y) = 1 / [1+d(x,y)]

– s varies from 1 to 0, as distance d varies from 0 to .

8


Data Mining Clustering and Nearest Neighbor Algorithms – Issues

• Clustering algorithms and nearest neighbor algorithms (for classification) require a distance or similarity metric.

• You must be especially careful with categorical data, which can be a problem. For example:– What is the distance between blue and

green? Is it larger than the distance from green to red?

– How do you “metrify” different attributes (color, shape, text labels)? This is essential in order to calculate distance in multi-dimensions. Is the distance from blue to green larger or smaller than the distance from round to square? Which of these are most similar?


Typical Error Matrix:

Class-A Class-B Totals

Class-A

Class-B

Totals

TRAINING DATA (actual classes)

NEU

RAL

NET

WO

RK

CLA

SSIF

ICATI

ON

(out

put)

3007

318(FN)

34213103(TN)

32763152 6428

173(FP)

2834(TP)

True Positive False PositiveFalse Negative True Negative

Key Concept for Data Mining: Classification Accuracy


Typical Measures of Accuracy

• Overall Accuracy = (TP+TN)/(TP+TN+FP+FN)

• Producer’s Accuracy (Class A) = TP/(TP+FN)• Producer’s Accuracy (Class B) = TN/(FP+TN)• User’s Accuracy (Class A) = TP/(TP+FP)• User’s Accuracy (Class B) = TN/(TN+FN)Accuracy of our Classification on preceding slide:• Overall Accuracy = 92.4%• Producer’s Accuracy (Class A) = 89.9%• Producer’s Accuracy (Class B) = 94.7%• User’s Accuracy (Class A) = 94.2%• User’s Accuracy (Class B) = 90.7%


Key Concept for Data Mining:Overfitting

d(x)

• g(x) is a poor fit (fitting a straight line through the points)

• h(x) is a good fit • d(x) is a very poor fit (fitting every point) = Overfitting


How to Avoid Overfitting in Data Mining Models

• In Data Mining, the problem arises because you are training the model on a set of training data (i.e., a subset of the total database).

• That training data set is simply intended to be representative of the entire database, not a precise exact copy of the database.

• So, if you try to fit every nuance in the training data, then you will probably over-constrain the problem and produce a bad fit.

• This is where a TEST DATA SET comes in very handy. You can train the data mining model (Decision Tree or Neural Network) on the TRAINING DATA, and then measure its accuracy with the TEST DATA, prior to unleashing the model (e.g., Classifier) on some real new data.

• Different ways of subsetting the TRAINING and TEST data sets:• 50-50 (50% of data used to TRAIN, 50% used to TEST)

• 10 different sets of 90-10 (90% for TRAINING, 10% for TESTING)


Schematic Approach to Avoiding Overfitting

Error

Training Epoch

Test Set error

TrainingSet error

To avoid overfitting, you need to know when to stoptraining the model. Although the Training Seterror may continue to decrease, you may simply beoverfitting the Training Data.Test this by applying themodel to Test Data (not partof Training Set). If the TestSet error starts to increase, then you know that you areoverfitting the Training Setand it is time to stop!

STOP Training HERE !


OUTLINE


2008 NVO Summer School 462008 NVO Summer School 46

Scientific Data Mining in Astronomy

2008 nvo summer school11 basic concepts in data mining kirk borne george mason university t he us n...

Documents