id3 decision tree algorithm
TRANSCRIPT
ID3 Decision Tree Algorithm
By: Giovanni Ponzio, Jory Weiss, and Enoc Flores Hernandez
What’s The Goal of ID3
● The Iterative Dichotomiser 3 can help make more informed decisions with the use of data. Using a visualization, it creates a simple tree with a few levels of decision nodes.
Who created ID3?
● John Ross Quinlan developed the ID3 Algorithm at the University of Sydney● Ross Quinlan is an Australian computer scientist and researcher in machine
learning, data mining, and decision theory
What is a Decision Tree?
● A decision tree is a data structure made of nodes and edges
● Must be built from a given dataset containing columns representing
attributes and rows corresponding to records
● Each node is used to either make a decision or represent an outcome
● The root and intermediate nodes represent decisions (called decision
nodes) while the leaf nodes represent outcomes
How should the data look?● The data that ID3 uses must meet a few
requirements:○ Must be an attribute-value pairs table (meaning each
column is an attribute containing discrete values)○ Each sample entry (each row) must have the same
attributes (columns), and each attribute must have a fixed number of possible discrete values.
○ Each attribute must have discrete values which are easily differentiable (hard, quite hard, flexible, soft, and quite soft would not be suitable values for an attribute)
○ There must be sufficient data so that valid patterns are distinguishable from chance occurrences.
Decision tree used to answer the question Is the person fit?
Root Node: Is the person younger than 30 years old? Decision Nodes: 1. Does the person eat junk food?
2. Does the person work out? Leaf Nodes: 1. The person is Fit
2. The person is Unfit
Some ID3 Terminology
Every column used to make decisions in the data set is called a feature or an attribute
The column used to make leaf nodes is the target attribute
The values pertaining to each attribute are referred to as classes
Entropy● The ID3 Algorithm will select the “best” feature at each step while building the
tree (greedy)● It will decide which feature is “best” using calculations involving entropy● Entropy is a fundamental theorem used to to measure the importance of
information relative to its size● In this context, the entropy of a dataset is the measure of disorder in the
target attribute of the dataset
Entropy(S) = - ∑ pᵢ * log₂(pᵢ) ; i = 1 to n
Where S is the dataset, n is the number of classes in the target column and pi is the probability of class i (number of rows with ‘i’ in the target column divided by the total
number of rows)
Information Gain
● Information gain is a measure used to determine how well a given attribute separates or classifies the target values.
● The attribute with the highest information gain is selected as the “best” one and is placed at the root decision node.
IG(S, A) = Entropy(S) - ∑((|Sᵥ| / |S|) * Entropy(Sᵥ))
Where SV is the set of rows in S for which the attribute A has value v
● ID3 uses Entropy and Information Gain to form a functioning decision tree
Steps in ID3
1. Calculate the Information Gain (and thus the entropy) of every attribute in the dataset.
2. Select the feature with the maximum Information Gain as the root node of the Decision Tree, with its edges corresponding to it’s classes.
3. Calculate the Information Gain of the remaining attributes, and, again, select the feature with the maximum Information Gain as the next decision tree node
4. If all rows belong to the same class (in the target column), make the current node as a leaf node with the class as it’s label
5. Repeat this process until there are no remaining features, or until the decision tree has all leaf nodes.
Applications of ID3
The applications for ID3 are endless. As shown in the examples, ID3 is a classification algorithm that creates a decision tree which is used to make an educated guess about an attribute (target attribute) of a specific subject based its other known attributes. A few examples of applications are:
● Deciding whether or not to play a sport based on different weather attributes● Deciding whether or not somebody has an illness/injury based on the
symptoms they’re showing● Detecting web based attacks using on past data● E-commerce - efficiently implementing promotional or paid target advertising
on websites for potential buyers using past purchase records● Deciding the best move in chess based off the board and possible endings
Animation: Covid-19 Mock Dataset
ReferencesElteir, Marwa K. Pseudocode of ID3 Decision Tree Algorithm, 2006, www.researchgate.net/figure/Pseudocode-of-ID3-Decision-Tree-Algorithm_fig13_259754610
“The ID3 Algorithm.” ID3, www.cise.ufl.edu/~ddd/cap6635/Fall-97/Short-papers/2.htm.
Jazuli, Hafidz. “An Introduction to Decision Tree Learning: ID3 Algorithm.” Medium, Machine Learning Guy, 13 May 2018,
medium.com/machine-learning-guy/an-introduction-to-decision-tree-learning-id3-algorithm-54c74eb2ad55.
News, SIGKDD. “SIGKDD Awards.” SIGKDD, 2020, www.kdd.org/awards/view/2011-sigkdd-innovation-award-dr.-j.-ross-quinlan.
Patil, M., et al. “Effective Concept for Detection of Web Based Attacks Using ID3 Algorithm.” Semantic Scholar, 1 Jan. 1970,
www.semanticscholar.org/paper/Effective-Concept-for-Detection-of-Web-Based-Using-Patil-Kapgate/d58f2b4698bfa064059956d77c7fa927f1645cc7.
Quinlan, Ross. “Ross Quinlan.” Ross Quinlan's Personal Homepage, www.rulequest.com/Personal/.
“Ross Quinlan.” Wikipedia, Wikimedia Foundation, 13 July 2020, en.wikipedia.org/wiki/Ross_Quinlan.
Sakkaf, Yaser. “Decision Trees for Classification: ID3 Algorithm Explained.” Medium, Towards Data Science, 12 Sept. 2020,
towardsdatascience.com/decision-trees-for-classification-id3-algorithm-explained-89df76e72df1.
Xiaohu, Wang, et al. “An Application of Decision Tree Based on ID3.” Physics Procedia, Elsevier, 12 Apr. 2012,
www.sciencedirect.com/science/article/pii/S1875389212006098.