DERIVING TASK-ORIENTED

DECISION STRUCTURES

FROM DECISION RULES

Ibrahim M Fahmi Imam

MLI 95middot7

October 1995

DERIVING TASK-ORIENTED DECISION STRUCTURES FROM DECISION RULES

Adissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy at George Mason University

By

Ibrahim M Fahmi Imam

Director Professor Ryszard S Michalski

PRC Chaired Professor of Computer Science amp Systems Engineering School of Information Technology and Engineering

George Mason University

Fall Semester 1995 George Mason University Fairfax Virginia 22030

copy 1995 Copyright by Ibrahim F Imam All rights reserved

ACKNOWLEDGMENT

I would like to thank Professor Ryszard S Michalski PRC Chaired Professor of Computer

Science and Engineering and my Dissertation Director for his support encouragement and

guidance I would like to thank my committee members Professor Larry Kerschburg Chair of the

Department of Information Systems and Software Engineering Professor David Rine Professor

of Information Technology and Engineering and Professor David Schum Professor of Excellence

in Information Technology and Engineering for their encomagement and help with many aspects

of my PhD

I would like to thank Professor Tomaz Arciszewki System Engineering Department for providing

me with application problems Ronny Kohavi Stanford University for discussion and providing

me with some related work on learning decision structures and decisionmiddot graphs and Professor

George Tecuci Computer Science Department for pointing some related work

I would like to thank my colleagues Nabil Al-Kharouf for reviewing my dissertation Eric

Bloedorn for reviewing an earlier draft of my dissertation and for using his program AQ17-DCI in

my experiments Srinivas Gutta for providing some application for my PhD work Mike Heib for

reviewing an earlier draft of my dissertation and helping me find relevant articles Ken Kaufman

for reviewing an earlier draft of my thesis Mark Maloof for providing me with script files which

made it easier to iteratively run AQ1Sc Halah Vafaie for working with her on application and

comparison of different aspects of my work and Janusz Wnek for using his DIAV program for

explaining my results

I would like to thank Professor Andrew P Sage Dean of the School of Information Technology

and Engineering and Professor Kenneth Bumgarner Dean of Student Services and Associate ViCe

President of George Mason University for their support and Professor Murray W Black

Associate Dean of the School of Infonnation Technology and Engineering for guidance on

preparing the PhD proposal

I would like to thank conferences organizers who supported me to attend their conferences and

present parts of my PhD work The organizers include Professor Moonis Ali Professor Frank

Anger Professor Zbigniew Ras the American Association for Artificial Intelligence and others I

would like also to thank the organizing committee of the Florida Artificial Intelligence Research

Symposium FLAIRS-95 for organizing my first workshop Thanks goes to Dr Doug Dankel Dr

Howard Hamilton Dr John Stewman and Dr Dan Tamir

I would like also to thank the many individuals who helped me in any way during my PhD Those

include Dr Ashraf Abdel-Wahab Dr Jerzy Bala Dr Alex Brodsky Dr Richard Carver Dr

Kenneth De Jong John Doulamis Tom Dybala Dean Ibrahim Farag Dr Ophir Frieder Csilla

Frakes Dr Mohamed Habib Ali Hadjarian Dr Hugo De Garis Zenon Kulpa Dr Paul Lehner

Dr George Michaels Dr Eugene Norris Dean James Palmer Mitch Potter Dr Ahmed Rafea

Jim Ribeiro Jsyshree Sarma Dr Arun Sood Dr Clifton Sutton Bradley Utz Dr Tibor Vamos

Patricia Zahra Dr Shaker Zahra and Dr Jianping Zhang

~J I yarJ I JJ I ~

U-t-oJ ~II ~) ~ oWAIl J

Dedication

To my mother my brothers and my sister

TABLE OF CONTENTS

TI1LE Page

ABS1RAcr II 1

CHAPrER 1

IN1RODUCI10N II 3

11 Motivation and Overview 3

12 The Problem Statement 6

CHAPrER2

REIAfED RESEARCFI 7

21 Learning Decision Trees from Decision Diagrams 7

22 Learning Decision Trees from Examples 10

221 Building Decision Trees Using Information-based Criteria 11

222 Building Decision Trees Using Statistics-based Criteria 16

223 Analysis of Attribute Selection Criteria 18

23 Learning Decision Structures 19

CHAPrER3

DESCRIPTION OF TIlE APPROACFI 23

31 General Methodology 23

32 A Brief Description of the AQ15 and AQ17 RuleLeaming Systems 25

33 Generating Decision Structures From Decision Rules 28

331 The AQDT-2 attribute selection method 29

332 The AQDT-2 algoridlm 37

vi

333 An example illustrating the algorithm 42

34 Tailoring Decision Structure to a decision-making situation 47

341 Learning Cost-Dependent Decision Structures 49

342 Assigning Decision Under Insufficient Infonnation 49

343 Coping with noise in training data 50

35 Analysis of the AQDT-2 Attribute Selection Criteria 51

36 Decision Structures vs Decision Trees 53

CHAPTER 4

EMPRICAL ANALYSIS AND COMPARATIVE STUDY OF TIlE MElHOD 58

41 Description of the Experimental Analysis 59

42 Experiments With Average Size Complex and Noise-Free Problems

Wind Brac-ings 60

43 Experiments With Small Size Simple and Noise-Free Problems

e MONK-I 69

44 Experiments With Small Size Complex and Noise-Free Problems

MONK-2 76

45 Experiments With Small Size Simple and Noisy Problems

MONK-3 79

46 Experiments With Large Size Complex and Noise-Free Problems

Diagnosing Breast Cancer 83

47 Experiments With Large Size Complex and Noisy Problems

Musm()()m classifications 84

48 Experiments With Small Size Structured and Noise-Free Problems

East-West Trains 85

49 Experiments With Small Size Simple and Noisy Problems

vii

Congressional Voting Records (1984) 87

410 Analysis of the Results 88

ClIAPfER 5 CONCIUSIONS 95

51 Summary 95

52 Contributions 96

REFERENCES 98

VITA 102

viii

LIST OF TABLES

No TITLE Page

2-1 An example of a decision table 9

2-2 A set of training examples used to illustrate the C45 system 15

2-3 The frequency of different attributes values to different decision classes 17

2-4 The expected values of the frequency of examples in Table 2-3 17

2-5 Attribute selection criteria and their basic evaluation measure 17

2-6 The contingency tables of Mingers example 18

2-7 Mingers results for determining the goodness of split 19

2-8 Mingers results for comaring the total accuracy and size of decision trees

provided by different attribute selection criteria from four problems 19

2-9 Comparing the AQDT approach with EDAG and HOODG approaches 22

3-1 The available tools and the factors that affect the process of testing a software 43

3-2 Calculating the disjointness of each attribute 44

3-3 Evaluation of the AQDT-2 criteria on the Quinlans example in Table 2-2 51

3-4 The data used in Mingers f11st experiments 52

3-5 The performance of AQDT-2 criteria (compare with the other criteria in Table 2-6) 52

3-6 The possible ranking domains and using condition of AQDT-2 criteria 53

3-7 Comparison between Decision Structures and Decision Tree 54

4-1 Evaluation of AQDT-2 attribute selection criteria for the wind bracing problem 62

4-2 The predictive accuracy of AQ15c and AQDT-2 for the wind bracing problem 67

4-3 Evaluation of AQDT-2 attribute selection criteria for the MONK-l problem 71

4-4 The predictive accuracy of AQ15c and AQDT-2 for the MONK-l problem 73

ix

4-5 The predictive accuracy of AQ15c and AQDT-2 for the MONK-2 problem 77

4-6 The predictive accuracy of AQ15c and AQDT-2 for the MONK-3 problem 81

4-7 The set of attributes and their values used in the trains problem 86

4-8 A tabular snmmary of the predictive accuracy of decision trees obtained

by AQDT -2 and C45 for the congressional voting data 88

4-9 Summary of the best parameter settings for the first subfunction of the approach

with different data characteristics 89

4-10 Summary of the performance of AQDT-2 and C45 on different problems 90

x

LIST OF FIGURES

No TITLE Page

2-1 An example to illustrate how attributes break rules 8

2-2 A decision tree learned from the decision table in Table 2-1 10

2-3 A decision tree learned using the gain criterion for selecting attributes 15

2-4 Decision rules and their Exception Directed Acyclic Graph (EDAG) 21

3-1 Architecture of the AQDT approach 24

3-2 A ruleset generated by AQ15 for the concept Voting pattern of

Democratic Representatives 27

3-3 Ven Diagrams ofPossible combinations of attribute values in two decision classes 33

3-4 Decision trees corresponding the Ven diagrams in Figure 3-3 33

3-5 Decision trees showing the maximum number of none leaf nodes 41

3-6 Decision rules for selecting the best tool for testing a software 43

3-7 A decision structure learned for classifying software testing tools 45

3-8 A diagrammatic visualization of the decision rules and the derived decision tree 46

3-9 Decision trees learned ignoring the support metric and the type of the testing tool 47

3-10 A decision tree learned without the cost attribute 47

3-11 Decision structures learned by AQDT-2 using different criteria 55

3-12 The Imams example Example where learning decision structures (trees)

from rules is better than learning them from examples 56

3-13 An example where decision rules are simpler than decision trees 57

4-1 Design of a complete experiment 59

Xl

4-2 Decision rules determined by AQ15c from the wind bracing data 61

4-3 A decision tree learned by C45 for the wind bracing data 63

4-4 A decision structure learned from AQ15c wind bracing rules 64

4-5 A decision structure that does not contain attribute xl 64

4-6 A decision structure without Xl with candidate decisions assigned to leaves 65

4-7 A decision structure determined from rules in Figure 4-4 under the

assumption of 10 classification error in the training data 65

4-8 Diagramatic visualization of decision trees learned for different decision

making situations for the wind bracing data 66

4-9 The accuracy of AQDT-2 and AQ15c with different parameter settings

for the wind bracing PIoblem 68

4-10 Analyzing different parameter setting of AQDT-2 using the wind bracing data 69

4-16 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-20 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-11 A comparison between AQ15c and AQDT-2 against C45 on the wind bracing data 69

4-12 A visualization diagram of the MONK-l problem 70

4-13 Decision rules learned by AQ15c for the MONK-1 problem 71

4-14 The decision tree for the MONK-1 problem generated both by AQDT-l 72

4-15 Compact decision structures generated by AQDT-2 for the MONK-1 problem 72

for the MONK-l Problem 74

4-17 Analyzing different parameter setting of AQDT-2 with the MONK-l data 75

4-18 Comparing AQ15c and AQDT-2 against C45 using the MONK-1 problem 75

4-19 A visualization diagram of the MONK-2 problem 76

for the MONK-2 Problem 78

4-21 Analyzing different parameter setting of AQDT-2 with the MONK-2 data 79

xii

4-22 Comparing AQ15c andAQDT-2 against C45 using the MONK-2 problem 79

4-24 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-35 A visualization diagram of a decision tree learned by AQDT -2 after reducing

4-23 A visualization diagram of the MONK-3 problem 80

for the MONK-3 Problem 82

4-25 Analyzing different parameter setting of AQDT-2 with the MONK-3 data 82

4-26 Comparing AQ15c and AQDT-2 against C45 using the MONK-3 problem 83

4-27 Comparing AQ15c and AQDT-2 against C45 using the breast cancer problem 84

4-28 Comparing AQ15c and AQDT-2 against C45 using the mushroom problem 85

4-29 Decision structures learned by AQDT-2 for different decision-making situations 87

4-30 Comparing decision trees for the Cong Voting-84 data learned by C45 amp AQDT -2 88

4-31 A visualization diagram of decision rules learned by AQ15c for the MONK-2 91

4-32 A visualization diagram shows the testing errors for the rules in Figure 4-31 92

4-33 A visualization diagram of the decision tree learned by AQDT-2 for the MONK-2 93

4-34 A visualization diagram shows testing errors of AQDT-2 decision tree bull 94

the generalization degree to 1 94

DERIVING TASKmiddotORIENTED DECISION STRUCTURES

FROM DECISION RULES

Ibrahim M Fahmi Imam PhD

School of Information Technology and Engineering

Geotge Mason University Fall 1995

Ryszard S Michalski Advisor

ABSTRACT

This dissenation is concerned with research on learning task-oriented decision structures from

decision rules The philosophy behind this research is that it is more appropriate to learn

knowledge and store it in a declarative form and then when a decision making situation occurs

generate from this knowledge the decision structure that is most suitable for the given decision

making situation Learning decision structures from decision rules was first introduced by

Michalski (1978) The flIst implementation of this approach was done by Imam and Michalski

(1993 ab) called AQDT-l

This approach separates the function of generating a knowledge-base from the function of using

the knowledge-base for decision-making The first function focuses on learning accurate

consistent and complete concept description expressed in a declarative form The second function

is performed whenever a new decision-making situation occurrs a task-oriented decision structure

is obtained to suit that situation Thsk-oriented knowledge is defined as knowledge that is adapted

for solving a given decision-making situation (Imam amp Michalski 1994 Michalski amp Imam

1994)

The dissertation introduces the system AQJJT-2 for learning task -oriented decision structures from

decision rules or examples Each decision making situation is defined by a set of parameters that

controls the learning process of the AQDT-2 system The extensive experiments on AQJJT-2 show

that decision structures learned by it usually outperform in terms of accuracy and average size of

the decision structures those learned from examples by other well known systems The results

show also that the system does not work very well with noisy data The system is illustrated and

compared using applications of artificial problems such as the three MONKs problems (Thrun

Mitchell amp Cheng 1991) and the East-West 1iain problem (Michie et al 1994) It was also

applied to real world problems of learning decision structures in the areas of construction

engineering (for determining the best wind bracing design for tall buildings) medical

diagnosis(for learning decision rules for recognizing breast cancer) agriculture diagnosis (for

learning classification rules for distinguishing between poisonous and non-poisonous

mushrooms) and political data (for characterizing democratic and republican voting records)

CHAPTER 1 INTRODUCTION

11 Motivation and Overview

Learning and discovery systems should be able not only to generate and store knowledge but also

to use this knowledge for decision-making The main step in the development of systems for

decision-making is the creation of a knowledge structure that characterizes the decision-making

process The form in which knowledge can be easily obtained may however differ from the form

in which it is most readily used for decision-making It is therefore important to identify the form

of knowledge representation that is most appropIiate for learning (eg due to ease of its

modification) and the form that is most convenient for decision making

A simple and effective tool for describing decision processes is a decision structure which is a

directed acyclic graph that specifies an order of tests to be appliedto an object (or a situation) to

arrive at a decision about that object The nodes of the structure are assigned individual tests

(which may correspond to a single attribute a function of attributes or a relation) the branches are

assigned possible test outcomes or ranges of outcomes and the leaves are assigned a specific

decision a set of candidate decisions with corresponding probabilities or an undetermined

decision A decision structure reduces to a familiar decision tree when each node is assigned a

single attribute and has at most one parent when the branches from each node are assigned single

values of that attribute and when leaves are assigned single definite decisions Thus the problem

of generating a decision structure is a generalization of the problem of generating a decision tree

Decision trees are typically generated from a set of examples of decisions The essential

characteristic of any such method is the attribute selection criterion used for choosing attributes to

be assigned to the nodes of the decision tree being built Such criteria include the entropy

reduction the gain and the gain ratio (Quinlan 1979 83 86) the gini index ofdiversity (Breiman

et al 1984) and others (Cestnik amp Bratko 1991 Cestnik amp Karalic 1991 Mingers 1989a)

3

4

Adecision treedecision structure representations can be an effective tool for describing a decision

process as long as all the required tests can be performed and the decision-making situations it

was designed for remain constant (eg in a doctor-patient example the doctor should detennine

that answers for all symptoms appear in the decision tree) Problems arise when these assumptions

do not hold For example in some situations measuring certain attributes may be difficult or costly

(eg in the doctor-patient example a brain or blood test is needed which is very expensive or the

tools needed are not available) In such situations it is desirable to refonnulate the decision

structure so that the inexpensive attributes are evaluated frrst (assigned to the nodes close to the

root) and the expensive attributes are evaluated only if necessary (by assignment to the nodes far

away from the root) If an attribute cannot be measured at all it is useful to either modify the

structure so that it does not contain that attribute or-when this is impossible-to indicate

alternative candidate decisions and their probabilities A restructuring is also desirable if there is a

significant change in the frequency of occurrence of different decisions (eg in the doctor-patient

example the doctor may request a decision structure expressed in a specific set of symptoms

biased to classify one or more diseases or specify a certain order of testing)

Arestructuring ofa decision structure (or a tree) in order to suit new requirements could be quite

difficult This is because a decision structure is a procedural representation that imposes an

evaluation order on the tests In contrast no evaluation order is imposed by a declarative

representation such as a set of decision rules Tests (conditions) of rules can be evaluated in any

order Thus for a given set of rules one can usually build a huge number of logically equivalent

decision structures (trees) which differ in the test ordering Due to the lack of order constraints

a declarative representation (rules) is much easier to modify to adapt to different situations than a

procedural one (a decision structure or a tree) On the other hand to apply decision rules to make a

decision one needs to decide in which order tests are evaluated and thus needs a decision

structure

An attractive solution to these opposite requirements is to acquire and store knowledge in a

declarative form and transform it to a decision structure when it is needed for decision-making

5

This method allows one to create a decision structure that is most appropriate in a given decisionshy

making situation Because the number of decision rules per decision class is usually small (each

rule is a generalization of a set of examples) generating a decision structure from decision rules

can potentially be perfonned much faster than by generation from training examples Thus this

process could be done on line without any delay noticeable to the user Such virtual decision

structures are easy to tailor to any given decision-making situation

This approach allows one to generate a decision structure that avoids or delays evaluating an

attribute that is difficult to measure in some decision-making situation or that fits well a particular

frequency distribution of decision classes In other situations it may be unnecessary to generate

complete decision structure but it may be sufficient to generate only a part of it which concerns

only with decision classes of interest Thus such an approach has many potential advantages

This dissertation presents a new system called AQflf-2 The AWf-2 system generates a taskshy

oriented decision structure (decision structure that is adapted to the given decision-making

situation) from decision rules The decision rules are learned by either rule leaming system AQ15

(Michalski et al 1986) or system AQ17-DCI which has extensive constructive induction

capabilities (Bloedorn et al 1993)

To associate the decision rules with a given decision-making task AQflf-2 provides a set of

features including 1) enabling the system to include in the decision structure nodes corresponding

to new attributes constructed during the process of learning the decision rules 2) controlling the

degree of generalization needed during the development of the decision structure 3) providing four

new criteria for selecting an attribute to be a node in the decision structure that allow the system to

generate many different but equivalent decision structures from the same set of rules 4) generating

unknown nodes in situations when there is insufficient infonnation for generating a complete

decision structure 5) learning decision structures from discriminant rules as well as

characteristic rules and 6) providing the most likely decision when the decision process stops

due to inability to evaluate an attribute associated with an intennediate node

6

To test the methooology of generating decision structures from decision rules an extensive set of

planned experiments have been designed to test different aspects of the approach The experiments

include testing different combinations of parameters for each sub-function of the approach

analyzing the relatioship between decision rules and the decision structure learned from them and

comparing decision trees learned by the AQUf2 system the well known C45 (Quinlan 1993)

system for learning decision trees from examples Different experiments were designed to examine

the new features of the AQUf approach for learning task-oriented decision structures The

experiments were applied to artificial domains as well as real-world domains including MONK-I

MONK-2 and MONK-3 (Thrun Mitchell amp Cheng 1991) East-West trains (Michie et al

1994) Engineering Design-wind bracings (Arciszewski et al 1992) Mushrooms Breast

Cancer (Mangasarian amp WolbeJg 1990) and Congressional bting Records of 1984 The

MONKs problems are concerned with learning classification rules for robot-like figures MONK-l

requires learning a DNF-type description MONK-2 requires learning a non-DNF-type description

(one that cannot be easily described as a DNF rule using the original attributes) MONK-3 concerns

learning a DNF rule from noisy data The East-West trains dataset is a structural domain that

classifies two sets of trains (Eastbound and Westbound) The Engineering Design-wind

bracing data involves learning conditions for applying different types of wind bracing for ta11

buildings The Mushrooms data is concerned with learning classification rules for distinguishing

between poisonous and non-poisonous mushrooms The Breast Cancer data involves learning

concept descriptions for recognizing breast cancer The congressional voting data includes voting

records on different issues AQUf2 outperformed C45 on the average with respect to both the

predictive accuracy and tree size for most problems Apf-2 did not work very well with noisy

data or with problems that have many rules covering very few examples

12 The Problem Statement

There are many limitations and problems accompanied using decision trees for decision-making

CHAPTER 2 RELATED RESEARCH

21 Learning Decision Trees from Decision Diagrams

The AQDT-2 method proposed here is based on the earlier work by Michalski (1978) which

introduced an algorithm for generating decision trees from decision lists The method proposed

several attribute selection criteria These criteria are of increasing power of the main criterion

order cost estimate (the nth order cost estimates n=l 2 ) Michalski also analyzed two

specific criteria MAL and DMAL for selecting the optimal attribute for a node in the tree

based on properties extracted from the decision diagram In order to better explain the method

it is necessary to define some terms These terms will be used later in the dissertation

Definition 2-1 A ~ of a decision class C is a set of rules whose union includes the set of all

examples of class C and does not include any examples of other decision classes

Definition 2-2 A ~ of a decision class C is disjoint if all rules (rules) are pairwise logically

disjoint In other words for any two rules there exists a condition with the same attribute but

with different values in each rule

Definition 2-3 An minimal cover is a disjoint cover which has the smallest number of rules

among all possible covers

Definition 2-4 A diagram for a given cover is a table constructed graphically by representing

in two dimension space of all possible combination of attributes values and locating on the

diagram all the condition parts of the given rules and marking them with the action specified

with each rule

Michalskis algorithm requires construction of minimalmiddot cover The minimal cover should be

consistent and complete The method is based on the fact that if there are n decision classes

any decision tree that correctly classifies the given rules and has n distinct leaves is a minimal

7

8

decision tree (any consistent decision tree should have at least n leaves) Michalski (1978) has

shown that if only one rule is broken by a selected attribute then instead of having one leaf

(which could potentially represent this rule or the decision class in the tree) there will have to

be at least two leaves representing this rule in the fmal decision tree

The attribute selection criterion MAL introduced in (Michalski 1978) prefers attributes that do

not break any rules or break as few as possible An attribute breaks a rule if the attribute can

divide the rule into two or more sub-rules Figure 2-1 shows two examples of two sets of rules

In the fJIst xl and x3 break at least two rules each (xl breaks the rule [x4=2] amp [x1=lv3] amp

[x3=2v3] and the rule [x4=3] amp [x1=1v2] amp [x3=1] and x3 breaks three rules the rule [x4=2]

amp [xl=2] amp [x3=2v3]the rule [x4=l] amp [xl=3] amp [x3=lv3] and the rule [x4=3] amp [xl=4] amp

[x3=2v3]) In the diagram on the right xl is the only attribute that does not break any rule

Figure 2-1 An example to illustrate how attributes break rules

One of the criteria defmed by Michalski is the first degree cost estimate which assigns to each

attribute an integer equal to the number of rules broken by that attribute This criterion is also

called the static cost estimate of an attribute or the criterion of minimizing added leaves

(MAL)

-

C3

9

The MAL criterion (Minimizing Added Leaves) seeks an attribute that minimizes the

estimated number of additional nodes in the decision tree being generated over a hypothetical

minimal decision tree When there is a tie between two attributes the attribute to be selected is

the one which breaks smaller rules (rules that cover fewer examples or more specialized

rules) AQDT-2 uses an approximate version of this criterion (the attribute dominance)

Another criterion introduced by Michalski was the DMAL criterion The DMAL criterion

(Dynamically Minimizing Added Leaves) is based on a principle similar to that of MAL but

is more complex because once an attribute is selected as a node in the tree some rules andor

parts of the broken rules at each branch are merged into one rule The DMAL ensures that the

value of the total cost estimate of an attribute is decreased by a value equal to the number of

merged rules minus one

Example Learn a decision tree from the following decision table

The minimal cover consists of the following rules

Al lt [x2=O] v [x1=0][x2=2] A2 lt [x2=I] v [xl=2][x2=2] A3 lt [xl=I][x2=2]

The evaluations ofthe MAL criterion for these attributes are 2 for xl 0 for x2 5 for x3 and 5

for x4 The attribute xl divides the two rules [x2=O] and [x2=I] so selecting it will add two

leaves to the optimal number of leaves It is clear that the attribute to be selected as a root of

10

the decision tree is x2 Then three branches are attached to the root node and the decision rules

are divided into subsets each corresponding to one branch For x2 = 0 or 1 a leave node is

generated For x2=2 another attribute is selected to be a node in the tree In this case xl has the

minimum MAL value Figure 2-2 shows the decision tree obtained using the MAL criterion

Al A3 A2

Figure 2-2 A decision tree learned from the decision table in Table 2-1

22 Learning Decision Trees from Examples

Decision tree learning is a field that concerned with generating decision tree that classifies a set

of examples according to the decision classes they belong to The essential aspect of any

inductive decision tree method is the attribute selection criterion The attribute selection

criterion measures how good the attributes are for discriminating among the given set of

decision classes The best attribute according to the selection criterion is chosen to be assigned

to a node in the tree The fIrst algorithm for generating decision trees from examples was

proposed by Hunt Marin and Stone (1966) Hunts algorithm uses a divide and conquer

algorithm for building decision trees This algorithm has been subsequently modified by

Quinlan (1979) and applied by many researchers to a variety of learning problems

Attribute selection criteria can be divided into three categories These categories are logicshy

based information-based and statistics-based The logic-based criteria for selecting attributes

use logical relationships between the attributes and the decision classes to determine the best

attribute to be a node in the decision tree such as the MAL criterion minimizing added leaves

(Michalski 1978) which uses conjunction and disjunction operators The information-based

criteria are based on the information theory These criteria measure the information conveyed

11

by dividing the training examples into subsets Examples of such criteria include the

information measure 1M the entropy reduction measure and the gain criteria (Quinlan 1979

83) the gini index of diversity (Breiman et al 1984) Gain-ratio measure (Quinlan 1986) and

others (Clark amp Niblett 1987 Bratko amp Lavrac 1987 Cestnik amp Karalic 1991) The

statistics-based criteria measure the correlation between the decision classes and the attributes

These criteria use statistical distributions for determining whether or not there is a correlation

The attribute with the highest correlation is selected to be a node in the tree Examples of

statistics-based criteria include Chi-square and G-statistic (Sokal amp Rohlf 1981 Han 1984

Mingers 1989a)

Niblett and Bratko (1986) Quinlan (1987) and Bratko and Kononeko (1987) extended the

method of learning decision trees to also handle data with noise (by pruning) Handling noise

extended the process of learning decision trees to include the creation of an initial complete

decision tree tree pruning which is done by removing subtrees with small statistical Validity

and replacing them by leaf nodes (MiDgers 1989b) More recently pruning has also been used

for simplifying decision trees even for problems without noise (Bohanec amp Bratko 1994)

Pruning decision trees improves their simplicity but reduces their predictive accuracy on the

training examples Quinlan (1990) proposed also a method to handle the unknown attributeshy

value problem by exploring probabilities of an example belonging to different classes

The rest of this section includes a brief description of the attribute selection criterion used by

C45 learning system (Quinlan 1993) C45 uses an information-based criterion for selecting

an attribute to be a node in the tree Also the section includes a brief description of the Chishy

square method for attribute selection (Mingers 1989a) The method is a statistics-based

method for selecting an attribute to be a node in the tree

221 Building Decision Trees Using htformation-based Criteria

12

This section presents a description of the inductive decision tree learning system C4S The

C4S learning system is considered to be one of the most stable accurate and fastest program

for learning decision trees from examples

Learning decision trees from examples requires a collection of examples Each example is

represented by a fixed number of attribute-value pairs C45 (Quinlan 1993) is a learning

program that induces classification decision trees from a set of given examples The C4S

learning system is descended from the learning system ID3 (Quinlan 1979) which is based on

Hunts method for constructing decision tree from a set ofcases (Hunt Marin amp Stone 1966)

The C45 system uses an attribute selection criterion called the Gain Ratio This criterion

calculates the gain in classifying information based on the residual information needed to

classify cases in a set of training examples and the information yielded by the test based on the

relative frequencies of the possible outcomes (decision classes) The gain ratio criterion is

based on an earlier criterion used by ID3 called the Gain Criterion The Gain Criterion uses

the frequency of each decision class in the given set of training examples

Once an attribute is chosen to be a node in the tree the system generates as many links as the

number of its values and classifies the set of examples based on these values If all the

examples at a certain node belong to one decision class the system generates a leaf node and

assigns it to that class Otherwise the system searches for another attribute to be a node in the

tree

The Gain Criterion The gain criterion is based on the information theory That is the

information conveyed by a message depends on its probability and can be measured in bits as

minus the logarithm (base 2) of that probability To explain the gain criterion suppose that for

a given problem Xu bullbull X are given attributes and Cu c are decision classes Suppose S is

any set of cases and T is the initial set of training cases The frequency of class C1 in the set S

is the number of examples in S that belong to class C i bull

13

freq(Cit S) = Number of examples in S belong to CI (2-1)

Suppose that lSI is the total number of examples in S the probability that an example selected

at random from S belongs to class Ci is freq(CIbullS) IISI

The information conveyed by the message that a selected example belongs to a given decision

class Cit is determined by -log1 (freq(Ch S) IISI) bits

The expected information from such a message stating class membership is given by

info(S) = -k

L (freq(C S) I lSI) log1 (freq(C S) IISI) bits (2-2)I

info(S) is known also as the entrQpy of the set S When S is the initial set of training examples

info(T) determines the average amount of information needed to identify the class of an

example in T

Suppose that we selected an attribute X to be the root of the tree and suppose that X has k

possible values The training set T will be divided into k subsets each corresponding to one of

Xs values The expected information of selecting X to partition the training set T infoz(T) can

be found as the sum over all subsets of multiplying the information conveyed by each subset

by its probability

k

infoz(T) =L (ITII 1m) infO(TIraquo (2-3) I

The information gained by partitioning the training examples T into subset using the attribute

X is given by

gain(X) =inlo(T) - inoIT) (2-4)

The attribute to be selected is the attribute with maximum gain value

The Gain Ratio Criterion This criterion indicates the proportion of information generated by

the split that appears helpful for classification Quinlan (1993) pointed out that the gain

criterion has a serious deficiency Basically it is strongly biased toward attributes with many

outcomes (values) For example for any data that contains attributes such as social security

14

number the gain criterion will select that attribute to be the root of the decision tree However

selecting such attributes increases the size of the decision tree Quinlan provided a solution to

this problem by introducing the gain ratio criterion which takes the ratio of the information that

is gained by partitioning the initial set of examples T by the attribute X to the potential

information generated by dividing T into n subsets

Following similar steps to obtain the information conveyed by dividing T into n subsets The

expected information generated by dividing T into n subsets and by analogy to equation 2-2 is

determined by

split info(T) = - Lt

(ITII lTI) 10gl (ITII m ) (2-5) I

The gain ratio is given by

gain ratio(X) = gain(X) I split info(X) (2-6)

and it expresses the proportion of information generated by the split that is useful for

classification

Example Consider the following example presented by Quinlan (1993) Table 2-2 shows the

set of training examples

First determine the amount of information gained by selecting the attribute outlook to be a

root of the decision tree This attribute divides the training examples into three subsets

sunny-with five examples two of which belong to the class Play overcast-with four

examples all of which belong to the class Play and rain-with five examples three of

which belong to the class Play To determine info(11 the average information needed to

identify the class of an example in T there are 14 training examples and two decision classes

Nine of these examples belong to the class Play and five belong to the class Dont Play

Info(11 = - 914 10gl (914) - 514 10gl (514) = 094 bits

When using outlook to divide the training examples the information becomes

15

infO(T) = 514 -25 log2 (25) - 35 10g2 (35)

+ 414 -44 10g2 (44) - 04 log2 (04)

+ 514 -35 logl (35) - 25 logl (25) = 0694 bits

By substituting in equation 2-4 the gain of information results from using the attribute

outlook to split the training examples equal to 0246 The gain information for windy is

0048

Figure 2-3 shows a decision tree learned for this problem using the gain criterion The split

information for outlook is determined as follows

split info(T) = - 514 logl (514) - 414 log2 (414) - 514 log2 (514) = 1577 bits

The gain ratio for outlook = 02461577 =0156

16

Figure 2 -3 A decision tree learned using the gain criterion for selecting attributes

The C45 system handles discrete values as well as continuous values To handle an attribute

with continuous values C45 uses a threshold to transform the continuous domain into two

intervals In other words for each continuous attribute C45 generates two branches one

where the values of that attribute is greater than the determined threshold and the other if the

value is less than or equal to the threshold

Tree pruning in C45 is a process of replacing subtrees with small classification validity by

leaves The C45 system uses Laplace ratio for determining the error rate of different subtrees

This ratio is defined as (e+l) (n+2) where n is the number of the training examples and e is

the number of misclassified examples at a given leaf

222 Building Decision Trees Using Statistics-based Criteria

The Chi-square method for selecting attributes was used by Hart (1984) and Mingers (1989a)

in building decisionmiddot trees The method uses Chi-square statistics to measure the association

between two attributes When building decision trees the method is implemented such that it

determines the association between each attribute and the decision classes The attribute to be

selected is the one with greatest value

To determine the Chi-square value for an attribute consider aij is the number of examples in

class number i where the attribute A takes value numberj In other words aij is the frequency

of the combination of decision class number i and the attribute value number j The Chi-square

value for attribute A is given by

n m

Chi-square (A) =L L [ (aij - Eijl2 Eij ] (2shy

i=1 j=1

where n is the number of decision classes m is the number ofvalues of a given attribute Also

17

Eij = (TCi TVj ) T (2shy

8)

where T Ci and T Vj are the total number of examples belong to the decision class a and the total

number of examples where the attribute A takes value vj respectively T is the total number of

examples

Consider Quinlans example in Table 2-2 Table 2-3 shows the frequency of different

combination of values between the decision class and both the outlook and the Windy

attributes Table 2-4 shows the expected values TCi and T Vj of the frequencies in Table 2-3

of different attributes values for different decision classes

To determine the association value between the decision classes and both the attribute Windy

and the attribute Outlook the observed Chi-square values are

Chi-square (Windy Class) =[(3-39)239] + [(3-21)221] + [(6-51)232] + [(2-29)232]

=021 + 039 + 025 + 025 = 11

Chi-square (Outlook Class) = [(2-32)232] + [(4-26)226] + [(3-32)232] + [(3-18118]

+ [(0-14)214] + [(2-18)2 18] =045 + 075 + 02 + 08 + 14 + 002 = 362

18

Applying the same method to the other attributes the results will favor the attribute Outlook

Once that attribute is selected to be a node in the tree the remaining set of examples are divided

into subsets and the same process is repeated on each subset

Table 2-5 shows a summary of these criteria and their basic evaluation function

Table 2-5 Attribute selection criteria and their basic evaluation measure

Info Measure (IM) Gain Gshystatistics and Gain Ratio Chi-square

II

Entropy(S) = - L (freq(~ 5) 1151) logz (freq(CIt 5) 1151 ) I

G-Statistics =2N 1M (N-No of examples) n m

Chi-square (A B) = L L [(aij - ~j)21Eij ] i=l

223 Analysis of Attribute Selection Criteria

This subsection introduces briefly the analysis of different selection criteria which was done by

Mingers (1989a) Mingers compared six attribute selection criteria that are used in decision

tree programs These criteria are the Information Measure (IM) Chi-square G statistics Gin

index of diversity Marshal correction and Gain RaJio The overall results show that the Gain

Ratio criterion had the strongest xesults

In the flISt experiment Minger tested the six criteria on a problem with ambiguous examples

(ie examples may belong to more than one decision class) to observe how the selected criteria

evaluate the given attributes The problem has two decision classes and two attributes X and

Y It was assumed that attribute X is better for classifying the examples than attribute Y The

training examples were unevenly spread between the two values of X Attribute Y has three

values and the examples were spread randomly among them Table 2-6 (a and b) shows the

contingency tables for both attributes Table 2-7 shows a summary of the goodness of split

provided by the six criteria Mingers noted that the measures that are not based on information

19

theory give radiation (attribute X here) less weighL This may be because the zero in the fIrst

row of radiation has a greater influence in the log calculation In the case of using the Chishy

square criterion the value zero adds the maximum association between any two attributes

because the Chi-square value of a zero cell is the expected value of this cell

b)

Now let us demonstrate results from another experiment done by Mingers In this experiment

Mingers used four different data sets to generate decision trees for eleven different criteria In

the final results he compared the total number of nodes and the total error rate provided by

each criterion over all given problems Table 2-8 shows the final results for five selected

criteria only

20

U lgt ~middot results for comparing the total accuracy and size of decision trees different attribute selection criteria from four VVLU

This experiment was performed on four real world data sets These data are concerned with

profiles of BA Business Studies degree students recurrence of breast cancer classifying types

of Iris and recognizing LCD display digits The data was divided randomly 70 for training

and 30 for testing For more details see Mingers (1989a)

23 Learning Decision Structures

Considering the proposed defInition of decision structures given above two related lines of

research are described in this section Gaines (1994) and Kohavi (1994) proposed two

approaches for generating decision structures that share some of the earlier ideas by Imam and

Michalski (1993b)

In the first approach Brian Gaines introduces a method for transforming decision rules or

decision trees into exceptional decision structures The method builds an Exception Directed

Acyclic Graph (EDAG) from a set of rules or decision trees The method starts by assigning

either a rule say RO or a conclusion say CO or both to the root node If it assigns a conclusion

to the root node it places it on a temporary conclusion list Then it generates a new child node

and add to it a new rule say Rl The method evaluates the new rule Rl if it satisfies the

conclusion CO in its parent node it builds a new child and repeats the processes Otherwise if

the rule Rl does not satisfy the conclusion CO it changes the temporary memory with the new

conclusion Cl which is satisfied by the rule RO The same process is repeated until all rules

that have common conditions with the rule at the root are evaluated The method then creates a

21

new child node from the root and repeat the process until all rules are evaluated In the decision

structure nodes containing rules only represent common conditions to all of its children

The main disadvantages of this approach is that it requires discriminant rules to build such a

decision structure Also such a structure is more complex than the traditional decision trees

that are used for decision-making

Figure 2-4 shows an example of set of rules and their equivalent exceptional decision structure

The decision structure can be read as follows It is Safe except if xl=1 amp x2=1 amp x3=1 and

either x4=3 or x5=I then it is Lost except if x6=1 it is Safe except if x7=1 it is Lost

The second approach introduced by Ronny Kohavi learns decision structures from examples

using a bottom-up algorithm The method uses a greedy Hill-climbing algorithm for inducing

Oblivious read-Once Decision Graphs (HOODO) A read-once decision graph is defined as a

decision graph where each attribute occurs at most once along any computational path In other

words for each path from the root to the leaves of the decision structure attributes may occur

as a node once at most However there may be more than one node with the same attribute in

the decision graph Kohavi also defined a leveled decision graph in which the nodes are

partitioned into a sequence of pairwise disjoint sets such that outgoing edges from each level

terminate at the next level An oblivious decision graph is a decision graph where all nodes at

a given level are labeled by the same attribute

22

Safe lt [xl=2] Safe lt [x2=2] Safe lt [x3=2] Safe lt [x4=1] amp [x5=2] Safe lt [x4=1] amp [x5=3] Safe lt [x6=l] amp [x7=2] Safe lt [x6=1] amp [x7=3] Safe lt [x4=2] amp [x5=2] Safe lt [x4=2] amp [x5=3]

Lost lt [x6=2] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x6=3] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x5=1] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=2] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=3] amp [x3=1] amp [x2=I] amp [xl=l]

The algorithm starts by generating a leaf node for each decision class Then it applies a

nondeterministic method to select an attribute to build new level of the decision structure This

attribute is removed from the data and the data is divided into subsets each corresponding to a

complex combination of that attributes values For each subset the process is repeated until all

the examples of a given subset belong to one decision class For example if the selected

attribute say A has two values (0 and 1) and there are two decision classes (CO and CI) The

data is divided into two subsets the fIrst subset contains examples where A takes value 0 and

belong to class CO or takes value 1 and belong to class Cl The second subset of examples is

the set where A takes value 0 and belong to class CI or takes value 1 and belong to class CO

The number of nodes of the fIrst level (after the leave nodes) is expected to be less than or

equal to ka where k is the number of decision classes and n is the number of values of the

selected attribute and it can increase exponentially before that number is reduced

exponentially to one

It is easy for the reader to fIgure out some major disadvantages of such approach including

The average size of such decision structures is estimated to be very large especially when there

23

is no similarity (ie strong patterns) or logical relationship in the data The time used to learn

such decision structure is relatively very high comparing to systems for learning decision trees

from examples and fmally it could be better to search for attribute which reduces the number

of generated subsets of the data instead of nondeterministically selecting an attribute to build a

new level of the decision structure Kohavi provided a comparison between the C45 system

and his system that can be found in (Kohavi 1994) Later Kohavi and Li (1995) used a

deterministic method for attribute selection which minimizes the width of the penultimate level

of the graph

Table 2-9 shows a comparison between the proposed approach and those two approaches The

EDAG and HOODG systems are unreleased prototype systems

or Decision structures are easy to Decision structures are Decision structures are easy to understand difficult to read understand

CHAPTER 3 DESCRIPTION OF THE APPROACH

31 General Methodology

In the proposed approach the function of learning or discovery is separated from the function of

using the discovered knowledge for decision-making The frrst function is performed by an

inductive learning program that searches for knowledge relevant to a given class of decisions

and stores the learned knowledge in the form of decision rules The second function is performed

when there is a need for assigning a decision to new data points in the database (eg bull a

classification decision) by a program that transforms obtained knowledge into a decision

structure optimized according to the given decision-making situation

The Learning Task GivenA set of training examples describing the concept to be learned Learning goal which specifies the decision classes to be learned from the training examples A background knowledge to control the learning process Determine A concept description in a declarative form of knowledge (decision rules) that satisfies the learning goal

The Decision-making Task GivenA set of decision rules in a conjunctive form A description of the new decision-making situation (eg attributes costs and order preference importance or frequency of decision classes etc) One or more examples that needed to be tested under the given decisionshymaking situation A set of parameters to control the learning process Determine A decision structure that suits the given decision-making situation

The decision rules used here are learned by either the AQ1S (Michalski et al 1986) or AQ17

(Bloedorn et aI 1993) learning systems The reasons for selecting decision rules as a form of

23

24

declarative knowledge are that they do not impose any order on the evaluation of the attributes

and due to the lack of order constraints decision rules can be evaluated in many different

ways which increases the flexibility of adapting them to the different tasks of decision-making

(Bergadano et al 1990) Since the number of rules learned per class is typically much smaller

than the number of examples per class generating a decision structure from decision rules can

potentially be done on line

Such virtual decision structures can be tailored to any given decision-making situation The

needed decision rules have to be generated only once and then they can be used many times for

generating decision structures according to changing requirements of decision-making tasks The

method uses the AQDT-2 system (Imam amp Michalski 1994) for learning decision structures

from decision rules Decision structures represent a procedural form of knowledge which makes

them easy to implement but also harder to change Consequently decision structures can be quite

effective and useful as long as they are used in decision-making situations for which they are

optimized and the attributes specified by the decision structure can be measured without much

cost Figure 3-1 shows an architecture of the proposed methodology

Data

Decision -

- Learning knowledge from database The decision making process

Figure 3 -1 Architecture of the AQDT approach

25

It is assumed that the database is not static but is regularly updated A decision-making problem

arises when there is a case or a set of cases to which the system has to assign a decision based on

the knowledge discovered Each decision-making situation is defmed by a set of attribute-values

Some attribute-values may be missing or unknown A new decision structure is obtained such

that it suits the given decision-making problem The learned decision structure associates the

new set of cases with the proper decisions

32 A Brief Description of the AQIS and AQ17 Rule Learning Programs

The decision rules are generated from examples by an AQ-type inductive learning system

specifically by AQ15 (Michalski et albull 1986) or AQI7-DCI (Bloedorn at al 1993) The rest of

this subsection includes a brief description of the inductive learning systems AQ15 and AQI7

AQ15 learns decision rules for a given set of decision classes from examples of decisions using

the STAR methodology (Michalski 1983) The simplest algorithm based on this methodology

called AQ starts with a seed example of a given decision class and generates a set of the

most general conjunctive descriptions of the seed (alternative decision rules for the seed

example) Such a set is called the star of the seed example The algorithm selects from the star

a description that optimizes a criterion reflecting the needs of the problem domain If the

criterion is not defined the program uses a default criterion that selects the description that

covers the largest number of positive examples (to minimize the total number of rules needed)

and with the second priority that involves the smallest number of attributes (to minimize the

number of attributes needed for arriving at a decision)

If the selected description does not cover all examples of a given decision class a new seed is

selected from uncovered examples and the process continues until a complete class description

is generated The algorithm can work with few examples or with many examples and can

optimize the description according to a variety of easily-modifiable hypothesis quality criteria

26

The learned descriptions are represented in the form of a set of decision rules expressed in an

attributional logic calculus called variable-valued logic 1 or VLl (Michalski 1973) A

distinctive feature of this representation is that it employs in addition to standard logic operators

the internal disjunction operator (a disjunction of values of the same attribute in a condition) and

the range operator (to express conditions involving a range of discrete or continuous values)

These operators help to simplify rules involving multi valued discrete attributes the second

operator is also used for creating logical expressions involving continuous attributes

AQ15 can generate decision rules that represent either characteristic or discriminant concept

descriptions depending on the settings of its parameters (Michalski 1983) A characteristic

description states properties that are true for all objects in the concept The simplest

characteristic concept description is in the form of a single conjunctive rule (in general it can be

a set of such rules) The most desirable is the maximal characteristic description that is a rule

with the longest condition part ie stating as many common properties of objects of the given

class as can be determined A discriminant description states properties that discriminate a given

concept from a fixed set of other concepts The most desirable is the minimal discriminant

descriptions that is a rule with the slwnest condition part For example to distinguish a given

set of tables from a set of chairs one may only need to indicate that tables have large flat top

A characteristic description of the tables would include also properties such as have four legs

have no back have four corners etc Discriminant descriptions are usually much shorter than

characteristic descriptions

Another option provided in AQ15 controls the relationship among the generated descriptions

(rulesets or covers) of different decision classes In the IC (Intersecting Covers) mode

rulesets of different classes may logically intersect over areas of the description space in which

there are no training examples In the DC (Disjoint Covers) mode descriptions of different

classes are logically disjoint The DC mode descriptions are usually more complex both in the

number of rules and the number of conditions There is also a DL mode (a Decision List mode

27

also called VL modeM-for variable-valued logic mode) in which the program generates rule sets

that are linearly ordered To assign a decision to an example using such rulesets the program

evaluates them in order If ruleset is satisfied by the example then the decision is made

otherwise the program proceeds to the evaluation of the ruleset +1 In IC and DC modes

rule sets can be evaluated in any order

Alternatively the system can use rules from the AQ17-DCI program for Data-driven

Constructive Induction AQ17-DCI differs from AQ15 mainly in that it contains a module for

generating additional attributes These attributes are various logical or mathematical

combinations the original attributes The program generates a large number of potential new

attributes and seleets from them those most promising based on an attribute quality criterion

To illustrate the format of rules generated by AQ15 (or AQ17-DCI) an exemplary ruleset is

shown in Figure 3-2 The ruleset (that can be re-represented as a disjunctive normal form

expression) describes a voting record of Democratic Representatives in the US Congress

Each rule is a conjunction of elementary conditions Each condition expresses a simple relational

statement For example the condition [State = northeast v northwest] states that the attribute

State (of the Representative) should take the value northeast or northwest to satisfy the

condition

Rl [Gas_concban = yes] amp [Soc_see_cut = no v not registered] R2 [Draft = Yes v not registered]amp[AlaslaLparks = yes v not registered] amp

[Food_stamp_cap = no] amp [State = northeast v northwest] R3 [Chrysler =yes v not registered] amp [Income =low] R4 [Education =yes] amp [Occupation = yes]

Figure 3-2 A ruleset generated by AQ15 for the concept Voting pattern of Democratic Representatives

The above rules were generated from examples of the voting records For illustration below is

an example of a voting record by a Democratic representative

Draft registration=no Ban of aid to Nicaragua=no Cut expenditure on mx missiles=yes Federal subsidy to nuclear power stations=yes Subsidy to national parks

28

in Alaska=yes Fair housing bill=yes Limit on Pac contributions=yes Limit onood stamp program=no Federal help to education=no StateFrom=north east State Population=large Occupation =unknown Cut in social security spending=no Federal help to Chrysler corp=not registered

By expressing elementary statements in the example as conditions and linking conditions by

conjunction the examples can be re-expressed as decision rules Thus decision rules and

examples formally differ only in the degree of generality

33 Generating Decision Structurestrees from Decision Rules

This section describes the AQDT-1 system for learning decision structures (trees) from decision

rules (Imam amp Michalski 1993ab) Also a description of the AQDT-2 method for learning taskshy

oriented decision structure from decision rules is included and fmally the methodology is

illustrated by two examples

Methods for learning decision trees from examples have been very popular in machine learning

due to their simplicity Decision trees built this way can be quite efficient as long as they are

used in decision-making situations for which they are optimized and these situations remain

relatively stable Problems arise when these situations significantly change and the assumptions

under which the tree was built do not hold anymore For example in some situations it may be

difficult to determine the value of the attribute assigned to some node One would like to avoid

measuring this attribute and still be able to classify the example if this is potentially possible

(Quinlan 1990) If the cost of measuring various attributes changes it is desirable to restructure

the tree so that the inexpensive attributes are evaluated f11st A tree restructuring is also

desirable if there is a significant change in the frequency of occurrence of examples from

different classes A restructuring of a decision tree to suit the above requirements is however

difficult to do The reason for this is that decision trees are a form of decision structure

representation and imposes constraints on the evaluation order of the attributes that are not

logically necessary

29

One problem in developing a method for generating decision structures from decision rules is to

design an attribute selection criterion that is based on the properties of the rules rather than of

the training examples A decision rule normally describes a number of possible examples Only

some of them are examples that have actually been observed ie training examples An attribute

selection criterion is needed to analyze the role of each attribute in the rules It cannot be based

on counting the numbers of training examples covered by each attribute-value and the

frequency of decision classes in the training examples as is done in learning decision trees from

examples because the training examples are assumed to be unavailable

Another problem in learning decision trees from decision rules stems from the fact that decision

rules constitute a more powerful knowledge representation than decision trees They can directly

represent a description in an arbitrary disjunctive normal form while decision trees can represent

directly only descriptions in the disjoint disjunctive normal form In such descriptions all

conjunctions are mutually logically disjoint Therefore when transforming a set of arbitrary

decision rules into a decision tree one faces an additional problem of handling logically

intersecting rules

The solution to both problems (attribute selection and logically intersected rules) in the AQDT-2

system is based on the earlier work by Michalski (1978) which introduced a general method for

generating decision trees from decision rules The method aimed at producing decision trees

with the minimum number of nodes or the minimum cost (where the cost was defined as the

total cost of classifying unknown examples given the cost of measuring individual attributes and

the expected probability distribution of examples of different decision classes) More

explanations are provided in the following section

331 Tbe AQDT-2attribute selection metbod

This section describes the AQDT -2 method for building a decision structure from decision rules

The method for building a single-parent decision structure is similar to that used in standard

30

methods of building a decision tree from examples The major difference is that it assigns tests

(attributes) to the nodes using criteria based on the properties of the decision rules (this includes

statistics about the examples covered by each rule in the case of learning rules from examples)

rather than statistics characterizing the frequency of training examples per decision classes per

attribute-values or per conjunctions of both Other differences are that the branches may be

assigned an internal disjunction of values (not only a single value as in a typical decision tree)

and leaves may be assigned a set of alternative decisions with probabilities Also the tests can be

attributes or names standing for logical or mathematical expressions that involve several

attributes or variables In the following we use the terms test and attribute interchangeably

(to distinguish between an attribute and a name standing for an expression the latter is called a

constructed attribute)

At each step the method chooses the test from an available set of tests that has the highest utility

(see below) for the given set of decision rules This test is assigned to the node The branches

stemming from this node are assigned test values or disjoint groups of values (in the form of

logical disjunction if such occur in the rules subsumed groups of values are removed) Each

branch is associated with a reduced set of rules determined by removing conditions in which the

selected attribute assumes value(s) assigned to this branch If all rules in the reduced ruleset

indicate the same decision class a leaf node is created and assigned this decision class The

process continues until all nodes are leaf nodes If it is not possible to reduce further the rule set

because some attribute is declared as unavailable (infmite cost) then the leaf is assigned a set of

candidate decisions with associated probabilities (see Sec 42)

The test (attribute) utility is a combination of one or more of the following elementary criteria 1)

cost which indicates the cost of using each attribute for making decision 2) disjointness which

captures the effectiveness of the test in discriminating among decision rules for different decision

classes 3) importance which determines the importance of a test in the rules 4) value

31

distribution which characterizes the distribution of the test importance over its of values and 5)

dominance which measures the test presence in the rules These criteria are dermed below

~ The cost of a test expresses the effort or cost needed to measure or apply the test

Djsjointness The disjointness of a test is defined as the sum of the class disjointness-the

disjointness of the test for each decision class Suppose decision classes are Cl C2 bull Cm and

decision rule sets for these classes have been determined Given test A let V 1 V 2 bull V m denote

sets of the values (outcomes) of A that are present in rule sets for classes C 1 C2bullbull Cm

respectively If a ruleset for some class say Ct contains a rule that does not involve test A than

Vt is the set of all possible values of A (the domain of A)

Definition 3 -1 The degree ofclass disjointness D(A Ci ) of test A for the ruleset ofclass Ci is

the sum of the degrees of disjointness D(A Cit Cj) between the ruleset for Ci and rulesets for

Cj j=l 2 m j J L The degree of disjointness between the ruleset for Ci and the ruleset for Cj is

dermed by

if Vi Vj ifVj gtVj (3-1) ifVinVj -tljgt amp -tVi amp -tVj ifVinVj=1jgt

where cjgt denotes the empty set Note that exchanging the second and third conditions of equation

(3-1) may seem to be an improved criteria However it does not clearly distinguish between both

cases (Le for both situations the disjointness will be similar) The current equation is better

because it gives higher scores to attributes that classify different subsets of the two decision

classes than attributes that classify only a subset of one decision class

Definition 3-2 The disjointness of the test A for evaluating a given set of decision rules is the

sum of the degrees of class disjointness of each decision class

32

m m Disjointness(A) = L D(A cn where D(A cn = L D(A Ci Cj) (3-2)

i=l i=l i~j

The disjointness of a test ranges from 0 when the test values in rulesets of different classes are

all the same to 3m(m-l) when every rule set of a given class contains a different set of the test

values If two tests have the same disjointness value the attribute to be selected is the one with

less number of values

Definition The A vera~ Number of Tests (ANT) required to make decision from a decision tree

is dermed as the average number of tests (attributes) to be examined from the root of the tree to

any leaf node in order to reach a decision

Definition A decision structure is a one-node-per-level decision structure if at each level there is

only one node and zero or more leaves

Such decision structure can be generated by combining together all branches that associated with

each one a set of decision rules belongs to more than one decision class

Theorem 1 Consider learning one-node-level decision structure for a database with two

decision classes The disjointness criterion ranks first attributes that add minimum number of

tests to the decision tree

Proof Consider all possible distributions of an attributes values within two decision classes Ci

and Cj There are three cases 1) subset (same as super set) 2) non-empty intersection but not

subset 3) no intersection Figure 3-3 shows all possible distributions (the case of having the

same set of values in both classes is trivial one) Consider that branches leading to one subset

with the same decision class are combined into one branch In the first case there will be two

branches only The first leads to a leaf node and the other leads to an intermediate node where

another attribute to be selected The minimum ANT in this case is 53 In the second case three

branches should be created Two branches leads to a leaf node where all values at each branch

belong to only one and different decision class The third branch leads to an intermediate node

33

where another attribute should be selected furtur classifies the decision classes The minimum

ANT in this case is 64 In the third case only two branches will be generated where each leads

to leaf node with different decision class In this case the minimum ANT is 1

Figure 3-4 shows decision trees equivalent to selecting the corresponding attribute Note that in

case of having more than one attribute-value at branches lead to leaves belonging to one decision

class they will be combined into one branch in the decision structure The symbol 1 means

that an attribute is needed to classify the two decision classes In such cases there will be at least

two additional paths

D(A Ci) = 0 D(A CJ) = 1 D(A Ci) =2 D(A Cj) = 2 D(A Ci) =3 D(A Cj) =3

Figure 3-3 Yen Diagrams of Possible combinations of attribute values in two decision classes

The average number of tests required for making a decision in each possible case is determined

in Figure 3-4 It is clear that in the case of two decision classes the disjointness criterion ranks

highly attributes that reduces the average number of tests required for decision-making The

theorem can be proved in the general case

i ~ i Cj C1middot bull Ci Cj

ANT=32middotANT=53 ANT=l

1 means at least one attribute is needed to complete the decision tree Figure 3-4 Decision trees corresponding the Yen diagrams inFigure 3-3

34

Theorem 2 The attribute with the highest non~zero disjointness is the best attribute to classify

the decision classes

Proof Suppose that the number of decision classes is n Assume also that there are two attributes

A and B where D(A) lt D(B) Since the disjointness criterion considers the mutual relationship

between any two decision classes this means that there are more decision classes where D(A

Ci) lt D(B Ci) than those where D(A Ci) gt D(B Ci) (ignoring classes with equal disjointness

for bCth attributes) Hence there are more decision classes where D(A Ci Cj) lt D(B Ci Cj)

than those where D(A Ci Cj) gt D(B Cit Cj) Let us frrst prove that if D(A Ci Cj) lt D(B Cit

Cj) then B classifies the decision classes better than A

For each pair of decision classes Ci and Cj the possible values of the disjointness of any attribute

are 0 14 or 6 For all positive values ofD(B)= 14 or 6 it is clear that attribute B should have

smaller ANT than attribute A which has lower disjointness This means that if D(A Cit Cj) lt

D(B Cit Cj) then attribute B can classify both decision classes better than attribute A

Similarly B is better for classifying more pairs of decision classes than A This implies that B is

a better classifier than A

Importance The second elementary criterion the importance of a test is based on the

importance score (IS) introduced in (Imam Michalski amp Kerschberg 1993) In the obtained

rules each test is assigned a score that represents the total number of training examples that

are covered by the rules involving this test Decision rules learned by an AQ learning program

are accompanied with information on their strength Rule strength is characterized by t-weight

and u~weight The t-weight (total-weight) of a rule for some class is the number of examples of

that class covered by the rule The importance score of a test is the aggregation of the totalshy

weights of all rules that contain that test in their condition part Given a set of decision rules for

m decision classes Cl Cm and involving n tests AlbullAn and the number of rules associated

with class Ci is denoted by tlrit The importance score is defmed as follow

35

Definition 3 -3 The importance score IS(Aj) of the test A j is detennined by

m IS(Aj)= L IS(Aj Ci) (3-31)

i=l

where ri IS(Aj Ci) = L Rik(Aj) (3-32)

k=l

and Rik the weight of a test Aj in the rule Rk of class Ci is given by

t - weight if A j belongs to rule Rik Rik (Amiddot) = (3-4)J 0 otherwise

where i=I n ik=I ri j=I m

The importance score method has been separately compared as a feature selection method with

a genetic algorithm-based method (Imam amp Vafaie 1994) The importance score method

produced an equal or higher accuracy on three real-world problems than those reported by the

GA method while selecting fewer attributes In addition the IS method was significantly faster

than the GA

yalue distribution The third elementary criterion value distribution concerns the number of

legal values of tests Given two tests with equal importance scores this criterion prefers the test

with the smaller number of legal values Experiments have shown that this criterion is especially

useful when using discriminant decision rules

Definition 34 A value distribution VD(Aj) of a test A j is defined by

VD(Aj) = IS(Aj) I Vj (3-5)

where v is the number of legal values of Aj

Dominance The fourth elementary criterion dominance prefers tests that appear in large

numbers of rules as this indicates their high relevance for discriminating among the rulesets of

given decision classes Since some conditions in the rules have values linked by internal

disjunction counting such rules directly would not reflect properly their relevance Therefore

36

for computing the dominance the rules are counted as if they were converted to rules that do not

have internal disjunction Such a conversion is done by multiplying out the condition parts of

the rules containing internal disjunction For example the condition part [x3=1 v 3]amp[x4=1] is

multiplied out to two rules with condition parts [x3=1]amp[x4=1] and [x3=3]amp[x4=I]

The above criteria are combined into one general test measure using the lexicographic

evaluation functional with tolerances (LEF) (Michalski 1973) LEF is a list of some or all of

the above elementary criteria each associated with a Ittolerance threshold It in percentage The

criteria are applied to tests in the order defined by LEF A test passes to the next criterion only if

it scores on the previous criterion within the range defined by the tolerance (from the top value)

The default LEF is

ltCost d Disjointness t2 Importance t3 Value distr t4 Dominance 0gt (3-6)

where d a t3 t4 and t5 are tolerance thresholds (in percentages) their default values are O

The default value of the cost of each test is 1

The above LEF ranks attributes this way First attributes are evaluated on the basis of their cost

If two or more attributes have the same cost they ranked by their degree of disjointness If two

or more attributes still share the same top score or their scores differ less than the assumed

tolerance threshold t2 the method evaluates these attributes using the second (Importance)

criterion If again two or more attributes share the same top score or their scores differ less than

the tolerance threshold t3 then the third criterion normalized IS is used then similarly the

fourth criterion (dominance) If there is still a tie the method selects the best attribute

randomly

If there is a non-uniform frequency distribution of examples of different classes then the

selection criterion uses a modified definition of the disjointness Namely the previously defmed

37

disjointness for each class is multiplied by the frequency of the class occurrence The class

occurrence is the expected number of future examples that are to be classified to a given class

m

Disjointness(A) = L D(A CO Frq(Ci) (3-7) i=l

where Frq(Cn is the expected frequency of examples of class Cit and it is assumed to be given

by the user The attribute ranking criteria in this case is defined by the LEF

ltCost d Disjointness t2 Importance 13 Normalized-IS 14 Dominance 15gt (3-8)

where the Cost denotes the evaluation cost of an attribute and is to be minimized while other

elementary criteria are treated the same way as in the default version

332 The AQDT-2 algorithm

The AQDT-2 algorithm constructs a decision structure from decision rules by recursively

selecting at each step the best test according to the ranking criteria described above and

assigning it to the new node The process stops when the algorithm creates terminal branches that

are assigned decision classes To facilitate such a process the system creates a special data

structure for each concept description (ruleset) This structure has fields such as the number of

rules the number of decision classes and the number of attributes present in the rules A set of

pointers connect this data structure to set of data structures each represents one decision class

The decision class structure contains fields with information on the number of rules belong to

that class the frequency of the decision class etc It is also connected to a set of data structures

representing the decision rules within each decision class The system creates independently a set

of data structures each is corresponding to one attribute Each attribute description contains the

attributes name domain type the number of legal values a list of the values the number of

rules that contain that attribute and values of that attribute for each rule The attributes are

arranged in an array in a lexicographic order flISt in the descending order of the number of rules

38

that contain that attribute and second in the ascending order of the number of the attributes

legal values

The system can work in two modes In the standard mode the system generates standard

decision trees in which each branch has a specific attribute-value assigned In the compact

mode the system builds a decision structure that may contain

A) or branches Le branches assigned an internal disjunction of attribute-values whenever it

leads to simpler structures For example if a node assigned attribute A has a branch marked by

values 1 v 2 then the control passes along this branch whenever A takes value 1 or 2 The

program creates or branches on the basis of the analysis of the value sets Vi while computing

the degree of attribute disjointness

B) nodes that are assigned derived attributes that is attributes that are certain logical or

mathematical combinations of the original attributes To produce decision structures with

derived attributes the input decision rules are generated by AQ17 (rather than AQI5) The

AQ17 rules may contain conditions involving attributes constructed by the program rather than

those originally given

To generate decision structures from rules the AQDT-2 method prefers either characteristic or

discriminant disjoint rule descriptions (given by an expert or learned by a system) Disjoint rules

are more suitable for building decision structures Assume that the description of each class is in

the form of a ruleset and that this set is the initial ruleset context The AQDT algorithm is

The AQDT2 Algorithm

Step 1 Evaluate each attribute occurring in the ruleset context using the LEF attribute ranking

measure Select the highest ranked attribute Let A represents this highest-ranked

attribute

Step 2 Create a node of the tree (initially the root afterwards a node attached to a branch) and

assign to it the attribute A In standard mode create as many branches from the node as

39

there are legal values of the attribute A and assign these values to the branches In

compact mode (decision structures) create as many branches as there are disjoint value

sets of this attribute in the decision rules and assign these sets to the branches

Step 3 For each branch associate with it a group of rules from the ruleset context that contain a

condition satisfied by the value(s) assigned to this branch For example if a branch is

assigned values i of attribute A then associate with it all rules containing condition [A=

i v ] If a branch is assigned values i v j then associate with it all rules containing

condition [A= i v j v ] Remove from the rules these conditions If there are rules in

the ruleset context that do not contain attribute A add these rules to all rule groups

associated with the branches stemming from the node assigned attribute A (This step is

justified by the consensus law [x=1] == [x=1] amp [y=a] v [x=1] amp [y=b] assuming

that a and b are the only legal values of y) All rules associated with the given branch

constitute a ruleset context for this branch

Step 4 If all the rules in a ruleset context for some branch belong to the same class create a leaf

node and assign to it that class If all branches of the trees have leaf nodes stop

Otherwise repeat steps 1 to 4 for each branch that has no leaf

To select an attribute to be a node in the decision tree (step 1 and 2 of the algorithm) the

algorithm performs two independent iterations In the first iteration it parses through all decision

rules and determine information about each attributes This information includes the importance

score of each attribute the number of rules containing a given attribute the disjoint value sets of

each attribute and the attribute values used in describing each decision class The second

iteration is only performed if the disjointness criterion is ranked first in the LEF function The

second iteration evaluates the attributes disjointness for each decision class against the other

decision classes

40

To determine the complexity of this process suppose that the maximum number of conditions

fonned by a single attribute value in one rule is s (where s ~ n the number of attributes) and r is

the total number of decision rules (in all decision classes)

m

r=LRt (m is the number of decision classes) 1=1

where Ri is the number of rules in decision class Ci bull The complexity of the frrst iteration can be

determined as

Cmpx(Itcl) =O(r s)

In the second iteration the disjointness is calculated between the decision classes and all

attributes The complexity of the second iteration can be given by

Cmpx(Itc2) = O(n m)

Assume that at each node 1 is the maximum of the number of decision classes to be classified at

this node and the number of rules associated with this node

l=max mr (3-9)

Since these two iteration are independent of each other the complexity of the AQDT algorithm

for building one node in the decision tree say Node Complexity NC(AQDT) is given by

NC(AQDT) = 0(1 n)

Usually I equals to the number of rules associated with the given node Thus the AQDT

complexity for building one node is a function of the number of attributes multiplied by the

number of rules associated with this node

At each level of the decision tree these two iterations are repeated for each non-leaf node The

Level Complexity of the AQDT algorithm LC(AQDT) can be given by

LC(AQDT) lt 0(1 n)

Which is less than the complexity of generating the root of the decision tree To explain this

consider the maximum possible number of non-leaf nodes at one level is half the number of the

initial decision rules (integer of rll) Figure 3-5-a shows an example of such situation where at

41

the lowest level each node classifies only two rules each belongs to different decision class This

decision tree could be a multimiddotvalues tree However the number of non-leaf nodes at each level

will be twice or more than the number of nonmiddot leaf nodes at the previous level Considering the

level complexity of the AQDT algorithm equal to (1 s 0) where 0 is the number of non-leaf

nodes at the given level In such cases it is either (1 0 S r) or (1 s lt r) In Figure 3middot5middota the

complexity of the AQDT algorithm at any lower level is given by

LC(AQDT) =0(2 s r(2) lt O(n I) =NC(AQDT)

a) per one level b) per one path

Figure 3 -5 Decision trees showing the maximum number of none leaf nodes

Note also that after selecting an attribute to be the root of the decision structure this attribute

and all conditions containing that attribute are removed from the data structure of the algorithm

Also if a leaf node is generated all rules belonging to the corresponding branch will not be

tested again

Since the disjointness criterion selects the attribute which minimizes the average number of tests

ANT it is true that the AQDT algorithm generates decision trees with the least number of levels

The number of levels per a decision tree is supposed to be less than or equal to the minimum of

both the number of attributes and the number of rules Consider k as the number of levels in a

given decision tree

k S min mr (3-10)

There are two cases represent the most complex situations Figure 3-5-a and 3-5-b In the fIrst

case where the decision rules were divided evenly the number of levels will be a function of the

42

logarithm of the number of rules In such case the complexity of the AQDT algorithm for

generating a decision tree from a set of decision rules is given by

Complexity(AQDT) = 0(1 n log r) (3-11)

The other situation is when the generated decision tree has the maximum number of levels The

maximum possible number of levels per a decision tree equals to one less than the number of

decision rules Figure 3-5-b Using the disjointness criterion it is not likely to get such a decision

tree because it has the maximum average number of test (ANT) that can be determined from the

same set of nodes and leaves However such a decision tree can be generated if the number of

decision classes is one less thanthe number of attributes In such case any disjQint decision rules

should have a maximum length that is less or equal to the floor of the logarithm of the number of

attributes Thus the level complexity of this decision tree is estimated as

LC(AQDT) =0(1 log n)

The maximum number of levels in such decision tree is k-l Thus the complexity of the AQDT

algorithm in such cases is given by

Complexity(AQDT) = 0(1 k log n) (3-12)

Since k is the minimum of n and r then n is the maximum of k and n Also since in (3-12)

r=n+1 then log r is greater than log n From 3-10 3-11 and 3-12 the complexity of the AQDT

algorithm is determined by

Cmplx(AQDT) = OCr k log 1) (3-13)

333 An example illustrating the algorithm

The following simple example illustrates how AQDT is used in selecting the optimal set of

testing resources for testing a software Suppose there are three tools for testing a software 1)

modeling (Tl) 2) checklist (T2) and 3) par_simul (T3) Also assume that there are four

different factors that affect the selection of any tool 1) the cost of using the tool (xl) 2) the

metrics that support the tool (x2) 3) the best phase for applying the tool (x3) and 4) type of tool

43

(automated semi-automated or manual) (x4) Table 3-1 shows these attributes and their possible

values

Suppose that the domain expert provided a set of rules to be used in testing a software Figure 3shy

6 shows a sample of these rules in AQ15c formats

Table 3-1 The available tools and the factors that affect the prclCes

Tl lt= [xl=2] amp [x2=2] v [xl=3] amp [x3=1 v 3] amp [x4=1] T2 lt= [xl=l v 2] amp [x2=3 v 4] v [xl=3] amp [x3=1 v 2] amp [x4=2] T3 lt= [xl=l] amp [x2=1] v [xl=4] amp [x3=2 v 3] amp [x4=3]

Figure 3-6 Decision rules for selecting the best tool for testing software

These rules can be interpreted as

Rule 1 Use the fltst tool for testing if you need average cost and the tool is

supported by the requirement metric

Rule 2 Use the fIrst tool for testing if you can afford high cost for testing either

in the requirement or the analysis phases and you need an automated tool

Rule 3 Use the second tool for testing if the cost limit is low or average and the

tool is supported by the system usage metric or the intractability metric

Rule 4 Use the second tool for testing if you can afford high cost for testing

either in the requirement or the design phases and you need a manual tool

Rule 5 Use the third tool for testing if the you are limited to low cost and the

tool is supported by the error rate metric

Rule 6 Use the third tool for testing if you can afford very-high cost for testing

either in the requirement or the system usage phases and you need a semishy

automated tool

44

Table 3-2 presents information on these rules and the disjointness values for all attributes For

each class the row marked Values lists values occurring in the ruleset for this class For

evaluating the disjointness of an attribute say A each rule in the ruleset above that does not

contain attribute A is characterized as having an additional condition [A= a vb ] where a b

are all legal values of A

The row Class disjointness specifies the class disjointness for each attribute The attribute xl

has the highest disjointness (11) and is assigned to the root of the tree For simplicity assume

the tolerances for each elementary criterion equal O

From the rules in Figure 3-6 we can also determine disjoint groupings of attribute-values used

for the compact mode of the algorithm This done as follows 1) determine for each attribute the

sets of values that the attribute takes in individual decision rules and remove those value sets

that subsume other value sets The remaining value sets are assigned to branches stemming from

the node marked by the given attribute For example x 1 has the following value sets in the

individual decision rules 2 3 I 2 I and 4 (Figure 3-6) Value set I 2 is

removed as it subsumes 2 and I In this case branches are assigned individual values of the

domain of x1 For attribute x2 the value sets are I 2 34 and I 2 3 4 In this case

branches are assigned value sets I 2 and 34

45

Attribute Xl ranks highest (as it is the highest disjointness) and is assigned to the root of the

tree Four branches are created each one corresponding to one of XIs possible values Since all

rules containing [x1=4] belong to class TI the branch marked by 4 is ended by a leafTI Rules

containing other values of X1 belong to more than one class This process is repeated for each

subset of rules until the decision tree is completed Figure 3-7 shows a decision structure learned

by AQDT-2 from the rules in Figure 3-6 The decision structure in Figure 3-7 can be used in

making decisions on which tools can be used for testing a given software

xl

Complexity No of nodes 4 No of leaves 7

Figure 3-7 A decision structure learned for classifying so tware testing tools

Figure 3-8-a shows the diagrammatic visualization of the decision rules and Figure 3-8-b shows

the visualization of the derived decision tree Each diagram in Figure 3-8 consists of cells

representing one combination of attribute-values Attributes and their legal values are shown on

scales surrounding the diagram (eg the horizontal scale for xl shows values 1 2 3 and 4)

Rules are represented by collections of cells in the intersection of the rows and columns

corresponding to the conditions in the rules

The shaded areas correspond to decision rules Rules of the same class have the same type of

shading Empty cells correspond to combination of attribute-values not assigned to any class For

illustration collections of cells corresponding to some of the initial rules are marked R 11 R 21

R31 and R32 Rll denotes the IlfSt rule of Class T1 ie [x1=2] amp [x2=2] R21 denotes the

IlfSt rule of class T2 ie [xl= 1 v 2] amp [x2= 3 v 4] R31 the first rule of class TI ie [x1=1]

46

amp [x2 =1] and denotes R32 denotes the second rule of class T3 Le [xl=4] amp [x3=2 v 3] amp

[x4=3]

a) Decision rules b) Derived decision tree

Comparing diagrams in Figure 3-8-a and 3-8-b AQDT-2 decision tree represents a slightly more

general description of Concepts Tl TI and T3 than the original rules Let us assume that it is

very costly to know which metrics suppon the required tools In other words suppose that we

would like to select the best tools independently of the metric they suppon (this is indicated to

AQDT-l by assigning very high cost to attribute x2) The algorithm again assigns attribute xl to

the root of the tree When xl takes value 1 or 2 it is impossible to assign a specific decision

without measuring attribute x2 For the value 1 of xl the recommended tool can be either TI or

T3 (see the diagram in Figure 3-9-a) and for the value 2 of xl the recommended tool can be

either Tl or T3 However for the value 3 of xl one can make a specific decision after

measuring attribute x4 For the value 1 of x4 tool is Tl and for the value 2 of x4 the

recommended tool is TI Figure 3-9-b shows another decision tree where the type of the tool

was ignored from the data Those decision trees are called indeterminate because some of their

leaves are assigned a disjunction of two or more class names

47

xl

xl

1 12

a) Ignoring the supporting metric b) Ignoring the type of the tooL Figure 3 -9 Decision trees learned ignoring the support metric and the type of the testing tool

It is clear that the given set of rules depend highly on amount of money can be spent Let us now

suppose that the cost attribute xl which was determined as the highest ranked attribute cannot

be measured The algorithm selected x4 to be the root of the new decision tree After continuing

the algorithm the tree in Figure 3-10 is obtainedThe nodes that are assigned one class indicate

situations in which it is possible to make a specific decision without knowing the value of

attribute xl

x4

Figure 3-10 A decision tree learned without the cost attribute

34 Tailoring Decision Structures to the Decision-making situation

Decision structures are among the simplest structures for organizing a decision-making process

A decision structure specifies explicitly the order in which attributes of an object or situation

need to be evaluated in the process of determining a decision A standard way to generate a

48

decision structure is to learn it from examples of decisions Such a process usually aims at

obtaining a decision structure that has the highest prediction accuracy that is the highest rate of

assigning correct decisions to given situations There can usually be a large number of logically

equivalent decision structures (Michalski 1990) As such they may have the same predictive

accuracy but differ in the way they organize the decision process and thus may differ in the cost

of arriving at a decision To minimize the average decision cost one needs to take into

consideration the distribution of the costs of attribute evaluation and the frequency of different

decisions This report presents an approach to building such Task-oriented decision structures

which advocates that they are built not from examples but rather from decision rules Decision

rules are learned from examples using the AQ15 or AQ17 inductive learning program or are

specified by an expert An efficient algorithm and a new system AQDT-2 that transforms

decision rules to task-oriented decision structures The system is illustrated by applying it to the

problem of learning decision structures in the area of construction engineering (for determining

the best wind bracing for tall buildings) In the experiments AQDT-2 outperformed all other

programs applied to the same data

Decision-making situations can vary in several respects In some situations complete

information about a data item is available (ie values of all attributes are specified) in others the

information may be incomplete To reflect such differences the user specifies a set of parameters

that control the process of creating a decision structure from decision rules AQDT-2 provides

several features for handling different decision-making problems 1) generating a decision

structure that tries to avoid unavailable or costly attributes (tests) 2) generating unknown

leaves in situations where there is insufficient information for generating a complete decision

structure 3) providing the user with the most likely decision when performing a required test is

impossible 4) providing alternative decisions with an estimate of the likelihood of their

correctness when the needed information can not be provided

49

341 Learning Costmiddot Dependent Decision Structures

As described in Sec 2 the LEF criterion enables the system to take into consideration the cost of

measuring tests (attributes) in developing a decision structure In the default setting of LEF the

tests cost is the [lIst criterion and its tolerance is O This means that only the least expensive

attributes pass to the next step of evaluation involving other elementary criteria If an attribute

has high cost or is impossible to measure (has an infinite cost) the LEF chooses another

cheaper attribute ifpossible

342 Assigning Decision Under Insufficient Information

In decision-making situations in which one or more attributes cannot be measured the system

may not be able to assign a definite decision for some cases If no more information can be

obtained but a decision has to be made it is useful to know the probability distribution for

different candidate decisions (Smyth Goodman amp Higgins 1990) The most probable decision is

then chosen The probability distribution can be estimated from the class frequency at the given

node Let us consider a node in the structure connected to the root by the sequence bl b2 bkof

attribute-values Let P(Ci I bl b2 bk) denote the conditional probability of class Ci i=I2 m

at that node given that an example to be classified has attribute-values assigned to branches bl

b2 bull bk in the decision structure Using Bayesian formula we have

P(Ci I bl~ bull bk) = P(Ci) P(bl bk I Ci) P(bl bk) (3-9)

where P(Ci) the apriori probability of class Ci P(bl bk I Ci) is the probability that the

example has attribute-values blbull bk given that it represents decision class Ci To approximate

these probabilities let us suppose that Wi is the number of training examples of Class Ci that

passed the tests leading to this node and twi - the total number of training examples of Class

Ci Let us use these numbers to estimate the probabilities in (9) Assuming that the apriori class

probabilities correspond to the frequency of training examples from different classes we have

50

m P(Ci) = twi IL twj (3-10)

j=l

P(b1 bkl Ci) = Wi I twi (3-11)

m m P(b1 bk) =L Wj IL twj (3-12)

j=l j=1

By substituting (1O) (11) and (12) in (9) we obtain

m P(Ci I b1 bk) = wilL Wj (3-13)

j=l

A related method for handling the problem of unavailability of an attribute is described by

Quinlan (1990) Quinlans method however assigns probabilities to the most probable decision

at the node associated with such an attribute The method does not restructure the decision tree

appropriately to fit the given decision-making situation (in this case to avoid measuring xl)

AQDT -2 assigns probabilities o~ly after it first created a decision structure that suits the given

decision-making si~ation An example is presented in section 42

343 Coping with noise in training data

The proposed methodology can be easily extended to handle the problem of learning from noisy

training data This is done based on ideas of rule truncation described in (Niblett amp Bratko 1986

Bergadanoet al 1992 Cestnik amp Karalic 1991) When noise is expected in the training data

the decision rules with t-weight below a certain threshold (reflecting the expected noise level) are

removed The rule truncation method seems to result in more accurate decision structures than

decision tree pruning method because truncation decisions are solely based on the importance of

the given rule or condition for the decision-making regardless of their evaluation order (unlike

the decision tree pruning which can only prunes attributes within a subtree and thus cannot

freely chose attributes to prune) Examples are presented in section 4

DERIVING TASK-ORIENTED DECISION STRUCTURES FROM DECISION RULES

Adissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy at George Mason University

By

Ibrahim M Fahmi Imam

Director Professor Ryszard S Michalski

PRC Chaired Professor of Computer Science amp Systems Engineering School of Information Technology and Engineering

George Mason University

Fall Semester 1995 George Mason University Fairfax Virginia 22030

copy 1995 Copyright by Ibrahim F Imam All rights reserved

ACKNOWLEDGMENT

I would like to thank Professor Ryszard S Michalski PRC Chaired Professor of Computer

Science and Engineering and my Dissertation Director for his support encouragement and

guidance I would like to thank my committee members Professor Larry Kerschburg Chair of the

Department of Information Systems and Software Engineering Professor David Rine Professor

of Information Technology and Engineering and Professor David Schum Professor of Excellence

in Information Technology and Engineering for their encomagement and help with many aspects

of my PhD

I would like to thank Professor Tomaz Arciszewki System Engineering Department for providing

me with application problems Ronny Kohavi Stanford University for discussion and providing

me with some related work on learning decision structures and decisionmiddot graphs and Professor

George Tecuci Computer Science Department for pointing some related work

I would like to thank my colleagues Nabil Al-Kharouf for reviewing my dissertation Eric

Bloedorn for reviewing an earlier draft of my dissertation and for using his program AQ17-DCI in

my experiments Srinivas Gutta for providing some application for my PhD work Mike Heib for

reviewing an earlier draft of my dissertation and helping me find relevant articles Ken Kaufman

for reviewing an earlier draft of my thesis Mark Maloof for providing me with script files which

made it easier to iteratively run AQ1Sc Halah Vafaie for working with her on application and

comparison of different aspects of my work and Janusz Wnek for using his DIAV program for

explaining my results

I would like to thank Professor Andrew P Sage Dean of the School of Information Technology

and Engineering and Professor Kenneth Bumgarner Dean of Student Services and Associate ViCe

President of George Mason University for their support and Professor Murray W Black

Associate Dean of the School of Infonnation Technology and Engineering for guidance on

preparing the PhD proposal

I would like to thank conferences organizers who supported me to attend their conferences and

present parts of my PhD work The organizers include Professor Moonis Ali Professor Frank

Anger Professor Zbigniew Ras the American Association for Artificial Intelligence and others I

would like also to thank the organizing committee of the Florida Artificial Intelligence Research

Symposium FLAIRS-95 for organizing my first workshop Thanks goes to Dr Doug Dankel Dr

Howard Hamilton Dr John Stewman and Dr Dan Tamir

I would like also to thank the many individuals who helped me in any way during my PhD Those

include Dr Ashraf Abdel-Wahab Dr Jerzy Bala Dr Alex Brodsky Dr Richard Carver Dr

Kenneth De Jong John Doulamis Tom Dybala Dean Ibrahim Farag Dr Ophir Frieder Csilla

Frakes Dr Mohamed Habib Ali Hadjarian Dr Hugo De Garis Zenon Kulpa Dr Paul Lehner

Dr George Michaels Dr Eugene Norris Dean James Palmer Mitch Potter Dr Ahmed Rafea

Jim Ribeiro Jsyshree Sarma Dr Arun Sood Dr Clifton Sutton Bradley Utz Dr Tibor Vamos

Patricia Zahra Dr Shaker Zahra and Dr Jianping Zhang

~J I yarJ I JJ I ~

U-t-oJ ~II ~) ~ oWAIl J

Dedication

To my mother my brothers and my sister

TABLE OF CONTENTS

TI1LE Page

ABS1RAcr II 1

CHAPrER 1

IN1RODUCI10N II 3

11 Motivation and Overview 3

12 The Problem Statement 6

CHAPrER2

REIAfED RESEARCFI 7

21 Learning Decision Trees from Decision Diagrams 7

22 Learning Decision Trees from Examples 10

221 Building Decision Trees Using Information-based Criteria 11

222 Building Decision Trees Using Statistics-based Criteria 16

223 Analysis of Attribute Selection Criteria 18

23 Learning Decision Structures 19

CHAPrER3

DESCRIPTION OF TIlE APPROACFI 23

31 General Methodology 23

32 A Brief Description of the AQ15 and AQ17 RuleLeaming Systems 25

33 Generating Decision Structures From Decision Rules 28

331 The AQDT-2 attribute selection method 29

332 The AQDT-2 algoridlm 37

vi

333 An example illustrating the algorithm 42

34 Tailoring Decision Structure to a decision-making situation 47

341 Learning Cost-Dependent Decision Structures 49

342 Assigning Decision Under Insufficient Infonnation 49

343 Coping with noise in training data 50

35 Analysis of the AQDT-2 Attribute Selection Criteria 51

36 Decision Structures vs Decision Trees 53

CHAPTER 4

EMPRICAL ANALYSIS AND COMPARATIVE STUDY OF TIlE MElHOD 58

41 Description of the Experimental Analysis 59

42 Experiments With Average Size Complex and Noise-Free Problems

Wind Brac-ings 60

43 Experiments With Small Size Simple and Noise-Free Problems

e MONK-I 69

44 Experiments With Small Size Complex and Noise-Free Problems

MONK-2 76

45 Experiments With Small Size Simple and Noisy Problems

MONK-3 79

46 Experiments With Large Size Complex and Noise-Free Problems

Diagnosing Breast Cancer 83

47 Experiments With Large Size Complex and Noisy Problems

Musm()()m classifications 84

48 Experiments With Small Size Structured and Noise-Free Problems

East-West Trains 85

49 Experiments With Small Size Simple and Noisy Problems

vii

Congressional Voting Records (1984) 87

410 Analysis of the Results 88

ClIAPfER 5 CONCIUSIONS 95

51 Summary 95

52 Contributions 96

REFERENCES 98

VITA 102

viii

LIST OF TABLES

No TITLE Page

2-1 An example of a decision table 9

2-2 A set of training examples used to illustrate the C45 system 15

2-3 The frequency of different attributes values to different decision classes 17

2-4 The expected values of the frequency of examples in Table 2-3 17

2-5 Attribute selection criteria and their basic evaluation measure 17

2-6 The contingency tables of Mingers example 18

2-7 Mingers results for determining the goodness of split 19

2-8 Mingers results for comaring the total accuracy and size of decision trees

provided by different attribute selection criteria from four problems 19

2-9 Comparing the AQDT approach with EDAG and HOODG approaches 22

3-1 The available tools and the factors that affect the process of testing a software 43

3-2 Calculating the disjointness of each attribute 44

3-3 Evaluation of the AQDT-2 criteria on the Quinlans example in Table 2-2 51

3-4 The data used in Mingers f11st experiments 52

3-5 The performance of AQDT-2 criteria (compare with the other criteria in Table 2-6) 52

3-6 The possible ranking domains and using condition of AQDT-2 criteria 53

3-7 Comparison between Decision Structures and Decision Tree 54

4-1 Evaluation of AQDT-2 attribute selection criteria for the wind bracing problem 62

4-2 The predictive accuracy of AQ15c and AQDT-2 for the wind bracing problem 67

4-3 Evaluation of AQDT-2 attribute selection criteria for the MONK-l problem 71

4-4 The predictive accuracy of AQ15c and AQDT-2 for the MONK-l problem 73

ix

4-5 The predictive accuracy of AQ15c and AQDT-2 for the MONK-2 problem 77

4-6 The predictive accuracy of AQ15c and AQDT-2 for the MONK-3 problem 81

4-7 The set of attributes and their values used in the trains problem 86

4-8 A tabular snmmary of the predictive accuracy of decision trees obtained

by AQDT -2 and C45 for the congressional voting data 88

4-9 Summary of the best parameter settings for the first subfunction of the approach

with different data characteristics 89

4-10 Summary of the performance of AQDT-2 and C45 on different problems 90

x

LIST OF FIGURES

No TITLE Page

2-1 An example to illustrate how attributes break rules 8

2-2 A decision tree learned from the decision table in Table 2-1 10

2-3 A decision tree learned using the gain criterion for selecting attributes 15

2-4 Decision rules and their Exception Directed Acyclic Graph (EDAG) 21

3-1 Architecture of the AQDT approach 24

3-2 A ruleset generated by AQ15 for the concept Voting pattern of

Democratic Representatives 27

3-3 Ven Diagrams ofPossible combinations of attribute values in two decision classes 33

3-4 Decision trees corresponding the Ven diagrams in Figure 3-3 33

3-5 Decision trees showing the maximum number of none leaf nodes 41

3-6 Decision rules for selecting the best tool for testing a software 43

3-7 A decision structure learned for classifying software testing tools 45

3-8 A diagrammatic visualization of the decision rules and the derived decision tree 46

3-9 Decision trees learned ignoring the support metric and the type of the testing tool 47

3-10 A decision tree learned without the cost attribute 47

3-11 Decision structures learned by AQDT-2 using different criteria 55

3-12 The Imams example Example where learning decision structures (trees)

from rules is better than learning them from examples 56

3-13 An example where decision rules are simpler than decision trees 57

4-1 Design of a complete experiment 59

Xl

4-2 Decision rules determined by AQ15c from the wind bracing data 61

4-3 A decision tree learned by C45 for the wind bracing data 63

4-4 A decision structure learned from AQ15c wind bracing rules 64

4-5 A decision structure that does not contain attribute xl 64

4-6 A decision structure without Xl with candidate decisions assigned to leaves 65

4-7 A decision structure determined from rules in Figure 4-4 under the

assumption of 10 classification error in the training data 65

4-8 Diagramatic visualization of decision trees learned for different decision

making situations for the wind bracing data 66

4-9 The accuracy of AQDT-2 and AQ15c with different parameter settings

for the wind bracing PIoblem 68

4-10 Analyzing different parameter setting of AQDT-2 using the wind bracing data 69

4-16 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-20 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-11 A comparison between AQ15c and AQDT-2 against C45 on the wind bracing data 69

4-12 A visualization diagram of the MONK-l problem 70

4-13 Decision rules learned by AQ15c for the MONK-1 problem 71

4-14 The decision tree for the MONK-1 problem generated both by AQDT-l 72

4-15 Compact decision structures generated by AQDT-2 for the MONK-1 problem 72

for the MONK-l Problem 74

4-17 Analyzing different parameter setting of AQDT-2 with the MONK-l data 75

4-18 Comparing AQ15c and AQDT-2 against C45 using the MONK-1 problem 75

4-19 A visualization diagram of the MONK-2 problem 76

for the MONK-2 Problem 78

4-21 Analyzing different parameter setting of AQDT-2 with the MONK-2 data 79

xii

4-22 Comparing AQ15c andAQDT-2 against C45 using the MONK-2 problem 79

4-24 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-35 A visualization diagram of a decision tree learned by AQDT -2 after reducing

4-23 A visualization diagram of the MONK-3 problem 80

for the MONK-3 Problem 82

4-25 Analyzing different parameter setting of AQDT-2 with the MONK-3 data 82

4-26 Comparing AQ15c and AQDT-2 against C45 using the MONK-3 problem 83

4-27 Comparing AQ15c and AQDT-2 against C45 using the breast cancer problem 84

4-28 Comparing AQ15c and AQDT-2 against C45 using the mushroom problem 85

4-29 Decision structures learned by AQDT-2 for different decision-making situations 87

4-30 Comparing decision trees for the Cong Voting-84 data learned by C45 amp AQDT -2 88

4-31 A visualization diagram of decision rules learned by AQ15c for the MONK-2 91

4-32 A visualization diagram shows the testing errors for the rules in Figure 4-31 92

4-33 A visualization diagram of the decision tree learned by AQDT-2 for the MONK-2 93

4-34 A visualization diagram shows testing errors of AQDT-2 decision tree bull 94

the generalization degree to 1 94

DERIVING TASKmiddotORIENTED DECISION STRUCTURES

FROM DECISION RULES

Ibrahim M Fahmi Imam PhD

School of Information Technology and Engineering

Geotge Mason University Fall 1995

Ryszard S Michalski Advisor

ABSTRACT

This dissenation is concerned with research on learning task-oriented decision structures from

decision rules The philosophy behind this research is that it is more appropriate to learn

knowledge and store it in a declarative form and then when a decision making situation occurs

generate from this knowledge the decision structure that is most suitable for the given decision

making situation Learning decision structures from decision rules was first introduced by

Michalski (1978) The flIst implementation of this approach was done by Imam and Michalski

(1993 ab) called AQDT-l

This approach separates the function of generating a knowledge-base from the function of using

the knowledge-base for decision-making The first function focuses on learning accurate

consistent and complete concept description expressed in a declarative form The second function

is performed whenever a new decision-making situation occurrs a task-oriented decision structure

is obtained to suit that situation Thsk-oriented knowledge is defined as knowledge that is adapted

for solving a given decision-making situation (Imam amp Michalski 1994 Michalski amp Imam

1994)

The dissertation introduces the system AQJJT-2 for learning task -oriented decision structures from

decision rules or examples Each decision making situation is defined by a set of parameters that

controls the learning process of the AQDT-2 system The extensive experiments on AQJJT-2 show

that decision structures learned by it usually outperform in terms of accuracy and average size of

the decision structures those learned from examples by other well known systems The results

show also that the system does not work very well with noisy data The system is illustrated and

compared using applications of artificial problems such as the three MONKs problems (Thrun

Mitchell amp Cheng 1991) and the East-West 1iain problem (Michie et al 1994) It was also

applied to real world problems of learning decision structures in the areas of construction

engineering (for determining the best wind bracing design for tall buildings) medical

diagnosis(for learning decision rules for recognizing breast cancer) agriculture diagnosis (for

learning classification rules for distinguishing between poisonous and non-poisonous

mushrooms) and political data (for characterizing democratic and republican voting records)

CHAPTER 1 INTRODUCTION

11 Motivation and Overview

Learning and discovery systems should be able not only to generate and store knowledge but also

to use this knowledge for decision-making The main step in the development of systems for

decision-making is the creation of a knowledge structure that characterizes the decision-making

process The form in which knowledge can be easily obtained may however differ from the form

in which it is most readily used for decision-making It is therefore important to identify the form

of knowledge representation that is most appropIiate for learning (eg due to ease of its

modification) and the form that is most convenient for decision making

A simple and effective tool for describing decision processes is a decision structure which is a

directed acyclic graph that specifies an order of tests to be appliedto an object (or a situation) to

arrive at a decision about that object The nodes of the structure are assigned individual tests

(which may correspond to a single attribute a function of attributes or a relation) the branches are

assigned possible test outcomes or ranges of outcomes and the leaves are assigned a specific

decision a set of candidate decisions with corresponding probabilities or an undetermined

decision A decision structure reduces to a familiar decision tree when each node is assigned a

single attribute and has at most one parent when the branches from each node are assigned single

values of that attribute and when leaves are assigned single definite decisions Thus the problem

of generating a decision structure is a generalization of the problem of generating a decision tree

Decision trees are typically generated from a set of examples of decisions The essential

characteristic of any such method is the attribute selection criterion used for choosing attributes to

be assigned to the nodes of the decision tree being built Such criteria include the entropy

reduction the gain and the gain ratio (Quinlan 1979 83 86) the gini index ofdiversity (Breiman

et al 1984) and others (Cestnik amp Bratko 1991 Cestnik amp Karalic 1991 Mingers 1989a)

3

4

Adecision treedecision structure representations can be an effective tool for describing a decision

process as long as all the required tests can be performed and the decision-making situations it

was designed for remain constant (eg in a doctor-patient example the doctor should detennine

that answers for all symptoms appear in the decision tree) Problems arise when these assumptions

do not hold For example in some situations measuring certain attributes may be difficult or costly

(eg in the doctor-patient example a brain or blood test is needed which is very expensive or the

tools needed are not available) In such situations it is desirable to refonnulate the decision

structure so that the inexpensive attributes are evaluated frrst (assigned to the nodes close to the

root) and the expensive attributes are evaluated only if necessary (by assignment to the nodes far

away from the root) If an attribute cannot be measured at all it is useful to either modify the

structure so that it does not contain that attribute or-when this is impossible-to indicate

alternative candidate decisions and their probabilities A restructuring is also desirable if there is a

significant change in the frequency of occurrence of different decisions (eg in the doctor-patient

example the doctor may request a decision structure expressed in a specific set of symptoms

biased to classify one or more diseases or specify a certain order of testing)

Arestructuring ofa decision structure (or a tree) in order to suit new requirements could be quite

difficult This is because a decision structure is a procedural representation that imposes an

evaluation order on the tests In contrast no evaluation order is imposed by a declarative

representation such as a set of decision rules Tests (conditions) of rules can be evaluated in any

order Thus for a given set of rules one can usually build a huge number of logically equivalent

decision structures (trees) which differ in the test ordering Due to the lack of order constraints

a declarative representation (rules) is much easier to modify to adapt to different situations than a

procedural one (a decision structure or a tree) On the other hand to apply decision rules to make a

decision one needs to decide in which order tests are evaluated and thus needs a decision

structure

An attractive solution to these opposite requirements is to acquire and store knowledge in a

declarative form and transform it to a decision structure when it is needed for decision-making

5

This method allows one to create a decision structure that is most appropriate in a given decisionshy

making situation Because the number of decision rules per decision class is usually small (each

rule is a generalization of a set of examples) generating a decision structure from decision rules

can potentially be perfonned much faster than by generation from training examples Thus this

process could be done on line without any delay noticeable to the user Such virtual decision

structures are easy to tailor to any given decision-making situation

This approach allows one to generate a decision structure that avoids or delays evaluating an

attribute that is difficult to measure in some decision-making situation or that fits well a particular

frequency distribution of decision classes In other situations it may be unnecessary to generate

complete decision structure but it may be sufficient to generate only a part of it which concerns

only with decision classes of interest Thus such an approach has many potential advantages

This dissertation presents a new system called AQflf-2 The AWf-2 system generates a taskshy

oriented decision structure (decision structure that is adapted to the given decision-making

situation) from decision rules The decision rules are learned by either rule leaming system AQ15

(Michalski et al 1986) or system AQ17-DCI which has extensive constructive induction

capabilities (Bloedorn et al 1993)

To associate the decision rules with a given decision-making task AQflf-2 provides a set of

features including 1) enabling the system to include in the decision structure nodes corresponding

to new attributes constructed during the process of learning the decision rules 2) controlling the

degree of generalization needed during the development of the decision structure 3) providing four

new criteria for selecting an attribute to be a node in the decision structure that allow the system to

generate many different but equivalent decision structures from the same set of rules 4) generating

unknown nodes in situations when there is insufficient infonnation for generating a complete

decision structure 5) learning decision structures from discriminant rules as well as

characteristic rules and 6) providing the most likely decision when the decision process stops

due to inability to evaluate an attribute associated with an intennediate node

6

To test the methooology of generating decision structures from decision rules an extensive set of

planned experiments have been designed to test different aspects of the approach The experiments

include testing different combinations of parameters for each sub-function of the approach

analyzing the relatioship between decision rules and the decision structure learned from them and

comparing decision trees learned by the AQUf2 system the well known C45 (Quinlan 1993)

system for learning decision trees from examples Different experiments were designed to examine

the new features of the AQUf approach for learning task-oriented decision structures The

experiments were applied to artificial domains as well as real-world domains including MONK-I

MONK-2 and MONK-3 (Thrun Mitchell amp Cheng 1991) East-West trains (Michie et al

1994) Engineering Design-wind bracings (Arciszewski et al 1992) Mushrooms Breast

Cancer (Mangasarian amp WolbeJg 1990) and Congressional bting Records of 1984 The

MONKs problems are concerned with learning classification rules for robot-like figures MONK-l

requires learning a DNF-type description MONK-2 requires learning a non-DNF-type description

(one that cannot be easily described as a DNF rule using the original attributes) MONK-3 concerns

learning a DNF rule from noisy data The East-West trains dataset is a structural domain that

classifies two sets of trains (Eastbound and Westbound) The Engineering Design-wind

bracing data involves learning conditions for applying different types of wind bracing for ta11

buildings The Mushrooms data is concerned with learning classification rules for distinguishing

between poisonous and non-poisonous mushrooms The Breast Cancer data involves learning

concept descriptions for recognizing breast cancer The congressional voting data includes voting

records on different issues AQUf2 outperformed C45 on the average with respect to both the

predictive accuracy and tree size for most problems Apf-2 did not work very well with noisy

data or with problems that have many rules covering very few examples

12 The Problem Statement

There are many limitations and problems accompanied using decision trees for decision-making

CHAPTER 2 RELATED RESEARCH

21 Learning Decision Trees from Decision Diagrams

The AQDT-2 method proposed here is based on the earlier work by Michalski (1978) which

introduced an algorithm for generating decision trees from decision lists The method proposed

several attribute selection criteria These criteria are of increasing power of the main criterion

order cost estimate (the nth order cost estimates n=l 2 ) Michalski also analyzed two

specific criteria MAL and DMAL for selecting the optimal attribute for a node in the tree

based on properties extracted from the decision diagram In order to better explain the method

it is necessary to define some terms These terms will be used later in the dissertation

Definition 2-1 A ~ of a decision class C is a set of rules whose union includes the set of all

examples of class C and does not include any examples of other decision classes

Definition 2-2 A ~ of a decision class C is disjoint if all rules (rules) are pairwise logically

disjoint In other words for any two rules there exists a condition with the same attribute but

with different values in each rule

Definition 2-3 An minimal cover is a disjoint cover which has the smallest number of rules

among all possible covers

Definition 2-4 A diagram for a given cover is a table constructed graphically by representing

in two dimension space of all possible combination of attributes values and locating on the

diagram all the condition parts of the given rules and marking them with the action specified

with each rule

Michalskis algorithm requires construction of minimalmiddot cover The minimal cover should be

consistent and complete The method is based on the fact that if there are n decision classes

any decision tree that correctly classifies the given rules and has n distinct leaves is a minimal

7

8

decision tree (any consistent decision tree should have at least n leaves) Michalski (1978) has

shown that if only one rule is broken by a selected attribute then instead of having one leaf

(which could potentially represent this rule or the decision class in the tree) there will have to

be at least two leaves representing this rule in the fmal decision tree

The attribute selection criterion MAL introduced in (Michalski 1978) prefers attributes that do

not break any rules or break as few as possible An attribute breaks a rule if the attribute can

divide the rule into two or more sub-rules Figure 2-1 shows two examples of two sets of rules

In the fJIst xl and x3 break at least two rules each (xl breaks the rule [x4=2] amp [x1=lv3] amp

[x3=2v3] and the rule [x4=3] amp [x1=1v2] amp [x3=1] and x3 breaks three rules the rule [x4=2]

amp [xl=2] amp [x3=2v3]the rule [x4=l] amp [xl=3] amp [x3=lv3] and the rule [x4=3] amp [xl=4] amp

[x3=2v3]) In the diagram on the right xl is the only attribute that does not break any rule

Figure 2-1 An example to illustrate how attributes break rules

One of the criteria defmed by Michalski is the first degree cost estimate which assigns to each

attribute an integer equal to the number of rules broken by that attribute This criterion is also

called the static cost estimate of an attribute or the criterion of minimizing added leaves

(MAL)

-

C3

9

The MAL criterion (Minimizing Added Leaves) seeks an attribute that minimizes the

estimated number of additional nodes in the decision tree being generated over a hypothetical

minimal decision tree When there is a tie between two attributes the attribute to be selected is

the one which breaks smaller rules (rules that cover fewer examples or more specialized

rules) AQDT-2 uses an approximate version of this criterion (the attribute dominance)

Another criterion introduced by Michalski was the DMAL criterion The DMAL criterion

(Dynamically Minimizing Added Leaves) is based on a principle similar to that of MAL but

is more complex because once an attribute is selected as a node in the tree some rules andor

parts of the broken rules at each branch are merged into one rule The DMAL ensures that the

value of the total cost estimate of an attribute is decreased by a value equal to the number of

merged rules minus one

Example Learn a decision tree from the following decision table

The minimal cover consists of the following rules

Al lt [x2=O] v [x1=0][x2=2] A2 lt [x2=I] v [xl=2][x2=2] A3 lt [xl=I][x2=2]

The evaluations ofthe MAL criterion for these attributes are 2 for xl 0 for x2 5 for x3 and 5

for x4 The attribute xl divides the two rules [x2=O] and [x2=I] so selecting it will add two

leaves to the optimal number of leaves It is clear that the attribute to be selected as a root of

10

the decision tree is x2 Then three branches are attached to the root node and the decision rules

are divided into subsets each corresponding to one branch For x2 = 0 or 1 a leave node is

generated For x2=2 another attribute is selected to be a node in the tree In this case xl has the

minimum MAL value Figure 2-2 shows the decision tree obtained using the MAL criterion

Al A3 A2

Figure 2-2 A decision tree learned from the decision table in Table 2-1

22 Learning Decision Trees from Examples

Decision tree learning is a field that concerned with generating decision tree that classifies a set

of examples according to the decision classes they belong to The essential aspect of any

inductive decision tree method is the attribute selection criterion The attribute selection

criterion measures how good the attributes are for discriminating among the given set of

decision classes The best attribute according to the selection criterion is chosen to be assigned

to a node in the tree The fIrst algorithm for generating decision trees from examples was

proposed by Hunt Marin and Stone (1966) Hunts algorithm uses a divide and conquer

algorithm for building decision trees This algorithm has been subsequently modified by

Quinlan (1979) and applied by many researchers to a variety of learning problems

Attribute selection criteria can be divided into three categories These categories are logicshy

based information-based and statistics-based The logic-based criteria for selecting attributes

use logical relationships between the attributes and the decision classes to determine the best

attribute to be a node in the decision tree such as the MAL criterion minimizing added leaves

(Michalski 1978) which uses conjunction and disjunction operators The information-based

criteria are based on the information theory These criteria measure the information conveyed

11

by dividing the training examples into subsets Examples of such criteria include the

information measure 1M the entropy reduction measure and the gain criteria (Quinlan 1979

83) the gini index of diversity (Breiman et al 1984) Gain-ratio measure (Quinlan 1986) and

others (Clark amp Niblett 1987 Bratko amp Lavrac 1987 Cestnik amp Karalic 1991) The

statistics-based criteria measure the correlation between the decision classes and the attributes

These criteria use statistical distributions for determining whether or not there is a correlation

The attribute with the highest correlation is selected to be a node in the tree Examples of

statistics-based criteria include Chi-square and G-statistic (Sokal amp Rohlf 1981 Han 1984

Mingers 1989a)

Niblett and Bratko (1986) Quinlan (1987) and Bratko and Kononeko (1987) extended the

method of learning decision trees to also handle data with noise (by pruning) Handling noise

extended the process of learning decision trees to include the creation of an initial complete

decision tree tree pruning which is done by removing subtrees with small statistical Validity

and replacing them by leaf nodes (MiDgers 1989b) More recently pruning has also been used

for simplifying decision trees even for problems without noise (Bohanec amp Bratko 1994)

Pruning decision trees improves their simplicity but reduces their predictive accuracy on the

training examples Quinlan (1990) proposed also a method to handle the unknown attributeshy

value problem by exploring probabilities of an example belonging to different classes

The rest of this section includes a brief description of the attribute selection criterion used by

C45 learning system (Quinlan 1993) C45 uses an information-based criterion for selecting

an attribute to be a node in the tree Also the section includes a brief description of the Chishy

square method for attribute selection (Mingers 1989a) The method is a statistics-based

method for selecting an attribute to be a node in the tree

221 Building Decision Trees Using htformation-based Criteria

12

This section presents a description of the inductive decision tree learning system C4S The

C4S learning system is considered to be one of the most stable accurate and fastest program

for learning decision trees from examples

Learning decision trees from examples requires a collection of examples Each example is

represented by a fixed number of attribute-value pairs C45 (Quinlan 1993) is a learning

program that induces classification decision trees from a set of given examples The C4S

learning system is descended from the learning system ID3 (Quinlan 1979) which is based on

Hunts method for constructing decision tree from a set ofcases (Hunt Marin amp Stone 1966)

The C45 system uses an attribute selection criterion called the Gain Ratio This criterion

calculates the gain in classifying information based on the residual information needed to

classify cases in a set of training examples and the information yielded by the test based on the

relative frequencies of the possible outcomes (decision classes) The gain ratio criterion is

based on an earlier criterion used by ID3 called the Gain Criterion The Gain Criterion uses

the frequency of each decision class in the given set of training examples

Once an attribute is chosen to be a node in the tree the system generates as many links as the

number of its values and classifies the set of examples based on these values If all the

examples at a certain node belong to one decision class the system generates a leaf node and

assigns it to that class Otherwise the system searches for another attribute to be a node in the

tree

The Gain Criterion The gain criterion is based on the information theory That is the

information conveyed by a message depends on its probability and can be measured in bits as

minus the logarithm (base 2) of that probability To explain the gain criterion suppose that for

a given problem Xu bullbull X are given attributes and Cu c are decision classes Suppose S is

any set of cases and T is the initial set of training cases The frequency of class C1 in the set S

is the number of examples in S that belong to class C i bull

13

freq(Cit S) = Number of examples in S belong to CI (2-1)

Suppose that lSI is the total number of examples in S the probability that an example selected

at random from S belongs to class Ci is freq(CIbullS) IISI

The information conveyed by the message that a selected example belongs to a given decision

class Cit is determined by -log1 (freq(Ch S) IISI) bits

The expected information from such a message stating class membership is given by

info(S) = -k

L (freq(C S) I lSI) log1 (freq(C S) IISI) bits (2-2)I

info(S) is known also as the entrQpy of the set S When S is the initial set of training examples

info(T) determines the average amount of information needed to identify the class of an

example in T

Suppose that we selected an attribute X to be the root of the tree and suppose that X has k

possible values The training set T will be divided into k subsets each corresponding to one of

Xs values The expected information of selecting X to partition the training set T infoz(T) can

be found as the sum over all subsets of multiplying the information conveyed by each subset

by its probability

k

infoz(T) =L (ITII 1m) infO(TIraquo (2-3) I

The information gained by partitioning the training examples T into subset using the attribute

X is given by

gain(X) =inlo(T) - inoIT) (2-4)

The attribute to be selected is the attribute with maximum gain value

The Gain Ratio Criterion This criterion indicates the proportion of information generated by

the split that appears helpful for classification Quinlan (1993) pointed out that the gain

criterion has a serious deficiency Basically it is strongly biased toward attributes with many

outcomes (values) For example for any data that contains attributes such as social security

14

number the gain criterion will select that attribute to be the root of the decision tree However

selecting such attributes increases the size of the decision tree Quinlan provided a solution to

this problem by introducing the gain ratio criterion which takes the ratio of the information that

is gained by partitioning the initial set of examples T by the attribute X to the potential

information generated by dividing T into n subsets

Following similar steps to obtain the information conveyed by dividing T into n subsets The

expected information generated by dividing T into n subsets and by analogy to equation 2-2 is

determined by

split info(T) = - Lt

(ITII lTI) 10gl (ITII m ) (2-5) I

The gain ratio is given by

gain ratio(X) = gain(X) I split info(X) (2-6)

and it expresses the proportion of information generated by the split that is useful for

classification

Example Consider the following example presented by Quinlan (1993) Table 2-2 shows the

set of training examples

First determine the amount of information gained by selecting the attribute outlook to be a

root of the decision tree This attribute divides the training examples into three subsets

sunny-with five examples two of which belong to the class Play overcast-with four

examples all of which belong to the class Play and rain-with five examples three of

which belong to the class Play To determine info(11 the average information needed to

identify the class of an example in T there are 14 training examples and two decision classes

Nine of these examples belong to the class Play and five belong to the class Dont Play

Info(11 = - 914 10gl (914) - 514 10gl (514) = 094 bits

When using outlook to divide the training examples the information becomes

15

infO(T) = 514 -25 log2 (25) - 35 10g2 (35)

+ 414 -44 10g2 (44) - 04 log2 (04)

+ 514 -35 logl (35) - 25 logl (25) = 0694 bits

By substituting in equation 2-4 the gain of information results from using the attribute

outlook to split the training examples equal to 0246 The gain information for windy is

0048

Figure 2-3 shows a decision tree learned for this problem using the gain criterion The split

information for outlook is determined as follows

split info(T) = - 514 logl (514) - 414 log2 (414) - 514 log2 (514) = 1577 bits

The gain ratio for outlook = 02461577 =0156

16

Figure 2 -3 A decision tree learned using the gain criterion for selecting attributes

The C45 system handles discrete values as well as continuous values To handle an attribute

with continuous values C45 uses a threshold to transform the continuous domain into two

intervals In other words for each continuous attribute C45 generates two branches one

where the values of that attribute is greater than the determined threshold and the other if the

value is less than or equal to the threshold

Tree pruning in C45 is a process of replacing subtrees with small classification validity by

leaves The C45 system uses Laplace ratio for determining the error rate of different subtrees

This ratio is defined as (e+l) (n+2) where n is the number of the training examples and e is

the number of misclassified examples at a given leaf

222 Building Decision Trees Using Statistics-based Criteria

The Chi-square method for selecting attributes was used by Hart (1984) and Mingers (1989a)

in building decisionmiddot trees The method uses Chi-square statistics to measure the association

between two attributes When building decision trees the method is implemented such that it

determines the association between each attribute and the decision classes The attribute to be

selected is the one with greatest value

To determine the Chi-square value for an attribute consider aij is the number of examples in

class number i where the attribute A takes value numberj In other words aij is the frequency

of the combination of decision class number i and the attribute value number j The Chi-square

value for attribute A is given by

n m

Chi-square (A) =L L [ (aij - Eijl2 Eij ] (2shy

i=1 j=1

where n is the number of decision classes m is the number ofvalues of a given attribute Also

17

Eij = (TCi TVj ) T (2shy

8)

where T Ci and T Vj are the total number of examples belong to the decision class a and the total

number of examples where the attribute A takes value vj respectively T is the total number of

examples

Consider Quinlans example in Table 2-2 Table 2-3 shows the frequency of different

combination of values between the decision class and both the outlook and the Windy

attributes Table 2-4 shows the expected values TCi and T Vj of the frequencies in Table 2-3

of different attributes values for different decision classes

To determine the association value between the decision classes and both the attribute Windy

and the attribute Outlook the observed Chi-square values are

Chi-square (Windy Class) =[(3-39)239] + [(3-21)221] + [(6-51)232] + [(2-29)232]

=021 + 039 + 025 + 025 = 11

Chi-square (Outlook Class) = [(2-32)232] + [(4-26)226] + [(3-32)232] + [(3-18118]

+ [(0-14)214] + [(2-18)2 18] =045 + 075 + 02 + 08 + 14 + 002 = 362

18

Applying the same method to the other attributes the results will favor the attribute Outlook

Once that attribute is selected to be a node in the tree the remaining set of examples are divided

into subsets and the same process is repeated on each subset

Table 2-5 shows a summary of these criteria and their basic evaluation function

Table 2-5 Attribute selection criteria and their basic evaluation measure

Info Measure (IM) Gain Gshystatistics and Gain Ratio Chi-square

II

Entropy(S) = - L (freq(~ 5) 1151) logz (freq(CIt 5) 1151 ) I

G-Statistics =2N 1M (N-No of examples) n m

Chi-square (A B) = L L [(aij - ~j)21Eij ] i=l

223 Analysis of Attribute Selection Criteria

This subsection introduces briefly the analysis of different selection criteria which was done by

Mingers (1989a) Mingers compared six attribute selection criteria that are used in decision

tree programs These criteria are the Information Measure (IM) Chi-square G statistics Gin

index of diversity Marshal correction and Gain RaJio The overall results show that the Gain

Ratio criterion had the strongest xesults

In the flISt experiment Minger tested the six criteria on a problem with ambiguous examples

(ie examples may belong to more than one decision class) to observe how the selected criteria

evaluate the given attributes The problem has two decision classes and two attributes X and

Y It was assumed that attribute X is better for classifying the examples than attribute Y The

training examples were unevenly spread between the two values of X Attribute Y has three

values and the examples were spread randomly among them Table 2-6 (a and b) shows the

contingency tables for both attributes Table 2-7 shows a summary of the goodness of split

provided by the six criteria Mingers noted that the measures that are not based on information

19

theory give radiation (attribute X here) less weighL This may be because the zero in the fIrst

row of radiation has a greater influence in the log calculation In the case of using the Chishy

square criterion the value zero adds the maximum association between any two attributes

because the Chi-square value of a zero cell is the expected value of this cell

b)

Now let us demonstrate results from another experiment done by Mingers In this experiment

Mingers used four different data sets to generate decision trees for eleven different criteria In

the final results he compared the total number of nodes and the total error rate provided by

each criterion over all given problems Table 2-8 shows the final results for five selected

criteria only

20

U lgt ~middot results for comparing the total accuracy and size of decision trees different attribute selection criteria from four VVLU

This experiment was performed on four real world data sets These data are concerned with

profiles of BA Business Studies degree students recurrence of breast cancer classifying types

of Iris and recognizing LCD display digits The data was divided randomly 70 for training

and 30 for testing For more details see Mingers (1989a)

23 Learning Decision Structures

Considering the proposed defInition of decision structures given above two related lines of

research are described in this section Gaines (1994) and Kohavi (1994) proposed two

approaches for generating decision structures that share some of the earlier ideas by Imam and

Michalski (1993b)

In the first approach Brian Gaines introduces a method for transforming decision rules or

decision trees into exceptional decision structures The method builds an Exception Directed

Acyclic Graph (EDAG) from a set of rules or decision trees The method starts by assigning

either a rule say RO or a conclusion say CO or both to the root node If it assigns a conclusion

to the root node it places it on a temporary conclusion list Then it generates a new child node

and add to it a new rule say Rl The method evaluates the new rule Rl if it satisfies the

conclusion CO in its parent node it builds a new child and repeats the processes Otherwise if

the rule Rl does not satisfy the conclusion CO it changes the temporary memory with the new

conclusion Cl which is satisfied by the rule RO The same process is repeated until all rules

that have common conditions with the rule at the root are evaluated The method then creates a

21

new child node from the root and repeat the process until all rules are evaluated In the decision

structure nodes containing rules only represent common conditions to all of its children

The main disadvantages of this approach is that it requires discriminant rules to build such a

decision structure Also such a structure is more complex than the traditional decision trees

that are used for decision-making

Figure 2-4 shows an example of set of rules and their equivalent exceptional decision structure

The decision structure can be read as follows It is Safe except if xl=1 amp x2=1 amp x3=1 and

either x4=3 or x5=I then it is Lost except if x6=1 it is Safe except if x7=1 it is Lost

The second approach introduced by Ronny Kohavi learns decision structures from examples

using a bottom-up algorithm The method uses a greedy Hill-climbing algorithm for inducing

Oblivious read-Once Decision Graphs (HOODO) A read-once decision graph is defined as a

decision graph where each attribute occurs at most once along any computational path In other

words for each path from the root to the leaves of the decision structure attributes may occur

as a node once at most However there may be more than one node with the same attribute in

the decision graph Kohavi also defined a leveled decision graph in which the nodes are

partitioned into a sequence of pairwise disjoint sets such that outgoing edges from each level

terminate at the next level An oblivious decision graph is a decision graph where all nodes at

a given level are labeled by the same attribute

22

Safe lt [xl=2] Safe lt [x2=2] Safe lt [x3=2] Safe lt [x4=1] amp [x5=2] Safe lt [x4=1] amp [x5=3] Safe lt [x6=l] amp [x7=2] Safe lt [x6=1] amp [x7=3] Safe lt [x4=2] amp [x5=2] Safe lt [x4=2] amp [x5=3]

Lost lt [x6=2] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x6=3] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x5=1] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=2] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=3] amp [x3=1] amp [x2=I] amp [xl=l]

The algorithm starts by generating a leaf node for each decision class Then it applies a

nondeterministic method to select an attribute to build new level of the decision structure This

attribute is removed from the data and the data is divided into subsets each corresponding to a

complex combination of that attributes values For each subset the process is repeated until all

the examples of a given subset belong to one decision class For example if the selected

attribute say A has two values (0 and 1) and there are two decision classes (CO and CI) The

data is divided into two subsets the fIrst subset contains examples where A takes value 0 and

belong to class CO or takes value 1 and belong to class Cl The second subset of examples is

the set where A takes value 0 and belong to class CI or takes value 1 and belong to class CO

The number of nodes of the fIrst level (after the leave nodes) is expected to be less than or

equal to ka where k is the number of decision classes and n is the number of values of the

selected attribute and it can increase exponentially before that number is reduced

exponentially to one

It is easy for the reader to fIgure out some major disadvantages of such approach including

The average size of such decision structures is estimated to be very large especially when there

23

is no similarity (ie strong patterns) or logical relationship in the data The time used to learn

such decision structure is relatively very high comparing to systems for learning decision trees

from examples and fmally it could be better to search for attribute which reduces the number

of generated subsets of the data instead of nondeterministically selecting an attribute to build a

new level of the decision structure Kohavi provided a comparison between the C45 system

and his system that can be found in (Kohavi 1994) Later Kohavi and Li (1995) used a

deterministic method for attribute selection which minimizes the width of the penultimate level

of the graph

Table 2-9 shows a comparison between the proposed approach and those two approaches The

EDAG and HOODG systems are unreleased prototype systems

or Decision structures are easy to Decision structures are Decision structures are easy to understand difficult to read understand

CHAPTER 3 DESCRIPTION OF THE APPROACH

31 General Methodology

In the proposed approach the function of learning or discovery is separated from the function of

using the discovered knowledge for decision-making The frrst function is performed by an

inductive learning program that searches for knowledge relevant to a given class of decisions

and stores the learned knowledge in the form of decision rules The second function is performed

when there is a need for assigning a decision to new data points in the database (eg bull a

classification decision) by a program that transforms obtained knowledge into a decision

structure optimized according to the given decision-making situation

The Learning Task GivenA set of training examples describing the concept to be learned Learning goal which specifies the decision classes to be learned from the training examples A background knowledge to control the learning process Determine A concept description in a declarative form of knowledge (decision rules) that satisfies the learning goal

The Decision-making Task GivenA set of decision rules in a conjunctive form A description of the new decision-making situation (eg attributes costs and order preference importance or frequency of decision classes etc) One or more examples that needed to be tested under the given decisionshymaking situation A set of parameters to control the learning process Determine A decision structure that suits the given decision-making situation

The decision rules used here are learned by either the AQ1S (Michalski et al 1986) or AQ17

(Bloedorn et aI 1993) learning systems The reasons for selecting decision rules as a form of

23

24

declarative knowledge are that they do not impose any order on the evaluation of the attributes

and due to the lack of order constraints decision rules can be evaluated in many different

ways which increases the flexibility of adapting them to the different tasks of decision-making

(Bergadano et al 1990) Since the number of rules learned per class is typically much smaller

than the number of examples per class generating a decision structure from decision rules can

potentially be done on line

Such virtual decision structures can be tailored to any given decision-making situation The

needed decision rules have to be generated only once and then they can be used many times for

generating decision structures according to changing requirements of decision-making tasks The

method uses the AQDT-2 system (Imam amp Michalski 1994) for learning decision structures

from decision rules Decision structures represent a procedural form of knowledge which makes

them easy to implement but also harder to change Consequently decision structures can be quite

effective and useful as long as they are used in decision-making situations for which they are

optimized and the attributes specified by the decision structure can be measured without much

cost Figure 3-1 shows an architecture of the proposed methodology

Data

Decision -

- Learning knowledge from database The decision making process

Figure 3 -1 Architecture of the AQDT approach

25

It is assumed that the database is not static but is regularly updated A decision-making problem

arises when there is a case or a set of cases to which the system has to assign a decision based on

the knowledge discovered Each decision-making situation is defmed by a set of attribute-values

Some attribute-values may be missing or unknown A new decision structure is obtained such

that it suits the given decision-making problem The learned decision structure associates the

new set of cases with the proper decisions

32 A Brief Description of the AQIS and AQ17 Rule Learning Programs

The decision rules are generated from examples by an AQ-type inductive learning system

specifically by AQ15 (Michalski et albull 1986) or AQI7-DCI (Bloedorn at al 1993) The rest of

this subsection includes a brief description of the inductive learning systems AQ15 and AQI7

AQ15 learns decision rules for a given set of decision classes from examples of decisions using

the STAR methodology (Michalski 1983) The simplest algorithm based on this methodology

called AQ starts with a seed example of a given decision class and generates a set of the

most general conjunctive descriptions of the seed (alternative decision rules for the seed

example) Such a set is called the star of the seed example The algorithm selects from the star

a description that optimizes a criterion reflecting the needs of the problem domain If the

criterion is not defined the program uses a default criterion that selects the description that

covers the largest number of positive examples (to minimize the total number of rules needed)

and with the second priority that involves the smallest number of attributes (to minimize the

number of attributes needed for arriving at a decision)

If the selected description does not cover all examples of a given decision class a new seed is

selected from uncovered examples and the process continues until a complete class description

is generated The algorithm can work with few examples or with many examples and can

optimize the description according to a variety of easily-modifiable hypothesis quality criteria

26

The learned descriptions are represented in the form of a set of decision rules expressed in an

attributional logic calculus called variable-valued logic 1 or VLl (Michalski 1973) A

distinctive feature of this representation is that it employs in addition to standard logic operators

the internal disjunction operator (a disjunction of values of the same attribute in a condition) and

the range operator (to express conditions involving a range of discrete or continuous values)

These operators help to simplify rules involving multi valued discrete attributes the second

operator is also used for creating logical expressions involving continuous attributes

AQ15 can generate decision rules that represent either characteristic or discriminant concept

descriptions depending on the settings of its parameters (Michalski 1983) A characteristic

description states properties that are true for all objects in the concept The simplest

characteristic concept description is in the form of a single conjunctive rule (in general it can be

a set of such rules) The most desirable is the maximal characteristic description that is a rule

with the longest condition part ie stating as many common properties of objects of the given

class as can be determined A discriminant description states properties that discriminate a given

concept from a fixed set of other concepts The most desirable is the minimal discriminant

descriptions that is a rule with the slwnest condition part For example to distinguish a given

set of tables from a set of chairs one may only need to indicate that tables have large flat top

A characteristic description of the tables would include also properties such as have four legs

have no back have four corners etc Discriminant descriptions are usually much shorter than

characteristic descriptions

Another option provided in AQ15 controls the relationship among the generated descriptions

(rulesets or covers) of different decision classes In the IC (Intersecting Covers) mode

rulesets of different classes may logically intersect over areas of the description space in which

there are no training examples In the DC (Disjoint Covers) mode descriptions of different

classes are logically disjoint The DC mode descriptions are usually more complex both in the

number of rules and the number of conditions There is also a DL mode (a Decision List mode

27

also called VL modeM-for variable-valued logic mode) in which the program generates rule sets

that are linearly ordered To assign a decision to an example using such rulesets the program

evaluates them in order If ruleset is satisfied by the example then the decision is made

otherwise the program proceeds to the evaluation of the ruleset +1 In IC and DC modes

rule sets can be evaluated in any order

Alternatively the system can use rules from the AQ17-DCI program for Data-driven

Constructive Induction AQ17-DCI differs from AQ15 mainly in that it contains a module for

generating additional attributes These attributes are various logical or mathematical

combinations the original attributes The program generates a large number of potential new

attributes and seleets from them those most promising based on an attribute quality criterion

To illustrate the format of rules generated by AQ15 (or AQ17-DCI) an exemplary ruleset is

shown in Figure 3-2 The ruleset (that can be re-represented as a disjunctive normal form

expression) describes a voting record of Democratic Representatives in the US Congress

Each rule is a conjunction of elementary conditions Each condition expresses a simple relational

statement For example the condition [State = northeast v northwest] states that the attribute

State (of the Representative) should take the value northeast or northwest to satisfy the

condition

Rl [Gas_concban = yes] amp [Soc_see_cut = no v not registered] R2 [Draft = Yes v not registered]amp[AlaslaLparks = yes v not registered] amp

[Food_stamp_cap = no] amp [State = northeast v northwest] R3 [Chrysler =yes v not registered] amp [Income =low] R4 [Education =yes] amp [Occupation = yes]

Figure 3-2 A ruleset generated by AQ15 for the concept Voting pattern of Democratic Representatives

The above rules were generated from examples of the voting records For illustration below is

an example of a voting record by a Democratic representative

Draft registration=no Ban of aid to Nicaragua=no Cut expenditure on mx missiles=yes Federal subsidy to nuclear power stations=yes Subsidy to national parks

28

in Alaska=yes Fair housing bill=yes Limit on Pac contributions=yes Limit onood stamp program=no Federal help to education=no StateFrom=north east State Population=large Occupation =unknown Cut in social security spending=no Federal help to Chrysler corp=not registered

By expressing elementary statements in the example as conditions and linking conditions by

conjunction the examples can be re-expressed as decision rules Thus decision rules and

examples formally differ only in the degree of generality

33 Generating Decision Structurestrees from Decision Rules

This section describes the AQDT-1 system for learning decision structures (trees) from decision

rules (Imam amp Michalski 1993ab) Also a description of the AQDT-2 method for learning taskshy

oriented decision structure from decision rules is included and fmally the methodology is

illustrated by two examples

Methods for learning decision trees from examples have been very popular in machine learning

due to their simplicity Decision trees built this way can be quite efficient as long as they are

used in decision-making situations for which they are optimized and these situations remain

relatively stable Problems arise when these situations significantly change and the assumptions

under which the tree was built do not hold anymore For example in some situations it may be

difficult to determine the value of the attribute assigned to some node One would like to avoid

measuring this attribute and still be able to classify the example if this is potentially possible

(Quinlan 1990) If the cost of measuring various attributes changes it is desirable to restructure

the tree so that the inexpensive attributes are evaluated f11st A tree restructuring is also

desirable if there is a significant change in the frequency of occurrence of examples from

different classes A restructuring of a decision tree to suit the above requirements is however

difficult to do The reason for this is that decision trees are a form of decision structure

representation and imposes constraints on the evaluation order of the attributes that are not

logically necessary

29

One problem in developing a method for generating decision structures from decision rules is to

design an attribute selection criterion that is based on the properties of the rules rather than of

the training examples A decision rule normally describes a number of possible examples Only

some of them are examples that have actually been observed ie training examples An attribute

selection criterion is needed to analyze the role of each attribute in the rules It cannot be based

on counting the numbers of training examples covered by each attribute-value and the

frequency of decision classes in the training examples as is done in learning decision trees from

examples because the training examples are assumed to be unavailable

Another problem in learning decision trees from decision rules stems from the fact that decision

rules constitute a more powerful knowledge representation than decision trees They can directly

represent a description in an arbitrary disjunctive normal form while decision trees can represent

directly only descriptions in the disjoint disjunctive normal form In such descriptions all

conjunctions are mutually logically disjoint Therefore when transforming a set of arbitrary

decision rules into a decision tree one faces an additional problem of handling logically

intersecting rules

The solution to both problems (attribute selection and logically intersected rules) in the AQDT-2

system is based on the earlier work by Michalski (1978) which introduced a general method for

generating decision trees from decision rules The method aimed at producing decision trees

with the minimum number of nodes or the minimum cost (where the cost was defined as the

total cost of classifying unknown examples given the cost of measuring individual attributes and

the expected probability distribution of examples of different decision classes) More

explanations are provided in the following section

331 Tbe AQDT-2attribute selection metbod

This section describes the AQDT -2 method for building a decision structure from decision rules

The method for building a single-parent decision structure is similar to that used in standard

30

methods of building a decision tree from examples The major difference is that it assigns tests

(attributes) to the nodes using criteria based on the properties of the decision rules (this includes

statistics about the examples covered by each rule in the case of learning rules from examples)

rather than statistics characterizing the frequency of training examples per decision classes per

attribute-values or per conjunctions of both Other differences are that the branches may be

assigned an internal disjunction of values (not only a single value as in a typical decision tree)

and leaves may be assigned a set of alternative decisions with probabilities Also the tests can be

attributes or names standing for logical or mathematical expressions that involve several

attributes or variables In the following we use the terms test and attribute interchangeably

(to distinguish between an attribute and a name standing for an expression the latter is called a

constructed attribute)

At each step the method chooses the test from an available set of tests that has the highest utility

(see below) for the given set of decision rules This test is assigned to the node The branches

stemming from this node are assigned test values or disjoint groups of values (in the form of

logical disjunction if such occur in the rules subsumed groups of values are removed) Each

branch is associated with a reduced set of rules determined by removing conditions in which the

selected attribute assumes value(s) assigned to this branch If all rules in the reduced ruleset

indicate the same decision class a leaf node is created and assigned this decision class The

process continues until all nodes are leaf nodes If it is not possible to reduce further the rule set

because some attribute is declared as unavailable (infmite cost) then the leaf is assigned a set of

candidate decisions with associated probabilities (see Sec 42)

The test (attribute) utility is a combination of one or more of the following elementary criteria 1)

cost which indicates the cost of using each attribute for making decision 2) disjointness which

captures the effectiveness of the test in discriminating among decision rules for different decision

classes 3) importance which determines the importance of a test in the rules 4) value

31

distribution which characterizes the distribution of the test importance over its of values and 5)

dominance which measures the test presence in the rules These criteria are dermed below

~ The cost of a test expresses the effort or cost needed to measure or apply the test

Djsjointness The disjointness of a test is defined as the sum of the class disjointness-the

disjointness of the test for each decision class Suppose decision classes are Cl C2 bull Cm and

decision rule sets for these classes have been determined Given test A let V 1 V 2 bull V m denote

sets of the values (outcomes) of A that are present in rule sets for classes C 1 C2bullbull Cm

respectively If a ruleset for some class say Ct contains a rule that does not involve test A than

Vt is the set of all possible values of A (the domain of A)

Definition 3 -1 The degree ofclass disjointness D(A Ci ) of test A for the ruleset ofclass Ci is

the sum of the degrees of disjointness D(A Cit Cj) between the ruleset for Ci and rulesets for

Cj j=l 2 m j J L The degree of disjointness between the ruleset for Ci and the ruleset for Cj is

dermed by

if Vi Vj ifVj gtVj (3-1) ifVinVj -tljgt amp -tVi amp -tVj ifVinVj=1jgt

where cjgt denotes the empty set Note that exchanging the second and third conditions of equation

(3-1) may seem to be an improved criteria However it does not clearly distinguish between both

cases (Le for both situations the disjointness will be similar) The current equation is better

because it gives higher scores to attributes that classify different subsets of the two decision

classes than attributes that classify only a subset of one decision class

Definition 3-2 The disjointness of the test A for evaluating a given set of decision rules is the

sum of the degrees of class disjointness of each decision class

32

m m Disjointness(A) = L D(A cn where D(A cn = L D(A Ci Cj) (3-2)

i=l i=l i~j

The disjointness of a test ranges from 0 when the test values in rulesets of different classes are

all the same to 3m(m-l) when every rule set of a given class contains a different set of the test

values If two tests have the same disjointness value the attribute to be selected is the one with

less number of values

Definition The A vera~ Number of Tests (ANT) required to make decision from a decision tree

is dermed as the average number of tests (attributes) to be examined from the root of the tree to

any leaf node in order to reach a decision

Definition A decision structure is a one-node-per-level decision structure if at each level there is

only one node and zero or more leaves

Such decision structure can be generated by combining together all branches that associated with

each one a set of decision rules belongs to more than one decision class

Theorem 1 Consider learning one-node-level decision structure for a database with two

decision classes The disjointness criterion ranks first attributes that add minimum number of

tests to the decision tree

Proof Consider all possible distributions of an attributes values within two decision classes Ci

and Cj There are three cases 1) subset (same as super set) 2) non-empty intersection but not

subset 3) no intersection Figure 3-3 shows all possible distributions (the case of having the

same set of values in both classes is trivial one) Consider that branches leading to one subset

with the same decision class are combined into one branch In the first case there will be two

branches only The first leads to a leaf node and the other leads to an intermediate node where

another attribute to be selected The minimum ANT in this case is 53 In the second case three

branches should be created Two branches leads to a leaf node where all values at each branch

belong to only one and different decision class The third branch leads to an intermediate node

33

where another attribute should be selected furtur classifies the decision classes The minimum

ANT in this case is 64 In the third case only two branches will be generated where each leads

to leaf node with different decision class In this case the minimum ANT is 1

Figure 3-4 shows decision trees equivalent to selecting the corresponding attribute Note that in

case of having more than one attribute-value at branches lead to leaves belonging to one decision

class they will be combined into one branch in the decision structure The symbol 1 means

that an attribute is needed to classify the two decision classes In such cases there will be at least

two additional paths

D(A Ci) = 0 D(A CJ) = 1 D(A Ci) =2 D(A Cj) = 2 D(A Ci) =3 D(A Cj) =3

Figure 3-3 Yen Diagrams of Possible combinations of attribute values in two decision classes

The average number of tests required for making a decision in each possible case is determined

in Figure 3-4 It is clear that in the case of two decision classes the disjointness criterion ranks

highly attributes that reduces the average number of tests required for decision-making The

theorem can be proved in the general case

i ~ i Cj C1middot bull Ci Cj

ANT=32middotANT=53 ANT=l

1 means at least one attribute is needed to complete the decision tree Figure 3-4 Decision trees corresponding the Yen diagrams inFigure 3-3

34

Theorem 2 The attribute with the highest non~zero disjointness is the best attribute to classify

the decision classes

Proof Suppose that the number of decision classes is n Assume also that there are two attributes

A and B where D(A) lt D(B) Since the disjointness criterion considers the mutual relationship

between any two decision classes this means that there are more decision classes where D(A

Ci) lt D(B Ci) than those where D(A Ci) gt D(B Ci) (ignoring classes with equal disjointness

for bCth attributes) Hence there are more decision classes where D(A Ci Cj) lt D(B Ci Cj)

than those where D(A Ci Cj) gt D(B Cit Cj) Let us frrst prove that if D(A Ci Cj) lt D(B Cit

Cj) then B classifies the decision classes better than A

For each pair of decision classes Ci and Cj the possible values of the disjointness of any attribute

are 0 14 or 6 For all positive values ofD(B)= 14 or 6 it is clear that attribute B should have

smaller ANT than attribute A which has lower disjointness This means that if D(A Cit Cj) lt

D(B Cit Cj) then attribute B can classify both decision classes better than attribute A

Similarly B is better for classifying more pairs of decision classes than A This implies that B is

a better classifier than A

Importance The second elementary criterion the importance of a test is based on the

importance score (IS) introduced in (Imam Michalski amp Kerschberg 1993) In the obtained

rules each test is assigned a score that represents the total number of training examples that

are covered by the rules involving this test Decision rules learned by an AQ learning program

are accompanied with information on their strength Rule strength is characterized by t-weight

and u~weight The t-weight (total-weight) of a rule for some class is the number of examples of

that class covered by the rule The importance score of a test is the aggregation of the totalshy

weights of all rules that contain that test in their condition part Given a set of decision rules for

m decision classes Cl Cm and involving n tests AlbullAn and the number of rules associated

with class Ci is denoted by tlrit The importance score is defmed as follow

35

Definition 3 -3 The importance score IS(Aj) of the test A j is detennined by

m IS(Aj)= L IS(Aj Ci) (3-31)

i=l

where ri IS(Aj Ci) = L Rik(Aj) (3-32)

k=l

and Rik the weight of a test Aj in the rule Rk of class Ci is given by

t - weight if A j belongs to rule Rik Rik (Amiddot) = (3-4)J 0 otherwise

where i=I n ik=I ri j=I m

The importance score method has been separately compared as a feature selection method with

a genetic algorithm-based method (Imam amp Vafaie 1994) The importance score method

produced an equal or higher accuracy on three real-world problems than those reported by the

GA method while selecting fewer attributes In addition the IS method was significantly faster

than the GA

yalue distribution The third elementary criterion value distribution concerns the number of

legal values of tests Given two tests with equal importance scores this criterion prefers the test

with the smaller number of legal values Experiments have shown that this criterion is especially

useful when using discriminant decision rules

Definition 34 A value distribution VD(Aj) of a test A j is defined by

VD(Aj) = IS(Aj) I Vj (3-5)

where v is the number of legal values of Aj

Dominance The fourth elementary criterion dominance prefers tests that appear in large

numbers of rules as this indicates their high relevance for discriminating among the rulesets of

given decision classes Since some conditions in the rules have values linked by internal

disjunction counting such rules directly would not reflect properly their relevance Therefore

36

for computing the dominance the rules are counted as if they were converted to rules that do not

have internal disjunction Such a conversion is done by multiplying out the condition parts of

the rules containing internal disjunction For example the condition part [x3=1 v 3]amp[x4=1] is

multiplied out to two rules with condition parts [x3=1]amp[x4=1] and [x3=3]amp[x4=I]

The above criteria are combined into one general test measure using the lexicographic

evaluation functional with tolerances (LEF) (Michalski 1973) LEF is a list of some or all of

the above elementary criteria each associated with a Ittolerance threshold It in percentage The

criteria are applied to tests in the order defined by LEF A test passes to the next criterion only if

it scores on the previous criterion within the range defined by the tolerance (from the top value)

The default LEF is

ltCost d Disjointness t2 Importance t3 Value distr t4 Dominance 0gt (3-6)

where d a t3 t4 and t5 are tolerance thresholds (in percentages) their default values are O

The default value of the cost of each test is 1

The above LEF ranks attributes this way First attributes are evaluated on the basis of their cost

If two or more attributes have the same cost they ranked by their degree of disjointness If two

or more attributes still share the same top score or their scores differ less than the assumed

tolerance threshold t2 the method evaluates these attributes using the second (Importance)

criterion If again two or more attributes share the same top score or their scores differ less than

the tolerance threshold t3 then the third criterion normalized IS is used then similarly the

fourth criterion (dominance) If there is still a tie the method selects the best attribute

randomly

If there is a non-uniform frequency distribution of examples of different classes then the

selection criterion uses a modified definition of the disjointness Namely the previously defmed

37

disjointness for each class is multiplied by the frequency of the class occurrence The class

occurrence is the expected number of future examples that are to be classified to a given class

m

Disjointness(A) = L D(A CO Frq(Ci) (3-7) i=l

where Frq(Cn is the expected frequency of examples of class Cit and it is assumed to be given

by the user The attribute ranking criteria in this case is defined by the LEF

ltCost d Disjointness t2 Importance 13 Normalized-IS 14 Dominance 15gt (3-8)

where the Cost denotes the evaluation cost of an attribute and is to be minimized while other

elementary criteria are treated the same way as in the default version

332 The AQDT-2 algorithm

The AQDT-2 algorithm constructs a decision structure from decision rules by recursively

selecting at each step the best test according to the ranking criteria described above and

assigning it to the new node The process stops when the algorithm creates terminal branches that

are assigned decision classes To facilitate such a process the system creates a special data

structure for each concept description (ruleset) This structure has fields such as the number of

rules the number of decision classes and the number of attributes present in the rules A set of

pointers connect this data structure to set of data structures each represents one decision class

The decision class structure contains fields with information on the number of rules belong to

that class the frequency of the decision class etc It is also connected to a set of data structures

representing the decision rules within each decision class The system creates independently a set

of data structures each is corresponding to one attribute Each attribute description contains the

attributes name domain type the number of legal values a list of the values the number of

rules that contain that attribute and values of that attribute for each rule The attributes are

arranged in an array in a lexicographic order flISt in the descending order of the number of rules

38

that contain that attribute and second in the ascending order of the number of the attributes

legal values

The system can work in two modes In the standard mode the system generates standard

decision trees in which each branch has a specific attribute-value assigned In the compact

mode the system builds a decision structure that may contain

A) or branches Le branches assigned an internal disjunction of attribute-values whenever it

leads to simpler structures For example if a node assigned attribute A has a branch marked by

values 1 v 2 then the control passes along this branch whenever A takes value 1 or 2 The

program creates or branches on the basis of the analysis of the value sets Vi while computing

the degree of attribute disjointness

B) nodes that are assigned derived attributes that is attributes that are certain logical or

mathematical combinations of the original attributes To produce decision structures with

derived attributes the input decision rules are generated by AQ17 (rather than AQI5) The

AQ17 rules may contain conditions involving attributes constructed by the program rather than

those originally given

To generate decision structures from rules the AQDT-2 method prefers either characteristic or

discriminant disjoint rule descriptions (given by an expert or learned by a system) Disjoint rules

are more suitable for building decision structures Assume that the description of each class is in

the form of a ruleset and that this set is the initial ruleset context The AQDT algorithm is

The AQDT2 Algorithm

Step 1 Evaluate each attribute occurring in the ruleset context using the LEF attribute ranking

measure Select the highest ranked attribute Let A represents this highest-ranked

attribute

Step 2 Create a node of the tree (initially the root afterwards a node attached to a branch) and

assign to it the attribute A In standard mode create as many branches from the node as

39

there are legal values of the attribute A and assign these values to the branches In

compact mode (decision structures) create as many branches as there are disjoint value

sets of this attribute in the decision rules and assign these sets to the branches

Step 3 For each branch associate with it a group of rules from the ruleset context that contain a

condition satisfied by the value(s) assigned to this branch For example if a branch is

assigned values i of attribute A then associate with it all rules containing condition [A=

i v ] If a branch is assigned values i v j then associate with it all rules containing

condition [A= i v j v ] Remove from the rules these conditions If there are rules in

the ruleset context that do not contain attribute A add these rules to all rule groups

associated with the branches stemming from the node assigned attribute A (This step is

justified by the consensus law [x=1] == [x=1] amp [y=a] v [x=1] amp [y=b] assuming

that a and b are the only legal values of y) All rules associated with the given branch

constitute a ruleset context for this branch

Step 4 If all the rules in a ruleset context for some branch belong to the same class create a leaf

node and assign to it that class If all branches of the trees have leaf nodes stop

Otherwise repeat steps 1 to 4 for each branch that has no leaf

To select an attribute to be a node in the decision tree (step 1 and 2 of the algorithm) the

algorithm performs two independent iterations In the first iteration it parses through all decision

rules and determine information about each attributes This information includes the importance

score of each attribute the number of rules containing a given attribute the disjoint value sets of

each attribute and the attribute values used in describing each decision class The second

iteration is only performed if the disjointness criterion is ranked first in the LEF function The

second iteration evaluates the attributes disjointness for each decision class against the other

decision classes

40

To determine the complexity of this process suppose that the maximum number of conditions

fonned by a single attribute value in one rule is s (where s ~ n the number of attributes) and r is

the total number of decision rules (in all decision classes)

m

r=LRt (m is the number of decision classes) 1=1

where Ri is the number of rules in decision class Ci bull The complexity of the frrst iteration can be

determined as

Cmpx(Itcl) =O(r s)

In the second iteration the disjointness is calculated between the decision classes and all

attributes The complexity of the second iteration can be given by

Cmpx(Itc2) = O(n m)

Assume that at each node 1 is the maximum of the number of decision classes to be classified at

this node and the number of rules associated with this node

l=max mr (3-9)

Since these two iteration are independent of each other the complexity of the AQDT algorithm

for building one node in the decision tree say Node Complexity NC(AQDT) is given by

NC(AQDT) = 0(1 n)

Usually I equals to the number of rules associated with the given node Thus the AQDT

complexity for building one node is a function of the number of attributes multiplied by the

number of rules associated with this node

At each level of the decision tree these two iterations are repeated for each non-leaf node The

Level Complexity of the AQDT algorithm LC(AQDT) can be given by

LC(AQDT) lt 0(1 n)

Which is less than the complexity of generating the root of the decision tree To explain this

consider the maximum possible number of non-leaf nodes at one level is half the number of the

initial decision rules (integer of rll) Figure 3-5-a shows an example of such situation where at

41

the lowest level each node classifies only two rules each belongs to different decision class This

decision tree could be a multimiddotvalues tree However the number of non-leaf nodes at each level

will be twice or more than the number of nonmiddot leaf nodes at the previous level Considering the

level complexity of the AQDT algorithm equal to (1 s 0) where 0 is the number of non-leaf

nodes at the given level In such cases it is either (1 0 S r) or (1 s lt r) In Figure 3middot5middota the

complexity of the AQDT algorithm at any lower level is given by

LC(AQDT) =0(2 s r(2) lt O(n I) =NC(AQDT)

a) per one level b) per one path

Figure 3 -5 Decision trees showing the maximum number of none leaf nodes

Note also that after selecting an attribute to be the root of the decision structure this attribute

and all conditions containing that attribute are removed from the data structure of the algorithm

Also if a leaf node is generated all rules belonging to the corresponding branch will not be

tested again

Since the disjointness criterion selects the attribute which minimizes the average number of tests

ANT it is true that the AQDT algorithm generates decision trees with the least number of levels

The number of levels per a decision tree is supposed to be less than or equal to the minimum of

both the number of attributes and the number of rules Consider k as the number of levels in a

given decision tree

k S min mr (3-10)

There are two cases represent the most complex situations Figure 3-5-a and 3-5-b In the fIrst

case where the decision rules were divided evenly the number of levels will be a function of the

42

logarithm of the number of rules In such case the complexity of the AQDT algorithm for

generating a decision tree from a set of decision rules is given by

Complexity(AQDT) = 0(1 n log r) (3-11)

The other situation is when the generated decision tree has the maximum number of levels The

maximum possible number of levels per a decision tree equals to one less than the number of

decision rules Figure 3-5-b Using the disjointness criterion it is not likely to get such a decision

tree because it has the maximum average number of test (ANT) that can be determined from the

same set of nodes and leaves However such a decision tree can be generated if the number of

decision classes is one less thanthe number of attributes In such case any disjQint decision rules

should have a maximum length that is less or equal to the floor of the logarithm of the number of

attributes Thus the level complexity of this decision tree is estimated as

LC(AQDT) =0(1 log n)

The maximum number of levels in such decision tree is k-l Thus the complexity of the AQDT

algorithm in such cases is given by

Complexity(AQDT) = 0(1 k log n) (3-12)

Since k is the minimum of n and r then n is the maximum of k and n Also since in (3-12)

r=n+1 then log r is greater than log n From 3-10 3-11 and 3-12 the complexity of the AQDT

algorithm is determined by

Cmplx(AQDT) = OCr k log 1) (3-13)

333 An example illustrating the algorithm

The following simple example illustrates how AQDT is used in selecting the optimal set of

testing resources for testing a software Suppose there are three tools for testing a software 1)

modeling (Tl) 2) checklist (T2) and 3) par_simul (T3) Also assume that there are four

different factors that affect the selection of any tool 1) the cost of using the tool (xl) 2) the

metrics that support the tool (x2) 3) the best phase for applying the tool (x3) and 4) type of tool

43

(automated semi-automated or manual) (x4) Table 3-1 shows these attributes and their possible

values

Suppose that the domain expert provided a set of rules to be used in testing a software Figure 3shy

6 shows a sample of these rules in AQ15c formats

Table 3-1 The available tools and the factors that affect the prclCes

Tl lt= [xl=2] amp [x2=2] v [xl=3] amp [x3=1 v 3] amp [x4=1] T2 lt= [xl=l v 2] amp [x2=3 v 4] v [xl=3] amp [x3=1 v 2] amp [x4=2] T3 lt= [xl=l] amp [x2=1] v [xl=4] amp [x3=2 v 3] amp [x4=3]

Figure 3-6 Decision rules for selecting the best tool for testing software

These rules can be interpreted as

Rule 1 Use the fltst tool for testing if you need average cost and the tool is

supported by the requirement metric

Rule 2 Use the fIrst tool for testing if you can afford high cost for testing either

in the requirement or the analysis phases and you need an automated tool

Rule 3 Use the second tool for testing if the cost limit is low or average and the

tool is supported by the system usage metric or the intractability metric

Rule 4 Use the second tool for testing if you can afford high cost for testing

either in the requirement or the design phases and you need a manual tool

Rule 5 Use the third tool for testing if the you are limited to low cost and the

tool is supported by the error rate metric

Rule 6 Use the third tool for testing if you can afford very-high cost for testing

either in the requirement or the system usage phases and you need a semishy

automated tool

44

Table 3-2 presents information on these rules and the disjointness values for all attributes For

each class the row marked Values lists values occurring in the ruleset for this class For

evaluating the disjointness of an attribute say A each rule in the ruleset above that does not

contain attribute A is characterized as having an additional condition [A= a vb ] where a b

are all legal values of A

The row Class disjointness specifies the class disjointness for each attribute The attribute xl

has the highest disjointness (11) and is assigned to the root of the tree For simplicity assume

the tolerances for each elementary criterion equal O

From the rules in Figure 3-6 we can also determine disjoint groupings of attribute-values used

for the compact mode of the algorithm This done as follows 1) determine for each attribute the

sets of values that the attribute takes in individual decision rules and remove those value sets

that subsume other value sets The remaining value sets are assigned to branches stemming from

the node marked by the given attribute For example x 1 has the following value sets in the

individual decision rules 2 3 I 2 I and 4 (Figure 3-6) Value set I 2 is

removed as it subsumes 2 and I In this case branches are assigned individual values of the

domain of x1 For attribute x2 the value sets are I 2 34 and I 2 3 4 In this case

branches are assigned value sets I 2 and 34

45

Attribute Xl ranks highest (as it is the highest disjointness) and is assigned to the root of the

tree Four branches are created each one corresponding to one of XIs possible values Since all

rules containing [x1=4] belong to class TI the branch marked by 4 is ended by a leafTI Rules

containing other values of X1 belong to more than one class This process is repeated for each

subset of rules until the decision tree is completed Figure 3-7 shows a decision structure learned

by AQDT-2 from the rules in Figure 3-6 The decision structure in Figure 3-7 can be used in

making decisions on which tools can be used for testing a given software

xl

Complexity No of nodes 4 No of leaves 7

Figure 3-7 A decision structure learned for classifying so tware testing tools

Figure 3-8-a shows the diagrammatic visualization of the decision rules and Figure 3-8-b shows

the visualization of the derived decision tree Each diagram in Figure 3-8 consists of cells

representing one combination of attribute-values Attributes and their legal values are shown on

scales surrounding the diagram (eg the horizontal scale for xl shows values 1 2 3 and 4)

Rules are represented by collections of cells in the intersection of the rows and columns

corresponding to the conditions in the rules

The shaded areas correspond to decision rules Rules of the same class have the same type of

shading Empty cells correspond to combination of attribute-values not assigned to any class For

illustration collections of cells corresponding to some of the initial rules are marked R 11 R 21

R31 and R32 Rll denotes the IlfSt rule of Class T1 ie [x1=2] amp [x2=2] R21 denotes the

IlfSt rule of class T2 ie [xl= 1 v 2] amp [x2= 3 v 4] R31 the first rule of class TI ie [x1=1]

46

amp [x2 =1] and denotes R32 denotes the second rule of class T3 Le [xl=4] amp [x3=2 v 3] amp

[x4=3]

a) Decision rules b) Derived decision tree

Comparing diagrams in Figure 3-8-a and 3-8-b AQDT-2 decision tree represents a slightly more

general description of Concepts Tl TI and T3 than the original rules Let us assume that it is

very costly to know which metrics suppon the required tools In other words suppose that we

would like to select the best tools independently of the metric they suppon (this is indicated to

AQDT-l by assigning very high cost to attribute x2) The algorithm again assigns attribute xl to

the root of the tree When xl takes value 1 or 2 it is impossible to assign a specific decision

without measuring attribute x2 For the value 1 of xl the recommended tool can be either TI or

T3 (see the diagram in Figure 3-9-a) and for the value 2 of xl the recommended tool can be

either Tl or T3 However for the value 3 of xl one can make a specific decision after

measuring attribute x4 For the value 1 of x4 tool is Tl and for the value 2 of x4 the

recommended tool is TI Figure 3-9-b shows another decision tree where the type of the tool

was ignored from the data Those decision trees are called indeterminate because some of their

leaves are assigned a disjunction of two or more class names

47

xl

xl

1 12

a) Ignoring the supporting metric b) Ignoring the type of the tooL Figure 3 -9 Decision trees learned ignoring the support metric and the type of the testing tool

It is clear that the given set of rules depend highly on amount of money can be spent Let us now

suppose that the cost attribute xl which was determined as the highest ranked attribute cannot

be measured The algorithm selected x4 to be the root of the new decision tree After continuing

the algorithm the tree in Figure 3-10 is obtainedThe nodes that are assigned one class indicate

situations in which it is possible to make a specific decision without knowing the value of

attribute xl

x4

Figure 3-10 A decision tree learned without the cost attribute

34 Tailoring Decision Structures to the Decision-making situation

Decision structures are among the simplest structures for organizing a decision-making process

A decision structure specifies explicitly the order in which attributes of an object or situation

need to be evaluated in the process of determining a decision A standard way to generate a

48

decision structure is to learn it from examples of decisions Such a process usually aims at

obtaining a decision structure that has the highest prediction accuracy that is the highest rate of

assigning correct decisions to given situations There can usually be a large number of logically

equivalent decision structures (Michalski 1990) As such they may have the same predictive

accuracy but differ in the way they organize the decision process and thus may differ in the cost

of arriving at a decision To minimize the average decision cost one needs to take into

consideration the distribution of the costs of attribute evaluation and the frequency of different

decisions This report presents an approach to building such Task-oriented decision structures

which advocates that they are built not from examples but rather from decision rules Decision

rules are learned from examples using the AQ15 or AQ17 inductive learning program or are

specified by an expert An efficient algorithm and a new system AQDT-2 that transforms

decision rules to task-oriented decision structures The system is illustrated by applying it to the

problem of learning decision structures in the area of construction engineering (for determining

the best wind bracing for tall buildings) In the experiments AQDT-2 outperformed all other

programs applied to the same data

Decision-making situations can vary in several respects In some situations complete

information about a data item is available (ie values of all attributes are specified) in others the

information may be incomplete To reflect such differences the user specifies a set of parameters

that control the process of creating a decision structure from decision rules AQDT-2 provides

several features for handling different decision-making problems 1) generating a decision

structure that tries to avoid unavailable or costly attributes (tests) 2) generating unknown

leaves in situations where there is insufficient information for generating a complete decision

structure 3) providing the user with the most likely decision when performing a required test is

impossible 4) providing alternative decisions with an estimate of the likelihood of their

correctness when the needed information can not be provided

49

341 Learning Costmiddot Dependent Decision Structures

As described in Sec 2 the LEF criterion enables the system to take into consideration the cost of

measuring tests (attributes) in developing a decision structure In the default setting of LEF the

tests cost is the [lIst criterion and its tolerance is O This means that only the least expensive

attributes pass to the next step of evaluation involving other elementary criteria If an attribute

has high cost or is impossible to measure (has an infinite cost) the LEF chooses another

cheaper attribute ifpossible

342 Assigning Decision Under Insufficient Information

In decision-making situations in which one or more attributes cannot be measured the system

may not be able to assign a definite decision for some cases If no more information can be

obtained but a decision has to be made it is useful to know the probability distribution for

different candidate decisions (Smyth Goodman amp Higgins 1990) The most probable decision is

then chosen The probability distribution can be estimated from the class frequency at the given

node Let us consider a node in the structure connected to the root by the sequence bl b2 bkof

attribute-values Let P(Ci I bl b2 bk) denote the conditional probability of class Ci i=I2 m

at that node given that an example to be classified has attribute-values assigned to branches bl

b2 bull bk in the decision structure Using Bayesian formula we have

P(Ci I bl~ bull bk) = P(Ci) P(bl bk I Ci) P(bl bk) (3-9)

where P(Ci) the apriori probability of class Ci P(bl bk I Ci) is the probability that the

example has attribute-values blbull bk given that it represents decision class Ci To approximate

these probabilities let us suppose that Wi is the number of training examples of Class Ci that

passed the tests leading to this node and twi - the total number of training examples of Class

Ci Let us use these numbers to estimate the probabilities in (9) Assuming that the apriori class

probabilities correspond to the frequency of training examples from different classes we have

50

m P(Ci) = twi IL twj (3-10)

j=l

P(b1 bkl Ci) = Wi I twi (3-11)

m m P(b1 bk) =L Wj IL twj (3-12)

j=l j=1

By substituting (1O) (11) and (12) in (9) we obtain

m P(Ci I b1 bk) = wilL Wj (3-13)

j=l

A related method for handling the problem of unavailability of an attribute is described by

Quinlan (1990) Quinlans method however assigns probabilities to the most probable decision

at the node associated with such an attribute The method does not restructure the decision tree

appropriately to fit the given decision-making situation (in this case to avoid measuring xl)

AQDT -2 assigns probabilities o~ly after it first created a decision structure that suits the given

decision-making si~ation An example is presented in section 42

343 Coping with noise in training data

The proposed methodology can be easily extended to handle the problem of learning from noisy

training data This is done based on ideas of rule truncation described in (Niblett amp Bratko 1986

Bergadanoet al 1992 Cestnik amp Karalic 1991) When noise is expected in the training data

the decision rules with t-weight below a certain threshold (reflecting the expected noise level) are

removed The rule truncation method seems to result in more accurate decision structures than

decision tree pruning method because truncation decisions are solely based on the importance of

the given rule or condition for the decision-making regardless of their evaluation order (unlike

the decision tree pruning which can only prunes attributes within a subtree and thus cannot

freely chose attributes to prune) Examples are presented in section 4

copy 1995 Copyright by Ibrahim F Imam All rights reserved

ACKNOWLEDGMENT

I would like to thank Professor Ryszard S Michalski PRC Chaired Professor of Computer

Science and Engineering and my Dissertation Director for his support encouragement and

guidance I would like to thank my committee members Professor Larry Kerschburg Chair of the

Department of Information Systems and Software Engineering Professor David Rine Professor

of Information Technology and Engineering and Professor David Schum Professor of Excellence

in Information Technology and Engineering for their encomagement and help with many aspects

of my PhD

I would like to thank Professor Tomaz Arciszewki System Engineering Department for providing

me with application problems Ronny Kohavi Stanford University for discussion and providing

me with some related work on learning decision structures and decisionmiddot graphs and Professor

George Tecuci Computer Science Department for pointing some related work

I would like to thank my colleagues Nabil Al-Kharouf for reviewing my dissertation Eric

Bloedorn for reviewing an earlier draft of my dissertation and for using his program AQ17-DCI in

my experiments Srinivas Gutta for providing some application for my PhD work Mike Heib for

reviewing an earlier draft of my dissertation and helping me find relevant articles Ken Kaufman

for reviewing an earlier draft of my thesis Mark Maloof for providing me with script files which

made it easier to iteratively run AQ1Sc Halah Vafaie for working with her on application and

comparison of different aspects of my work and Janusz Wnek for using his DIAV program for

explaining my results

I would like to thank Professor Andrew P Sage Dean of the School of Information Technology

and Engineering and Professor Kenneth Bumgarner Dean of Student Services and Associate ViCe

President of George Mason University for their support and Professor Murray W Black

Associate Dean of the School of Infonnation Technology and Engineering for guidance on

preparing the PhD proposal

I would like to thank conferences organizers who supported me to attend their conferences and

present parts of my PhD work The organizers include Professor Moonis Ali Professor Frank

Anger Professor Zbigniew Ras the American Association for Artificial Intelligence and others I

would like also to thank the organizing committee of the Florida Artificial Intelligence Research

Symposium FLAIRS-95 for organizing my first workshop Thanks goes to Dr Doug Dankel Dr

Howard Hamilton Dr John Stewman and Dr Dan Tamir

I would like also to thank the many individuals who helped me in any way during my PhD Those

include Dr Ashraf Abdel-Wahab Dr Jerzy Bala Dr Alex Brodsky Dr Richard Carver Dr

Kenneth De Jong John Doulamis Tom Dybala Dean Ibrahim Farag Dr Ophir Frieder Csilla

Frakes Dr Mohamed Habib Ali Hadjarian Dr Hugo De Garis Zenon Kulpa Dr Paul Lehner

Dr George Michaels Dr Eugene Norris Dean James Palmer Mitch Potter Dr Ahmed Rafea

Jim Ribeiro Jsyshree Sarma Dr Arun Sood Dr Clifton Sutton Bradley Utz Dr Tibor Vamos

Patricia Zahra Dr Shaker Zahra and Dr Jianping Zhang

~J I yarJ I JJ I ~

U-t-oJ ~II ~) ~ oWAIl J

Dedication

To my mother my brothers and my sister

TABLE OF CONTENTS

TI1LE Page

ABS1RAcr II 1

CHAPrER 1

IN1RODUCI10N II 3

11 Motivation and Overview 3

12 The Problem Statement 6

CHAPrER2

REIAfED RESEARCFI 7

21 Learning Decision Trees from Decision Diagrams 7

22 Learning Decision Trees from Examples 10

221 Building Decision Trees Using Information-based Criteria 11

222 Building Decision Trees Using Statistics-based Criteria 16

223 Analysis of Attribute Selection Criteria 18

23 Learning Decision Structures 19

CHAPrER3

DESCRIPTION OF TIlE APPROACFI 23

31 General Methodology 23

32 A Brief Description of the AQ15 and AQ17 RuleLeaming Systems 25

33 Generating Decision Structures From Decision Rules 28

331 The AQDT-2 attribute selection method 29

332 The AQDT-2 algoridlm 37

vi

333 An example illustrating the algorithm 42

34 Tailoring Decision Structure to a decision-making situation 47

341 Learning Cost-Dependent Decision Structures 49

342 Assigning Decision Under Insufficient Infonnation 49

343 Coping with noise in training data 50

35 Analysis of the AQDT-2 Attribute Selection Criteria 51

36 Decision Structures vs Decision Trees 53

CHAPTER 4

EMPRICAL ANALYSIS AND COMPARATIVE STUDY OF TIlE MElHOD 58

41 Description of the Experimental Analysis 59

42 Experiments With Average Size Complex and Noise-Free Problems

Wind Brac-ings 60

43 Experiments With Small Size Simple and Noise-Free Problems

e MONK-I 69

44 Experiments With Small Size Complex and Noise-Free Problems

MONK-2 76

45 Experiments With Small Size Simple and Noisy Problems

MONK-3 79

46 Experiments With Large Size Complex and Noise-Free Problems

Diagnosing Breast Cancer 83

47 Experiments With Large Size Complex and Noisy Problems

Musm()()m classifications 84

48 Experiments With Small Size Structured and Noise-Free Problems

East-West Trains 85

49 Experiments With Small Size Simple and Noisy Problems

vii

Congressional Voting Records (1984) 87

410 Analysis of the Results 88

ClIAPfER 5 CONCIUSIONS 95

51 Summary 95

52 Contributions 96

REFERENCES 98

VITA 102

viii

LIST OF TABLES

No TITLE Page

2-1 An example of a decision table 9

2-2 A set of training examples used to illustrate the C45 system 15

2-3 The frequency of different attributes values to different decision classes 17

2-4 The expected values of the frequency of examples in Table 2-3 17

2-5 Attribute selection criteria and their basic evaluation measure 17

2-6 The contingency tables of Mingers example 18

2-7 Mingers results for determining the goodness of split 19

2-8 Mingers results for comaring the total accuracy and size of decision trees

provided by different attribute selection criteria from four problems 19

2-9 Comparing the AQDT approach with EDAG and HOODG approaches 22

3-1 The available tools and the factors that affect the process of testing a software 43

3-2 Calculating the disjointness of each attribute 44

3-3 Evaluation of the AQDT-2 criteria on the Quinlans example in Table 2-2 51

3-4 The data used in Mingers f11st experiments 52

3-5 The performance of AQDT-2 criteria (compare with the other criteria in Table 2-6) 52

3-6 The possible ranking domains and using condition of AQDT-2 criteria 53

3-7 Comparison between Decision Structures and Decision Tree 54

4-1 Evaluation of AQDT-2 attribute selection criteria for the wind bracing problem 62

4-2 The predictive accuracy of AQ15c and AQDT-2 for the wind bracing problem 67

4-3 Evaluation of AQDT-2 attribute selection criteria for the MONK-l problem 71

4-4 The predictive accuracy of AQ15c and AQDT-2 for the MONK-l problem 73

ix

4-5 The predictive accuracy of AQ15c and AQDT-2 for the MONK-2 problem 77

4-6 The predictive accuracy of AQ15c and AQDT-2 for the MONK-3 problem 81

4-7 The set of attributes and their values used in the trains problem 86

4-8 A tabular snmmary of the predictive accuracy of decision trees obtained

by AQDT -2 and C45 for the congressional voting data 88

4-9 Summary of the best parameter settings for the first subfunction of the approach

with different data characteristics 89

4-10 Summary of the performance of AQDT-2 and C45 on different problems 90

x

LIST OF FIGURES

No TITLE Page

2-1 An example to illustrate how attributes break rules 8

2-2 A decision tree learned from the decision table in Table 2-1 10

2-3 A decision tree learned using the gain criterion for selecting attributes 15

2-4 Decision rules and their Exception Directed Acyclic Graph (EDAG) 21

3-1 Architecture of the AQDT approach 24

3-2 A ruleset generated by AQ15 for the concept Voting pattern of

Democratic Representatives 27

3-3 Ven Diagrams ofPossible combinations of attribute values in two decision classes 33

3-4 Decision trees corresponding the Ven diagrams in Figure 3-3 33

3-5 Decision trees showing the maximum number of none leaf nodes 41

3-6 Decision rules for selecting the best tool for testing a software 43

3-7 A decision structure learned for classifying software testing tools 45

3-8 A diagrammatic visualization of the decision rules and the derived decision tree 46

3-9 Decision trees learned ignoring the support metric and the type of the testing tool 47

3-10 A decision tree learned without the cost attribute 47

3-11 Decision structures learned by AQDT-2 using different criteria 55

3-12 The Imams example Example where learning decision structures (trees)

from rules is better than learning them from examples 56

3-13 An example where decision rules are simpler than decision trees 57

4-1 Design of a complete experiment 59

Xl

4-2 Decision rules determined by AQ15c from the wind bracing data 61

4-3 A decision tree learned by C45 for the wind bracing data 63

4-4 A decision structure learned from AQ15c wind bracing rules 64

4-5 A decision structure that does not contain attribute xl 64

4-6 A decision structure without Xl with candidate decisions assigned to leaves 65

4-7 A decision structure determined from rules in Figure 4-4 under the

assumption of 10 classification error in the training data 65

4-8 Diagramatic visualization of decision trees learned for different decision

making situations for the wind bracing data 66

4-9 The accuracy of AQDT-2 and AQ15c with different parameter settings

for the wind bracing PIoblem 68

4-10 Analyzing different parameter setting of AQDT-2 using the wind bracing data 69

4-16 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-20 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-11 A comparison between AQ15c and AQDT-2 against C45 on the wind bracing data 69

4-12 A visualization diagram of the MONK-l problem 70

4-13 Decision rules learned by AQ15c for the MONK-1 problem 71

4-14 The decision tree for the MONK-1 problem generated both by AQDT-l 72

4-15 Compact decision structures generated by AQDT-2 for the MONK-1 problem 72

for the MONK-l Problem 74

4-17 Analyzing different parameter setting of AQDT-2 with the MONK-l data 75

4-18 Comparing AQ15c and AQDT-2 against C45 using the MONK-1 problem 75

4-19 A visualization diagram of the MONK-2 problem 76

for the MONK-2 Problem 78

4-21 Analyzing different parameter setting of AQDT-2 with the MONK-2 data 79

xii

4-22 Comparing AQ15c andAQDT-2 against C45 using the MONK-2 problem 79

4-24 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-35 A visualization diagram of a decision tree learned by AQDT -2 after reducing

4-23 A visualization diagram of the MONK-3 problem 80

for the MONK-3 Problem 82

4-25 Analyzing different parameter setting of AQDT-2 with the MONK-3 data 82

4-26 Comparing AQ15c and AQDT-2 against C45 using the MONK-3 problem 83

4-27 Comparing AQ15c and AQDT-2 against C45 using the breast cancer problem 84

4-28 Comparing AQ15c and AQDT-2 against C45 using the mushroom problem 85

4-29 Decision structures learned by AQDT-2 for different decision-making situations 87

4-30 Comparing decision trees for the Cong Voting-84 data learned by C45 amp AQDT -2 88

4-31 A visualization diagram of decision rules learned by AQ15c for the MONK-2 91

4-32 A visualization diagram shows the testing errors for the rules in Figure 4-31 92

4-33 A visualization diagram of the decision tree learned by AQDT-2 for the MONK-2 93

4-34 A visualization diagram shows testing errors of AQDT-2 decision tree bull 94

the generalization degree to 1 94

DERIVING TASKmiddotORIENTED DECISION STRUCTURES

FROM DECISION RULES

Ibrahim M Fahmi Imam PhD

School of Information Technology and Engineering

Geotge Mason University Fall 1995

Ryszard S Michalski Advisor

ABSTRACT

This dissenation is concerned with research on learning task-oriented decision structures from

decision rules The philosophy behind this research is that it is more appropriate to learn

knowledge and store it in a declarative form and then when a decision making situation occurs

generate from this knowledge the decision structure that is most suitable for the given decision

making situation Learning decision structures from decision rules was first introduced by

Michalski (1978) The flIst implementation of this approach was done by Imam and Michalski

(1993 ab) called AQDT-l

This approach separates the function of generating a knowledge-base from the function of using

the knowledge-base for decision-making The first function focuses on learning accurate

consistent and complete concept description expressed in a declarative form The second function

is performed whenever a new decision-making situation occurrs a task-oriented decision structure

is obtained to suit that situation Thsk-oriented knowledge is defined as knowledge that is adapted

for solving a given decision-making situation (Imam amp Michalski 1994 Michalski amp Imam

1994)

The dissertation introduces the system AQJJT-2 for learning task -oriented decision structures from

decision rules or examples Each decision making situation is defined by a set of parameters that

controls the learning process of the AQDT-2 system The extensive experiments on AQJJT-2 show

that decision structures learned by it usually outperform in terms of accuracy and average size of

the decision structures those learned from examples by other well known systems The results

show also that the system does not work very well with noisy data The system is illustrated and

compared using applications of artificial problems such as the three MONKs problems (Thrun

Mitchell amp Cheng 1991) and the East-West 1iain problem (Michie et al 1994) It was also

applied to real world problems of learning decision structures in the areas of construction

engineering (for determining the best wind bracing design for tall buildings) medical

diagnosis(for learning decision rules for recognizing breast cancer) agriculture diagnosis (for

learning classification rules for distinguishing between poisonous and non-poisonous

mushrooms) and political data (for characterizing democratic and republican voting records)

CHAPTER 1 INTRODUCTION

11 Motivation and Overview

Learning and discovery systems should be able not only to generate and store knowledge but also

to use this knowledge for decision-making The main step in the development of systems for

decision-making is the creation of a knowledge structure that characterizes the decision-making

process The form in which knowledge can be easily obtained may however differ from the form

in which it is most readily used for decision-making It is therefore important to identify the form

of knowledge representation that is most appropIiate for learning (eg due to ease of its

modification) and the form that is most convenient for decision making

A simple and effective tool for describing decision processes is a decision structure which is a

directed acyclic graph that specifies an order of tests to be appliedto an object (or a situation) to

arrive at a decision about that object The nodes of the structure are assigned individual tests

(which may correspond to a single attribute a function of attributes or a relation) the branches are

assigned possible test outcomes or ranges of outcomes and the leaves are assigned a specific

decision a set of candidate decisions with corresponding probabilities or an undetermined

decision A decision structure reduces to a familiar decision tree when each node is assigned a

single attribute and has at most one parent when the branches from each node are assigned single

values of that attribute and when leaves are assigned single definite decisions Thus the problem

of generating a decision structure is a generalization of the problem of generating a decision tree

Decision trees are typically generated from a set of examples of decisions The essential

characteristic of any such method is the attribute selection criterion used for choosing attributes to

be assigned to the nodes of the decision tree being built Such criteria include the entropy

reduction the gain and the gain ratio (Quinlan 1979 83 86) the gini index ofdiversity (Breiman

et al 1984) and others (Cestnik amp Bratko 1991 Cestnik amp Karalic 1991 Mingers 1989a)

3

4

Adecision treedecision structure representations can be an effective tool for describing a decision

process as long as all the required tests can be performed and the decision-making situations it

was designed for remain constant (eg in a doctor-patient example the doctor should detennine

that answers for all symptoms appear in the decision tree) Problems arise when these assumptions

do not hold For example in some situations measuring certain attributes may be difficult or costly

(eg in the doctor-patient example a brain or blood test is needed which is very expensive or the

tools needed are not available) In such situations it is desirable to refonnulate the decision

structure so that the inexpensive attributes are evaluated frrst (assigned to the nodes close to the

root) and the expensive attributes are evaluated only if necessary (by assignment to the nodes far

away from the root) If an attribute cannot be measured at all it is useful to either modify the

structure so that it does not contain that attribute or-when this is impossible-to indicate

alternative candidate decisions and their probabilities A restructuring is also desirable if there is a

significant change in the frequency of occurrence of different decisions (eg in the doctor-patient

example the doctor may request a decision structure expressed in a specific set of symptoms

biased to classify one or more diseases or specify a certain order of testing)

Arestructuring ofa decision structure (or a tree) in order to suit new requirements could be quite

difficult This is because a decision structure is a procedural representation that imposes an

evaluation order on the tests In contrast no evaluation order is imposed by a declarative

representation such as a set of decision rules Tests (conditions) of rules can be evaluated in any

order Thus for a given set of rules one can usually build a huge number of logically equivalent

decision structures (trees) which differ in the test ordering Due to the lack of order constraints

a declarative representation (rules) is much easier to modify to adapt to different situations than a

procedural one (a decision structure or a tree) On the other hand to apply decision rules to make a

decision one needs to decide in which order tests are evaluated and thus needs a decision

structure

An attractive solution to these opposite requirements is to acquire and store knowledge in a

declarative form and transform it to a decision structure when it is needed for decision-making

5

This method allows one to create a decision structure that is most appropriate in a given decisionshy

making situation Because the number of decision rules per decision class is usually small (each

rule is a generalization of a set of examples) generating a decision structure from decision rules

can potentially be perfonned much faster than by generation from training examples Thus this

process could be done on line without any delay noticeable to the user Such virtual decision

structures are easy to tailor to any given decision-making situation

This approach allows one to generate a decision structure that avoids or delays evaluating an

attribute that is difficult to measure in some decision-making situation or that fits well a particular

frequency distribution of decision classes In other situations it may be unnecessary to generate

complete decision structure but it may be sufficient to generate only a part of it which concerns

only with decision classes of interest Thus such an approach has many potential advantages

This dissertation presents a new system called AQflf-2 The AWf-2 system generates a taskshy

oriented decision structure (decision structure that is adapted to the given decision-making

situation) from decision rules The decision rules are learned by either rule leaming system AQ15

(Michalski et al 1986) or system AQ17-DCI which has extensive constructive induction

capabilities (Bloedorn et al 1993)

To associate the decision rules with a given decision-making task AQflf-2 provides a set of

features including 1) enabling the system to include in the decision structure nodes corresponding

to new attributes constructed during the process of learning the decision rules 2) controlling the

degree of generalization needed during the development of the decision structure 3) providing four

new criteria for selecting an attribute to be a node in the decision structure that allow the system to

generate many different but equivalent decision structures from the same set of rules 4) generating

unknown nodes in situations when there is insufficient infonnation for generating a complete

decision structure 5) learning decision structures from discriminant rules as well as

characteristic rules and 6) providing the most likely decision when the decision process stops

due to inability to evaluate an attribute associated with an intennediate node

6

To test the methooology of generating decision structures from decision rules an extensive set of

planned experiments have been designed to test different aspects of the approach The experiments

include testing different combinations of parameters for each sub-function of the approach

analyzing the relatioship between decision rules and the decision structure learned from them and

comparing decision trees learned by the AQUf2 system the well known C45 (Quinlan 1993)

system for learning decision trees from examples Different experiments were designed to examine

the new features of the AQUf approach for learning task-oriented decision structures The

experiments were applied to artificial domains as well as real-world domains including MONK-I

MONK-2 and MONK-3 (Thrun Mitchell amp Cheng 1991) East-West trains (Michie et al

1994) Engineering Design-wind bracings (Arciszewski et al 1992) Mushrooms Breast

Cancer (Mangasarian amp WolbeJg 1990) and Congressional bting Records of 1984 The

MONKs problems are concerned with learning classification rules for robot-like figures MONK-l

requires learning a DNF-type description MONK-2 requires learning a non-DNF-type description

(one that cannot be easily described as a DNF rule using the original attributes) MONK-3 concerns

learning a DNF rule from noisy data The East-West trains dataset is a structural domain that

classifies two sets of trains (Eastbound and Westbound) The Engineering Design-wind

bracing data involves learning conditions for applying different types of wind bracing for ta11

buildings The Mushrooms data is concerned with learning classification rules for distinguishing

between poisonous and non-poisonous mushrooms The Breast Cancer data involves learning

concept descriptions for recognizing breast cancer The congressional voting data includes voting

records on different issues AQUf2 outperformed C45 on the average with respect to both the

predictive accuracy and tree size for most problems Apf-2 did not work very well with noisy

data or with problems that have many rules covering very few examples

12 The Problem Statement

There are many limitations and problems accompanied using decision trees for decision-making

CHAPTER 2 RELATED RESEARCH

21 Learning Decision Trees from Decision Diagrams

The AQDT-2 method proposed here is based on the earlier work by Michalski (1978) which

introduced an algorithm for generating decision trees from decision lists The method proposed

several attribute selection criteria These criteria are of increasing power of the main criterion

order cost estimate (the nth order cost estimates n=l 2 ) Michalski also analyzed two

specific criteria MAL and DMAL for selecting the optimal attribute for a node in the tree

based on properties extracted from the decision diagram In order to better explain the method

it is necessary to define some terms These terms will be used later in the dissertation

Definition 2-1 A ~ of a decision class C is a set of rules whose union includes the set of all

examples of class C and does not include any examples of other decision classes

Definition 2-2 A ~ of a decision class C is disjoint if all rules (rules) are pairwise logically

disjoint In other words for any two rules there exists a condition with the same attribute but

with different values in each rule

Definition 2-3 An minimal cover is a disjoint cover which has the smallest number of rules

among all possible covers

Definition 2-4 A diagram for a given cover is a table constructed graphically by representing

in two dimension space of all possible combination of attributes values and locating on the

diagram all the condition parts of the given rules and marking them with the action specified

with each rule

Michalskis algorithm requires construction of minimalmiddot cover The minimal cover should be

consistent and complete The method is based on the fact that if there are n decision classes

any decision tree that correctly classifies the given rules and has n distinct leaves is a minimal

7

8

decision tree (any consistent decision tree should have at least n leaves) Michalski (1978) has

shown that if only one rule is broken by a selected attribute then instead of having one leaf

(which could potentially represent this rule or the decision class in the tree) there will have to

be at least two leaves representing this rule in the fmal decision tree

The attribute selection criterion MAL introduced in (Michalski 1978) prefers attributes that do

not break any rules or break as few as possible An attribute breaks a rule if the attribute can

divide the rule into two or more sub-rules Figure 2-1 shows two examples of two sets of rules

In the fJIst xl and x3 break at least two rules each (xl breaks the rule [x4=2] amp [x1=lv3] amp

[x3=2v3] and the rule [x4=3] amp [x1=1v2] amp [x3=1] and x3 breaks three rules the rule [x4=2]

amp [xl=2] amp [x3=2v3]the rule [x4=l] amp [xl=3] amp [x3=lv3] and the rule [x4=3] amp [xl=4] amp

[x3=2v3]) In the diagram on the right xl is the only attribute that does not break any rule

Figure 2-1 An example to illustrate how attributes break rules

One of the criteria defmed by Michalski is the first degree cost estimate which assigns to each

attribute an integer equal to the number of rules broken by that attribute This criterion is also

called the static cost estimate of an attribute or the criterion of minimizing added leaves

(MAL)

-

C3

9

The MAL criterion (Minimizing Added Leaves) seeks an attribute that minimizes the

estimated number of additional nodes in the decision tree being generated over a hypothetical

minimal decision tree When there is a tie between two attributes the attribute to be selected is

the one which breaks smaller rules (rules that cover fewer examples or more specialized

rules) AQDT-2 uses an approximate version of this criterion (the attribute dominance)

Another criterion introduced by Michalski was the DMAL criterion The DMAL criterion

(Dynamically Minimizing Added Leaves) is based on a principle similar to that of MAL but

is more complex because once an attribute is selected as a node in the tree some rules andor

parts of the broken rules at each branch are merged into one rule The DMAL ensures that the

value of the total cost estimate of an attribute is decreased by a value equal to the number of

merged rules minus one

Example Learn a decision tree from the following decision table

The minimal cover consists of the following rules

Al lt [x2=O] v [x1=0][x2=2] A2 lt [x2=I] v [xl=2][x2=2] A3 lt [xl=I][x2=2]

The evaluations ofthe MAL criterion for these attributes are 2 for xl 0 for x2 5 for x3 and 5

for x4 The attribute xl divides the two rules [x2=O] and [x2=I] so selecting it will add two

leaves to the optimal number of leaves It is clear that the attribute to be selected as a root of

10

the decision tree is x2 Then three branches are attached to the root node and the decision rules

are divided into subsets each corresponding to one branch For x2 = 0 or 1 a leave node is

generated For x2=2 another attribute is selected to be a node in the tree In this case xl has the

minimum MAL value Figure 2-2 shows the decision tree obtained using the MAL criterion

Al A3 A2

Figure 2-2 A decision tree learned from the decision table in Table 2-1

22 Learning Decision Trees from Examples

Decision tree learning is a field that concerned with generating decision tree that classifies a set

of examples according to the decision classes they belong to The essential aspect of any

inductive decision tree method is the attribute selection criterion The attribute selection

criterion measures how good the attributes are for discriminating among the given set of

decision classes The best attribute according to the selection criterion is chosen to be assigned

to a node in the tree The fIrst algorithm for generating decision trees from examples was

proposed by Hunt Marin and Stone (1966) Hunts algorithm uses a divide and conquer

algorithm for building decision trees This algorithm has been subsequently modified by

Quinlan (1979) and applied by many researchers to a variety of learning problems

Attribute selection criteria can be divided into three categories These categories are logicshy

based information-based and statistics-based The logic-based criteria for selecting attributes

use logical relationships between the attributes and the decision classes to determine the best

attribute to be a node in the decision tree such as the MAL criterion minimizing added leaves

(Michalski 1978) which uses conjunction and disjunction operators The information-based

criteria are based on the information theory These criteria measure the information conveyed

11

by dividing the training examples into subsets Examples of such criteria include the

information measure 1M the entropy reduction measure and the gain criteria (Quinlan 1979

83) the gini index of diversity (Breiman et al 1984) Gain-ratio measure (Quinlan 1986) and

others (Clark amp Niblett 1987 Bratko amp Lavrac 1987 Cestnik amp Karalic 1991) The

statistics-based criteria measure the correlation between the decision classes and the attributes

These criteria use statistical distributions for determining whether or not there is a correlation

The attribute with the highest correlation is selected to be a node in the tree Examples of

statistics-based criteria include Chi-square and G-statistic (Sokal amp Rohlf 1981 Han 1984

Mingers 1989a)

Niblett and Bratko (1986) Quinlan (1987) and Bratko and Kononeko (1987) extended the

method of learning decision trees to also handle data with noise (by pruning) Handling noise

extended the process of learning decision trees to include the creation of an initial complete

decision tree tree pruning which is done by removing subtrees with small statistical Validity

and replacing them by leaf nodes (MiDgers 1989b) More recently pruning has also been used

for simplifying decision trees even for problems without noise (Bohanec amp Bratko 1994)

Pruning decision trees improves their simplicity but reduces their predictive accuracy on the

training examples Quinlan (1990) proposed also a method to handle the unknown attributeshy

value problem by exploring probabilities of an example belonging to different classes

The rest of this section includes a brief description of the attribute selection criterion used by

C45 learning system (Quinlan 1993) C45 uses an information-based criterion for selecting

an attribute to be a node in the tree Also the section includes a brief description of the Chishy

square method for attribute selection (Mingers 1989a) The method is a statistics-based

method for selecting an attribute to be a node in the tree

221 Building Decision Trees Using htformation-based Criteria

12

This section presents a description of the inductive decision tree learning system C4S The

C4S learning system is considered to be one of the most stable accurate and fastest program

for learning decision trees from examples

Learning decision trees from examples requires a collection of examples Each example is

represented by a fixed number of attribute-value pairs C45 (Quinlan 1993) is a learning

program that induces classification decision trees from a set of given examples The C4S

learning system is descended from the learning system ID3 (Quinlan 1979) which is based on

Hunts method for constructing decision tree from a set ofcases (Hunt Marin amp Stone 1966)

The C45 system uses an attribute selection criterion called the Gain Ratio This criterion

calculates the gain in classifying information based on the residual information needed to

classify cases in a set of training examples and the information yielded by the test based on the

relative frequencies of the possible outcomes (decision classes) The gain ratio criterion is

based on an earlier criterion used by ID3 called the Gain Criterion The Gain Criterion uses

the frequency of each decision class in the given set of training examples

Once an attribute is chosen to be a node in the tree the system generates as many links as the

number of its values and classifies the set of examples based on these values If all the

examples at a certain node belong to one decision class the system generates a leaf node and

assigns it to that class Otherwise the system searches for another attribute to be a node in the

tree

The Gain Criterion The gain criterion is based on the information theory That is the

information conveyed by a message depends on its probability and can be measured in bits as

minus the logarithm (base 2) of that probability To explain the gain criterion suppose that for

a given problem Xu bullbull X are given attributes and Cu c are decision classes Suppose S is

any set of cases and T is the initial set of training cases The frequency of class C1 in the set S

is the number of examples in S that belong to class C i bull

13

freq(Cit S) = Number of examples in S belong to CI (2-1)

Suppose that lSI is the total number of examples in S the probability that an example selected

at random from S belongs to class Ci is freq(CIbullS) IISI

The information conveyed by the message that a selected example belongs to a given decision

class Cit is determined by -log1 (freq(Ch S) IISI) bits

The expected information from such a message stating class membership is given by

info(S) = -k

L (freq(C S) I lSI) log1 (freq(C S) IISI) bits (2-2)I

info(S) is known also as the entrQpy of the set S When S is the initial set of training examples

info(T) determines the average amount of information needed to identify the class of an

example in T

Suppose that we selected an attribute X to be the root of the tree and suppose that X has k

possible values The training set T will be divided into k subsets each corresponding to one of

Xs values The expected information of selecting X to partition the training set T infoz(T) can

be found as the sum over all subsets of multiplying the information conveyed by each subset

by its probability

k

infoz(T) =L (ITII 1m) infO(TIraquo (2-3) I

The information gained by partitioning the training examples T into subset using the attribute

X is given by

gain(X) =inlo(T) - inoIT) (2-4)

The attribute to be selected is the attribute with maximum gain value

The Gain Ratio Criterion This criterion indicates the proportion of information generated by

the split that appears helpful for classification Quinlan (1993) pointed out that the gain

criterion has a serious deficiency Basically it is strongly biased toward attributes with many

outcomes (values) For example for any data that contains attributes such as social security

14

number the gain criterion will select that attribute to be the root of the decision tree However

selecting such attributes increases the size of the decision tree Quinlan provided a solution to

this problem by introducing the gain ratio criterion which takes the ratio of the information that

is gained by partitioning the initial set of examples T by the attribute X to the potential

information generated by dividing T into n subsets

Following similar steps to obtain the information conveyed by dividing T into n subsets The

expected information generated by dividing T into n subsets and by analogy to equation 2-2 is

determined by

split info(T) = - Lt

(ITII lTI) 10gl (ITII m ) (2-5) I

The gain ratio is given by

gain ratio(X) = gain(X) I split info(X) (2-6)

and it expresses the proportion of information generated by the split that is useful for

classification

Example Consider the following example presented by Quinlan (1993) Table 2-2 shows the

set of training examples

First determine the amount of information gained by selecting the attribute outlook to be a

root of the decision tree This attribute divides the training examples into three subsets

sunny-with five examples two of which belong to the class Play overcast-with four

examples all of which belong to the class Play and rain-with five examples three of

which belong to the class Play To determine info(11 the average information needed to

identify the class of an example in T there are 14 training examples and two decision classes

Nine of these examples belong to the class Play and five belong to the class Dont Play

Info(11 = - 914 10gl (914) - 514 10gl (514) = 094 bits

When using outlook to divide the training examples the information becomes

15

infO(T) = 514 -25 log2 (25) - 35 10g2 (35)

+ 414 -44 10g2 (44) - 04 log2 (04)

+ 514 -35 logl (35) - 25 logl (25) = 0694 bits

By substituting in equation 2-4 the gain of information results from using the attribute

outlook to split the training examples equal to 0246 The gain information for windy is

0048

Figure 2-3 shows a decision tree learned for this problem using the gain criterion The split

information for outlook is determined as follows

split info(T) = - 514 logl (514) - 414 log2 (414) - 514 log2 (514) = 1577 bits

The gain ratio for outlook = 02461577 =0156

16

Figure 2 -3 A decision tree learned using the gain criterion for selecting attributes

The C45 system handles discrete values as well as continuous values To handle an attribute

with continuous values C45 uses a threshold to transform the continuous domain into two

intervals In other words for each continuous attribute C45 generates two branches one

where the values of that attribute is greater than the determined threshold and the other if the

value is less than or equal to the threshold

Tree pruning in C45 is a process of replacing subtrees with small classification validity by

leaves The C45 system uses Laplace ratio for determining the error rate of different subtrees

This ratio is defined as (e+l) (n+2) where n is the number of the training examples and e is

the number of misclassified examples at a given leaf

222 Building Decision Trees Using Statistics-based Criteria

The Chi-square method for selecting attributes was used by Hart (1984) and Mingers (1989a)

in building decisionmiddot trees The method uses Chi-square statistics to measure the association

between two attributes When building decision trees the method is implemented such that it

determines the association between each attribute and the decision classes The attribute to be

selected is the one with greatest value

To determine the Chi-square value for an attribute consider aij is the number of examples in

class number i where the attribute A takes value numberj In other words aij is the frequency

of the combination of decision class number i and the attribute value number j The Chi-square

value for attribute A is given by

n m

Chi-square (A) =L L [ (aij - Eijl2 Eij ] (2shy

i=1 j=1

where n is the number of decision classes m is the number ofvalues of a given attribute Also

17

Eij = (TCi TVj ) T (2shy

8)

where T Ci and T Vj are the total number of examples belong to the decision class a and the total

number of examples where the attribute A takes value vj respectively T is the total number of

examples

Consider Quinlans example in Table 2-2 Table 2-3 shows the frequency of different

combination of values between the decision class and both the outlook and the Windy

attributes Table 2-4 shows the expected values TCi and T Vj of the frequencies in Table 2-3

of different attributes values for different decision classes

To determine the association value between the decision classes and both the attribute Windy

and the attribute Outlook the observed Chi-square values are

Chi-square (Windy Class) =[(3-39)239] + [(3-21)221] + [(6-51)232] + [(2-29)232]

=021 + 039 + 025 + 025 = 11

Chi-square (Outlook Class) = [(2-32)232] + [(4-26)226] + [(3-32)232] + [(3-18118]

+ [(0-14)214] + [(2-18)2 18] =045 + 075 + 02 + 08 + 14 + 002 = 362

18

Applying the same method to the other attributes the results will favor the attribute Outlook

Once that attribute is selected to be a node in the tree the remaining set of examples are divided

into subsets and the same process is repeated on each subset

Table 2-5 shows a summary of these criteria and their basic evaluation function

Table 2-5 Attribute selection criteria and their basic evaluation measure

Info Measure (IM) Gain Gshystatistics and Gain Ratio Chi-square

II

Entropy(S) = - L (freq(~ 5) 1151) logz (freq(CIt 5) 1151 ) I

G-Statistics =2N 1M (N-No of examples) n m

Chi-square (A B) = L L [(aij - ~j)21Eij ] i=l

223 Analysis of Attribute Selection Criteria

This subsection introduces briefly the analysis of different selection criteria which was done by

Mingers (1989a) Mingers compared six attribute selection criteria that are used in decision

tree programs These criteria are the Information Measure (IM) Chi-square G statistics Gin

index of diversity Marshal correction and Gain RaJio The overall results show that the Gain

Ratio criterion had the strongest xesults

In the flISt experiment Minger tested the six criteria on a problem with ambiguous examples

(ie examples may belong to more than one decision class) to observe how the selected criteria

evaluate the given attributes The problem has two decision classes and two attributes X and

Y It was assumed that attribute X is better for classifying the examples than attribute Y The

training examples were unevenly spread between the two values of X Attribute Y has three

values and the examples were spread randomly among them Table 2-6 (a and b) shows the

contingency tables for both attributes Table 2-7 shows a summary of the goodness of split

provided by the six criteria Mingers noted that the measures that are not based on information

19

theory give radiation (attribute X here) less weighL This may be because the zero in the fIrst

row of radiation has a greater influence in the log calculation In the case of using the Chishy

square criterion the value zero adds the maximum association between any two attributes

because the Chi-square value of a zero cell is the expected value of this cell

b)

Now let us demonstrate results from another experiment done by Mingers In this experiment

Mingers used four different data sets to generate decision trees for eleven different criteria In

the final results he compared the total number of nodes and the total error rate provided by

each criterion over all given problems Table 2-8 shows the final results for five selected

criteria only

20

U lgt ~middot results for comparing the total accuracy and size of decision trees different attribute selection criteria from four VVLU

This experiment was performed on four real world data sets These data are concerned with

profiles of BA Business Studies degree students recurrence of breast cancer classifying types

of Iris and recognizing LCD display digits The data was divided randomly 70 for training

and 30 for testing For more details see Mingers (1989a)

23 Learning Decision Structures

Considering the proposed defInition of decision structures given above two related lines of

research are described in this section Gaines (1994) and Kohavi (1994) proposed two

approaches for generating decision structures that share some of the earlier ideas by Imam and

Michalski (1993b)

In the first approach Brian Gaines introduces a method for transforming decision rules or

decision trees into exceptional decision structures The method builds an Exception Directed

Acyclic Graph (EDAG) from a set of rules or decision trees The method starts by assigning

either a rule say RO or a conclusion say CO or both to the root node If it assigns a conclusion

to the root node it places it on a temporary conclusion list Then it generates a new child node

and add to it a new rule say Rl The method evaluates the new rule Rl if it satisfies the

conclusion CO in its parent node it builds a new child and repeats the processes Otherwise if

the rule Rl does not satisfy the conclusion CO it changes the temporary memory with the new

conclusion Cl which is satisfied by the rule RO The same process is repeated until all rules

that have common conditions with the rule at the root are evaluated The method then creates a

21

new child node from the root and repeat the process until all rules are evaluated In the decision

structure nodes containing rules only represent common conditions to all of its children

The main disadvantages of this approach is that it requires discriminant rules to build such a

decision structure Also such a structure is more complex than the traditional decision trees

that are used for decision-making

Figure 2-4 shows an example of set of rules and their equivalent exceptional decision structure

The decision structure can be read as follows It is Safe except if xl=1 amp x2=1 amp x3=1 and

either x4=3 or x5=I then it is Lost except if x6=1 it is Safe except if x7=1 it is Lost

The second approach introduced by Ronny Kohavi learns decision structures from examples

using a bottom-up algorithm The method uses a greedy Hill-climbing algorithm for inducing

Oblivious read-Once Decision Graphs (HOODO) A read-once decision graph is defined as a

decision graph where each attribute occurs at most once along any computational path In other

words for each path from the root to the leaves of the decision structure attributes may occur

as a node once at most However there may be more than one node with the same attribute in

the decision graph Kohavi also defined a leveled decision graph in which the nodes are

partitioned into a sequence of pairwise disjoint sets such that outgoing edges from each level

terminate at the next level An oblivious decision graph is a decision graph where all nodes at

a given level are labeled by the same attribute

22

Safe lt [xl=2] Safe lt [x2=2] Safe lt [x3=2] Safe lt [x4=1] amp [x5=2] Safe lt [x4=1] amp [x5=3] Safe lt [x6=l] amp [x7=2] Safe lt [x6=1] amp [x7=3] Safe lt [x4=2] amp [x5=2] Safe lt [x4=2] amp [x5=3]

Lost lt [x6=2] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x6=3] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x5=1] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=2] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=3] amp [x3=1] amp [x2=I] amp [xl=l]

The algorithm starts by generating a leaf node for each decision class Then it applies a

nondeterministic method to select an attribute to build new level of the decision structure This

attribute is removed from the data and the data is divided into subsets each corresponding to a

complex combination of that attributes values For each subset the process is repeated until all

the examples of a given subset belong to one decision class For example if the selected

attribute say A has two values (0 and 1) and there are two decision classes (CO and CI) The

data is divided into two subsets the fIrst subset contains examples where A takes value 0 and

belong to class CO or takes value 1 and belong to class Cl The second subset of examples is

the set where A takes value 0 and belong to class CI or takes value 1 and belong to class CO

The number of nodes of the fIrst level (after the leave nodes) is expected to be less than or

equal to ka where k is the number of decision classes and n is the number of values of the

selected attribute and it can increase exponentially before that number is reduced

exponentially to one

It is easy for the reader to fIgure out some major disadvantages of such approach including

The average size of such decision structures is estimated to be very large especially when there

23

is no similarity (ie strong patterns) or logical relationship in the data The time used to learn

such decision structure is relatively very high comparing to systems for learning decision trees

from examples and fmally it could be better to search for attribute which reduces the number

of generated subsets of the data instead of nondeterministically selecting an attribute to build a

new level of the decision structure Kohavi provided a comparison between the C45 system

and his system that can be found in (Kohavi 1994) Later Kohavi and Li (1995) used a

deterministic method for attribute selection which minimizes the width of the penultimate level

of the graph

Table 2-9 shows a comparison between the proposed approach and those two approaches The

EDAG and HOODG systems are unreleased prototype systems

or Decision structures are easy to Decision structures are Decision structures are easy to understand difficult to read understand

CHAPTER 3 DESCRIPTION OF THE APPROACH

31 General Methodology

In the proposed approach the function of learning or discovery is separated from the function of

using the discovered knowledge for decision-making The frrst function is performed by an

inductive learning program that searches for knowledge relevant to a given class of decisions

and stores the learned knowledge in the form of decision rules The second function is performed

when there is a need for assigning a decision to new data points in the database (eg bull a

classification decision) by a program that transforms obtained knowledge into a decision

structure optimized according to the given decision-making situation

The Learning Task GivenA set of training examples describing the concept to be learned Learning goal which specifies the decision classes to be learned from the training examples A background knowledge to control the learning process Determine A concept description in a declarative form of knowledge (decision rules) that satisfies the learning goal

The Decision-making Task GivenA set of decision rules in a conjunctive form A description of the new decision-making situation (eg attributes costs and order preference importance or frequency of decision classes etc) One or more examples that needed to be tested under the given decisionshymaking situation A set of parameters to control the learning process Determine A decision structure that suits the given decision-making situation

The decision rules used here are learned by either the AQ1S (Michalski et al 1986) or AQ17

(Bloedorn et aI 1993) learning systems The reasons for selecting decision rules as a form of

23

24

declarative knowledge are that they do not impose any order on the evaluation of the attributes

and due to the lack of order constraints decision rules can be evaluated in many different

ways which increases the flexibility of adapting them to the different tasks of decision-making

(Bergadano et al 1990) Since the number of rules learned per class is typically much smaller

than the number of examples per class generating a decision structure from decision rules can

potentially be done on line

Such virtual decision structures can be tailored to any given decision-making situation The

needed decision rules have to be generated only once and then they can be used many times for

generating decision structures according to changing requirements of decision-making tasks The

method uses the AQDT-2 system (Imam amp Michalski 1994) for learning decision structures

from decision rules Decision structures represent a procedural form of knowledge which makes

them easy to implement but also harder to change Consequently decision structures can be quite

effective and useful as long as they are used in decision-making situations for which they are

optimized and the attributes specified by the decision structure can be measured without much

cost Figure 3-1 shows an architecture of the proposed methodology

Data

Decision -

- Learning knowledge from database The decision making process

Figure 3 -1 Architecture of the AQDT approach

25

It is assumed that the database is not static but is regularly updated A decision-making problem

arises when there is a case or a set of cases to which the system has to assign a decision based on

the knowledge discovered Each decision-making situation is defmed by a set of attribute-values

Some attribute-values may be missing or unknown A new decision structure is obtained such

that it suits the given decision-making problem The learned decision structure associates the

new set of cases with the proper decisions

32 A Brief Description of the AQIS and AQ17 Rule Learning Programs

The decision rules are generated from examples by an AQ-type inductive learning system

specifically by AQ15 (Michalski et albull 1986) or AQI7-DCI (Bloedorn at al 1993) The rest of

this subsection includes a brief description of the inductive learning systems AQ15 and AQI7

AQ15 learns decision rules for a given set of decision classes from examples of decisions using

the STAR methodology (Michalski 1983) The simplest algorithm based on this methodology

called AQ starts with a seed example of a given decision class and generates a set of the

most general conjunctive descriptions of the seed (alternative decision rules for the seed

example) Such a set is called the star of the seed example The algorithm selects from the star

a description that optimizes a criterion reflecting the needs of the problem domain If the

criterion is not defined the program uses a default criterion that selects the description that

covers the largest number of positive examples (to minimize the total number of rules needed)

and with the second priority that involves the smallest number of attributes (to minimize the

number of attributes needed for arriving at a decision)

If the selected description does not cover all examples of a given decision class a new seed is

selected from uncovered examples and the process continues until a complete class description

is generated The algorithm can work with few examples or with many examples and can

optimize the description according to a variety of easily-modifiable hypothesis quality criteria

26

The learned descriptions are represented in the form of a set of decision rules expressed in an

attributional logic calculus called variable-valued logic 1 or VLl (Michalski 1973) A

distinctive feature of this representation is that it employs in addition to standard logic operators

the internal disjunction operator (a disjunction of values of the same attribute in a condition) and

the range operator (to express conditions involving a range of discrete or continuous values)

These operators help to simplify rules involving multi valued discrete attributes the second

operator is also used for creating logical expressions involving continuous attributes

AQ15 can generate decision rules that represent either characteristic or discriminant concept

descriptions depending on the settings of its parameters (Michalski 1983) A characteristic

description states properties that are true for all objects in the concept The simplest

characteristic concept description is in the form of a single conjunctive rule (in general it can be

a set of such rules) The most desirable is the maximal characteristic description that is a rule

with the longest condition part ie stating as many common properties of objects of the given

class as can be determined A discriminant description states properties that discriminate a given

concept from a fixed set of other concepts The most desirable is the minimal discriminant

descriptions that is a rule with the slwnest condition part For example to distinguish a given

set of tables from a set of chairs one may only need to indicate that tables have large flat top

A characteristic description of the tables would include also properties such as have four legs

have no back have four corners etc Discriminant descriptions are usually much shorter than

characteristic descriptions

Another option provided in AQ15 controls the relationship among the generated descriptions

(rulesets or covers) of different decision classes In the IC (Intersecting Covers) mode

rulesets of different classes may logically intersect over areas of the description space in which

there are no training examples In the DC (Disjoint Covers) mode descriptions of different

classes are logically disjoint The DC mode descriptions are usually more complex both in the

number of rules and the number of conditions There is also a DL mode (a Decision List mode

27

also called VL modeM-for variable-valued logic mode) in which the program generates rule sets

that are linearly ordered To assign a decision to an example using such rulesets the program

evaluates them in order If ruleset is satisfied by the example then the decision is made

otherwise the program proceeds to the evaluation of the ruleset +1 In IC and DC modes

rule sets can be evaluated in any order

Alternatively the system can use rules from the AQ17-DCI program for Data-driven

Constructive Induction AQ17-DCI differs from AQ15 mainly in that it contains a module for

generating additional attributes These attributes are various logical or mathematical

combinations the original attributes The program generates a large number of potential new

attributes and seleets from them those most promising based on an attribute quality criterion

To illustrate the format of rules generated by AQ15 (or AQ17-DCI) an exemplary ruleset is

shown in Figure 3-2 The ruleset (that can be re-represented as a disjunctive normal form

expression) describes a voting record of Democratic Representatives in the US Congress

Each rule is a conjunction of elementary conditions Each condition expresses a simple relational

statement For example the condition [State = northeast v northwest] states that the attribute

State (of the Representative) should take the value northeast or northwest to satisfy the

condition

Rl [Gas_concban = yes] amp [Soc_see_cut = no v not registered] R2 [Draft = Yes v not registered]amp[AlaslaLparks = yes v not registered] amp

[Food_stamp_cap = no] amp [State = northeast v northwest] R3 [Chrysler =yes v not registered] amp [Income =low] R4 [Education =yes] amp [Occupation = yes]

Figure 3-2 A ruleset generated by AQ15 for the concept Voting pattern of Democratic Representatives

The above rules were generated from examples of the voting records For illustration below is

an example of a voting record by a Democratic representative

Draft registration=no Ban of aid to Nicaragua=no Cut expenditure on mx missiles=yes Federal subsidy to nuclear power stations=yes Subsidy to national parks

28

in Alaska=yes Fair housing bill=yes Limit on Pac contributions=yes Limit onood stamp program=no Federal help to education=no StateFrom=north east State Population=large Occupation =unknown Cut in social security spending=no Federal help to Chrysler corp=not registered

By expressing elementary statements in the example as conditions and linking conditions by

conjunction the examples can be re-expressed as decision rules Thus decision rules and

examples formally differ only in the degree of generality

33 Generating Decision Structurestrees from Decision Rules

This section describes the AQDT-1 system for learning decision structures (trees) from decision

rules (Imam amp Michalski 1993ab) Also a description of the AQDT-2 method for learning taskshy

oriented decision structure from decision rules is included and fmally the methodology is

illustrated by two examples

Methods for learning decision trees from examples have been very popular in machine learning

due to their simplicity Decision trees built this way can be quite efficient as long as they are

used in decision-making situations for which they are optimized and these situations remain

relatively stable Problems arise when these situations significantly change and the assumptions

under which the tree was built do not hold anymore For example in some situations it may be

difficult to determine the value of the attribute assigned to some node One would like to avoid

measuring this attribute and still be able to classify the example if this is potentially possible

(Quinlan 1990) If the cost of measuring various attributes changes it is desirable to restructure

the tree so that the inexpensive attributes are evaluated f11st A tree restructuring is also

desirable if there is a significant change in the frequency of occurrence of examples from

different classes A restructuring of a decision tree to suit the above requirements is however

difficult to do The reason for this is that decision trees are a form of decision structure

representation and imposes constraints on the evaluation order of the attributes that are not

logically necessary

29

One problem in developing a method for generating decision structures from decision rules is to

design an attribute selection criterion that is based on the properties of the rules rather than of

the training examples A decision rule normally describes a number of possible examples Only

some of them are examples that have actually been observed ie training examples An attribute

selection criterion is needed to analyze the role of each attribute in the rules It cannot be based

on counting the numbers of training examples covered by each attribute-value and the

frequency of decision classes in the training examples as is done in learning decision trees from

examples because the training examples are assumed to be unavailable

Another problem in learning decision trees from decision rules stems from the fact that decision

rules constitute a more powerful knowledge representation than decision trees They can directly

represent a description in an arbitrary disjunctive normal form while decision trees can represent

directly only descriptions in the disjoint disjunctive normal form In such descriptions all

conjunctions are mutually logically disjoint Therefore when transforming a set of arbitrary

decision rules into a decision tree one faces an additional problem of handling logically

intersecting rules

The solution to both problems (attribute selection and logically intersected rules) in the AQDT-2

system is based on the earlier work by Michalski (1978) which introduced a general method for

generating decision trees from decision rules The method aimed at producing decision trees

with the minimum number of nodes or the minimum cost (where the cost was defined as the

total cost of classifying unknown examples given the cost of measuring individual attributes and

the expected probability distribution of examples of different decision classes) More

explanations are provided in the following section

331 Tbe AQDT-2attribute selection metbod

This section describes the AQDT -2 method for building a decision structure from decision rules

The method for building a single-parent decision structure is similar to that used in standard

30

methods of building a decision tree from examples The major difference is that it assigns tests

(attributes) to the nodes using criteria based on the properties of the decision rules (this includes

statistics about the examples covered by each rule in the case of learning rules from examples)

rather than statistics characterizing the frequency of training examples per decision classes per

attribute-values or per conjunctions of both Other differences are that the branches may be

assigned an internal disjunction of values (not only a single value as in a typical decision tree)

and leaves may be assigned a set of alternative decisions with probabilities Also the tests can be

attributes or names standing for logical or mathematical expressions that involve several

attributes or variables In the following we use the terms test and attribute interchangeably

(to distinguish between an attribute and a name standing for an expression the latter is called a

constructed attribute)

At each step the method chooses the test from an available set of tests that has the highest utility

(see below) for the given set of decision rules This test is assigned to the node The branches

stemming from this node are assigned test values or disjoint groups of values (in the form of

logical disjunction if such occur in the rules subsumed groups of values are removed) Each

branch is associated with a reduced set of rules determined by removing conditions in which the

selected attribute assumes value(s) assigned to this branch If all rules in the reduced ruleset

indicate the same decision class a leaf node is created and assigned this decision class The

process continues until all nodes are leaf nodes If it is not possible to reduce further the rule set

because some attribute is declared as unavailable (infmite cost) then the leaf is assigned a set of

candidate decisions with associated probabilities (see Sec 42)

The test (attribute) utility is a combination of one or more of the following elementary criteria 1)

cost which indicates the cost of using each attribute for making decision 2) disjointness which

captures the effectiveness of the test in discriminating among decision rules for different decision

classes 3) importance which determines the importance of a test in the rules 4) value

31

distribution which characterizes the distribution of the test importance over its of values and 5)

dominance which measures the test presence in the rules These criteria are dermed below

~ The cost of a test expresses the effort or cost needed to measure or apply the test

Djsjointness The disjointness of a test is defined as the sum of the class disjointness-the

disjointness of the test for each decision class Suppose decision classes are Cl C2 bull Cm and

decision rule sets for these classes have been determined Given test A let V 1 V 2 bull V m denote

sets of the values (outcomes) of A that are present in rule sets for classes C 1 C2bullbull Cm

respectively If a ruleset for some class say Ct contains a rule that does not involve test A than

Vt is the set of all possible values of A (the domain of A)

Definition 3 -1 The degree ofclass disjointness D(A Ci ) of test A for the ruleset ofclass Ci is

the sum of the degrees of disjointness D(A Cit Cj) between the ruleset for Ci and rulesets for

Cj j=l 2 m j J L The degree of disjointness between the ruleset for Ci and the ruleset for Cj is

dermed by

if Vi Vj ifVj gtVj (3-1) ifVinVj -tljgt amp -tVi amp -tVj ifVinVj=1jgt

where cjgt denotes the empty set Note that exchanging the second and third conditions of equation

(3-1) may seem to be an improved criteria However it does not clearly distinguish between both

cases (Le for both situations the disjointness will be similar) The current equation is better

because it gives higher scores to attributes that classify different subsets of the two decision

classes than attributes that classify only a subset of one decision class

Definition 3-2 The disjointness of the test A for evaluating a given set of decision rules is the

sum of the degrees of class disjointness of each decision class

32

m m Disjointness(A) = L D(A cn where D(A cn = L D(A Ci Cj) (3-2)

i=l i=l i~j

The disjointness of a test ranges from 0 when the test values in rulesets of different classes are

all the same to 3m(m-l) when every rule set of a given class contains a different set of the test

values If two tests have the same disjointness value the attribute to be selected is the one with

less number of values

Definition The A vera~ Number of Tests (ANT) required to make decision from a decision tree

is dermed as the average number of tests (attributes) to be examined from the root of the tree to

any leaf node in order to reach a decision

Definition A decision structure is a one-node-per-level decision structure if at each level there is

only one node and zero or more leaves

Such decision structure can be generated by combining together all branches that associated with

each one a set of decision rules belongs to more than one decision class

Theorem 1 Consider learning one-node-level decision structure for a database with two

decision classes The disjointness criterion ranks first attributes that add minimum number of

tests to the decision tree

Proof Consider all possible distributions of an attributes values within two decision classes Ci

and Cj There are three cases 1) subset (same as super set) 2) non-empty intersection but not

subset 3) no intersection Figure 3-3 shows all possible distributions (the case of having the

same set of values in both classes is trivial one) Consider that branches leading to one subset

with the same decision class are combined into one branch In the first case there will be two

branches only The first leads to a leaf node and the other leads to an intermediate node where

another attribute to be selected The minimum ANT in this case is 53 In the second case three

branches should be created Two branches leads to a leaf node where all values at each branch

belong to only one and different decision class The third branch leads to an intermediate node

33

where another attribute should be selected furtur classifies the decision classes The minimum

ANT in this case is 64 In the third case only two branches will be generated where each leads

to leaf node with different decision class In this case the minimum ANT is 1

Figure 3-4 shows decision trees equivalent to selecting the corresponding attribute Note that in

case of having more than one attribute-value at branches lead to leaves belonging to one decision

class they will be combined into one branch in the decision structure The symbol 1 means

that an attribute is needed to classify the two decision classes In such cases there will be at least

two additional paths

D(A Ci) = 0 D(A CJ) = 1 D(A Ci) =2 D(A Cj) = 2 D(A Ci) =3 D(A Cj) =3

Figure 3-3 Yen Diagrams of Possible combinations of attribute values in two decision classes

The average number of tests required for making a decision in each possible case is determined

in Figure 3-4 It is clear that in the case of two decision classes the disjointness criterion ranks

highly attributes that reduces the average number of tests required for decision-making The

theorem can be proved in the general case

i ~ i Cj C1middot bull Ci Cj

ANT=32middotANT=53 ANT=l

1 means at least one attribute is needed to complete the decision tree Figure 3-4 Decision trees corresponding the Yen diagrams inFigure 3-3

34

Theorem 2 The attribute with the highest non~zero disjointness is the best attribute to classify

the decision classes

Proof Suppose that the number of decision classes is n Assume also that there are two attributes

A and B where D(A) lt D(B) Since the disjointness criterion considers the mutual relationship

between any two decision classes this means that there are more decision classes where D(A

Ci) lt D(B Ci) than those where D(A Ci) gt D(B Ci) (ignoring classes with equal disjointness

for bCth attributes) Hence there are more decision classes where D(A Ci Cj) lt D(B Ci Cj)

than those where D(A Ci Cj) gt D(B Cit Cj) Let us frrst prove that if D(A Ci Cj) lt D(B Cit

Cj) then B classifies the decision classes better than A

For each pair of decision classes Ci and Cj the possible values of the disjointness of any attribute

are 0 14 or 6 For all positive values ofD(B)= 14 or 6 it is clear that attribute B should have

smaller ANT than attribute A which has lower disjointness This means that if D(A Cit Cj) lt

D(B Cit Cj) then attribute B can classify both decision classes better than attribute A

Similarly B is better for classifying more pairs of decision classes than A This implies that B is

a better classifier than A

Importance The second elementary criterion the importance of a test is based on the

importance score (IS) introduced in (Imam Michalski amp Kerschberg 1993) In the obtained

rules each test is assigned a score that represents the total number of training examples that

are covered by the rules involving this test Decision rules learned by an AQ learning program

are accompanied with information on their strength Rule strength is characterized by t-weight

and u~weight The t-weight (total-weight) of a rule for some class is the number of examples of

that class covered by the rule The importance score of a test is the aggregation of the totalshy

weights of all rules that contain that test in their condition part Given a set of decision rules for

m decision classes Cl Cm and involving n tests AlbullAn and the number of rules associated

with class Ci is denoted by tlrit The importance score is defmed as follow

35

Definition 3 -3 The importance score IS(Aj) of the test A j is detennined by

m IS(Aj)= L IS(Aj Ci) (3-31)

i=l

where ri IS(Aj Ci) = L Rik(Aj) (3-32)

k=l

and Rik the weight of a test Aj in the rule Rk of class Ci is given by

t - weight if A j belongs to rule Rik Rik (Amiddot) = (3-4)J 0 otherwise

where i=I n ik=I ri j=I m

The importance score method has been separately compared as a feature selection method with

a genetic algorithm-based method (Imam amp Vafaie 1994) The importance score method

produced an equal or higher accuracy on three real-world problems than those reported by the

GA method while selecting fewer attributes In addition the IS method was significantly faster

than the GA

yalue distribution The third elementary criterion value distribution concerns the number of

legal values of tests Given two tests with equal importance scores this criterion prefers the test

with the smaller number of legal values Experiments have shown that this criterion is especially

useful when using discriminant decision rules

Definition 34 A value distribution VD(Aj) of a test A j is defined by

VD(Aj) = IS(Aj) I Vj (3-5)

where v is the number of legal values of Aj

Dominance The fourth elementary criterion dominance prefers tests that appear in large

numbers of rules as this indicates their high relevance for discriminating among the rulesets of

given decision classes Since some conditions in the rules have values linked by internal

disjunction counting such rules directly would not reflect properly their relevance Therefore

36

for computing the dominance the rules are counted as if they were converted to rules that do not

have internal disjunction Such a conversion is done by multiplying out the condition parts of

the rules containing internal disjunction For example the condition part [x3=1 v 3]amp[x4=1] is

multiplied out to two rules with condition parts [x3=1]amp[x4=1] and [x3=3]amp[x4=I]

The above criteria are combined into one general test measure using the lexicographic

evaluation functional with tolerances (LEF) (Michalski 1973) LEF is a list of some or all of

the above elementary criteria each associated with a Ittolerance threshold It in percentage The

criteria are applied to tests in the order defined by LEF A test passes to the next criterion only if

it scores on the previous criterion within the range defined by the tolerance (from the top value)

The default LEF is

ltCost d Disjointness t2 Importance t3 Value distr t4 Dominance 0gt (3-6)

where d a t3 t4 and t5 are tolerance thresholds (in percentages) their default values are O

The default value of the cost of each test is 1

The above LEF ranks attributes this way First attributes are evaluated on the basis of their cost

If two or more attributes have the same cost they ranked by their degree of disjointness If two

or more attributes still share the same top score or their scores differ less than the assumed

tolerance threshold t2 the method evaluates these attributes using the second (Importance)

criterion If again two or more attributes share the same top score or their scores differ less than

the tolerance threshold t3 then the third criterion normalized IS is used then similarly the

fourth criterion (dominance) If there is still a tie the method selects the best attribute

randomly

If there is a non-uniform frequency distribution of examples of different classes then the

selection criterion uses a modified definition of the disjointness Namely the previously defmed

37

disjointness for each class is multiplied by the frequency of the class occurrence The class

occurrence is the expected number of future examples that are to be classified to a given class

m

Disjointness(A) = L D(A CO Frq(Ci) (3-7) i=l

where Frq(Cn is the expected frequency of examples of class Cit and it is assumed to be given

by the user The attribute ranking criteria in this case is defined by the LEF

ltCost d Disjointness t2 Importance 13 Normalized-IS 14 Dominance 15gt (3-8)

where the Cost denotes the evaluation cost of an attribute and is to be minimized while other

elementary criteria are treated the same way as in the default version

332 The AQDT-2 algorithm

The AQDT-2 algorithm constructs a decision structure from decision rules by recursively

selecting at each step the best test according to the ranking criteria described above and

assigning it to the new node The process stops when the algorithm creates terminal branches that

are assigned decision classes To facilitate such a process the system creates a special data

structure for each concept description (ruleset) This structure has fields such as the number of

rules the number of decision classes and the number of attributes present in the rules A set of

pointers connect this data structure to set of data structures each represents one decision class

The decision class structure contains fields with information on the number of rules belong to

that class the frequency of the decision class etc It is also connected to a set of data structures

representing the decision rules within each decision class The system creates independently a set

of data structures each is corresponding to one attribute Each attribute description contains the

attributes name domain type the number of legal values a list of the values the number of

rules that contain that attribute and values of that attribute for each rule The attributes are

arranged in an array in a lexicographic order flISt in the descending order of the number of rules

38

that contain that attribute and second in the ascending order of the number of the attributes

legal values

The system can work in two modes In the standard mode the system generates standard

decision trees in which each branch has a specific attribute-value assigned In the compact

mode the system builds a decision structure that may contain

A) or branches Le branches assigned an internal disjunction of attribute-values whenever it

leads to simpler structures For example if a node assigned attribute A has a branch marked by

values 1 v 2 then the control passes along this branch whenever A takes value 1 or 2 The

program creates or branches on the basis of the analysis of the value sets Vi while computing

the degree of attribute disjointness

B) nodes that are assigned derived attributes that is attributes that are certain logical or

mathematical combinations of the original attributes To produce decision structures with

derived attributes the input decision rules are generated by AQ17 (rather than AQI5) The

AQ17 rules may contain conditions involving attributes constructed by the program rather than

those originally given

To generate decision structures from rules the AQDT-2 method prefers either characteristic or

discriminant disjoint rule descriptions (given by an expert or learned by a system) Disjoint rules

are more suitable for building decision structures Assume that the description of each class is in

the form of a ruleset and that this set is the initial ruleset context The AQDT algorithm is

The AQDT2 Algorithm

Step 1 Evaluate each attribute occurring in the ruleset context using the LEF attribute ranking

measure Select the highest ranked attribute Let A represents this highest-ranked

attribute

Step 2 Create a node of the tree (initially the root afterwards a node attached to a branch) and

assign to it the attribute A In standard mode create as many branches from the node as

39

there are legal values of the attribute A and assign these values to the branches In

compact mode (decision structures) create as many branches as there are disjoint value

sets of this attribute in the decision rules and assign these sets to the branches

Step 3 For each branch associate with it a group of rules from the ruleset context that contain a

condition satisfied by the value(s) assigned to this branch For example if a branch is

assigned values i of attribute A then associate with it all rules containing condition [A=

i v ] If a branch is assigned values i v j then associate with it all rules containing

condition [A= i v j v ] Remove from the rules these conditions If there are rules in

the ruleset context that do not contain attribute A add these rules to all rule groups

associated with the branches stemming from the node assigned attribute A (This step is

justified by the consensus law [x=1] == [x=1] amp [y=a] v [x=1] amp [y=b] assuming

that a and b are the only legal values of y) All rules associated with the given branch

constitute a ruleset context for this branch

Step 4 If all the rules in a ruleset context for some branch belong to the same class create a leaf

node and assign to it that class If all branches of the trees have leaf nodes stop

Otherwise repeat steps 1 to 4 for each branch that has no leaf

To select an attribute to be a node in the decision tree (step 1 and 2 of the algorithm) the

algorithm performs two independent iterations In the first iteration it parses through all decision

rules and determine information about each attributes This information includes the importance

score of each attribute the number of rules containing a given attribute the disjoint value sets of

each attribute and the attribute values used in describing each decision class The second

iteration is only performed if the disjointness criterion is ranked first in the LEF function The

second iteration evaluates the attributes disjointness for each decision class against the other

decision classes

40

To determine the complexity of this process suppose that the maximum number of conditions

fonned by a single attribute value in one rule is s (where s ~ n the number of attributes) and r is

the total number of decision rules (in all decision classes)

m

r=LRt (m is the number of decision classes) 1=1

where Ri is the number of rules in decision class Ci bull The complexity of the frrst iteration can be

determined as

Cmpx(Itcl) =O(r s)

In the second iteration the disjointness is calculated between the decision classes and all

attributes The complexity of the second iteration can be given by

Cmpx(Itc2) = O(n m)

Assume that at each node 1 is the maximum of the number of decision classes to be classified at

this node and the number of rules associated with this node

l=max mr (3-9)

Since these two iteration are independent of each other the complexity of the AQDT algorithm

for building one node in the decision tree say Node Complexity NC(AQDT) is given by

NC(AQDT) = 0(1 n)

Usually I equals to the number of rules associated with the given node Thus the AQDT

complexity for building one node is a function of the number of attributes multiplied by the

number of rules associated with this node

At each level of the decision tree these two iterations are repeated for each non-leaf node The

Level Complexity of the AQDT algorithm LC(AQDT) can be given by

LC(AQDT) lt 0(1 n)

Which is less than the complexity of generating the root of the decision tree To explain this

consider the maximum possible number of non-leaf nodes at one level is half the number of the

initial decision rules (integer of rll) Figure 3-5-a shows an example of such situation where at

41

the lowest level each node classifies only two rules each belongs to different decision class This

decision tree could be a multimiddotvalues tree However the number of non-leaf nodes at each level

will be twice or more than the number of nonmiddot leaf nodes at the previous level Considering the

level complexity of the AQDT algorithm equal to (1 s 0) where 0 is the number of non-leaf

nodes at the given level In such cases it is either (1 0 S r) or (1 s lt r) In Figure 3middot5middota the

complexity of the AQDT algorithm at any lower level is given by

LC(AQDT) =0(2 s r(2) lt O(n I) =NC(AQDT)

a) per one level b) per one path

Figure 3 -5 Decision trees showing the maximum number of none leaf nodes

Note also that after selecting an attribute to be the root of the decision structure this attribute

and all conditions containing that attribute are removed from the data structure of the algorithm

Also if a leaf node is generated all rules belonging to the corresponding branch will not be

tested again

Since the disjointness criterion selects the attribute which minimizes the average number of tests

ANT it is true that the AQDT algorithm generates decision trees with the least number of levels

The number of levels per a decision tree is supposed to be less than or equal to the minimum of

both the number of attributes and the number of rules Consider k as the number of levels in a

given decision tree

k S min mr (3-10)

There are two cases represent the most complex situations Figure 3-5-a and 3-5-b In the fIrst

case where the decision rules were divided evenly the number of levels will be a function of the

42

logarithm of the number of rules In such case the complexity of the AQDT algorithm for

generating a decision tree from a set of decision rules is given by

Complexity(AQDT) = 0(1 n log r) (3-11)

The other situation is when the generated decision tree has the maximum number of levels The

maximum possible number of levels per a decision tree equals to one less than the number of

decision rules Figure 3-5-b Using the disjointness criterion it is not likely to get such a decision

tree because it has the maximum average number of test (ANT) that can be determined from the

same set of nodes and leaves However such a decision tree can be generated if the number of

decision classes is one less thanthe number of attributes In such case any disjQint decision rules

should have a maximum length that is less or equal to the floor of the logarithm of the number of

attributes Thus the level complexity of this decision tree is estimated as

LC(AQDT) =0(1 log n)

The maximum number of levels in such decision tree is k-l Thus the complexity of the AQDT

algorithm in such cases is given by

Complexity(AQDT) = 0(1 k log n) (3-12)

Since k is the minimum of n and r then n is the maximum of k and n Also since in (3-12)

r=n+1 then log r is greater than log n From 3-10 3-11 and 3-12 the complexity of the AQDT

algorithm is determined by

Cmplx(AQDT) = OCr k log 1) (3-13)

333 An example illustrating the algorithm

The following simple example illustrates how AQDT is used in selecting the optimal set of

testing resources for testing a software Suppose there are three tools for testing a software 1)

modeling (Tl) 2) checklist (T2) and 3) par_simul (T3) Also assume that there are four

different factors that affect the selection of any tool 1) the cost of using the tool (xl) 2) the

metrics that support the tool (x2) 3) the best phase for applying the tool (x3) and 4) type of tool

43

(automated semi-automated or manual) (x4) Table 3-1 shows these attributes and their possible

values

Suppose that the domain expert provided a set of rules to be used in testing a software Figure 3shy

6 shows a sample of these rules in AQ15c formats

Table 3-1 The available tools and the factors that affect the prclCes

Tl lt= [xl=2] amp [x2=2] v [xl=3] amp [x3=1 v 3] amp [x4=1] T2 lt= [xl=l v 2] amp [x2=3 v 4] v [xl=3] amp [x3=1 v 2] amp [x4=2] T3 lt= [xl=l] amp [x2=1] v [xl=4] amp [x3=2 v 3] amp [x4=3]

Figure 3-6 Decision rules for selecting the best tool for testing software

These rules can be interpreted as

Rule 1 Use the fltst tool for testing if you need average cost and the tool is

supported by the requirement metric

Rule 2 Use the fIrst tool for testing if you can afford high cost for testing either

in the requirement or the analysis phases and you need an automated tool

Rule 3 Use the second tool for testing if the cost limit is low or average and the

tool is supported by the system usage metric or the intractability metric

Rule 4 Use the second tool for testing if you can afford high cost for testing

either in the requirement or the design phases and you need a manual tool

Rule 5 Use the third tool for testing if the you are limited to low cost and the

tool is supported by the error rate metric

Rule 6 Use the third tool for testing if you can afford very-high cost for testing

either in the requirement or the system usage phases and you need a semishy

automated tool

44

Table 3-2 presents information on these rules and the disjointness values for all attributes For

each class the row marked Values lists values occurring in the ruleset for this class For

evaluating the disjointness of an attribute say A each rule in the ruleset above that does not

contain attribute A is characterized as having an additional condition [A= a vb ] where a b

are all legal values of A

The row Class disjointness specifies the class disjointness for each attribute The attribute xl

has the highest disjointness (11) and is assigned to the root of the tree For simplicity assume

the tolerances for each elementary criterion equal O

From the rules in Figure 3-6 we can also determine disjoint groupings of attribute-values used

for the compact mode of the algorithm This done as follows 1) determine for each attribute the

sets of values that the attribute takes in individual decision rules and remove those value sets

that subsume other value sets The remaining value sets are assigned to branches stemming from

the node marked by the given attribute For example x 1 has the following value sets in the

individual decision rules 2 3 I 2 I and 4 (Figure 3-6) Value set I 2 is

removed as it subsumes 2 and I In this case branches are assigned individual values of the

domain of x1 For attribute x2 the value sets are I 2 34 and I 2 3 4 In this case

branches are assigned value sets I 2 and 34

45

Attribute Xl ranks highest (as it is the highest disjointness) and is assigned to the root of the

tree Four branches are created each one corresponding to one of XIs possible values Since all

rules containing [x1=4] belong to class TI the branch marked by 4 is ended by a leafTI Rules

containing other values of X1 belong to more than one class This process is repeated for each

subset of rules until the decision tree is completed Figure 3-7 shows a decision structure learned

by AQDT-2 from the rules in Figure 3-6 The decision structure in Figure 3-7 can be used in

making decisions on which tools can be used for testing a given software

xl

Complexity No of nodes 4 No of leaves 7

Figure 3-7 A decision structure learned for classifying so tware testing tools

Figure 3-8-a shows the diagrammatic visualization of the decision rules and Figure 3-8-b shows

the visualization of the derived decision tree Each diagram in Figure 3-8 consists of cells

representing one combination of attribute-values Attributes and their legal values are shown on

scales surrounding the diagram (eg the horizontal scale for xl shows values 1 2 3 and 4)

Rules are represented by collections of cells in the intersection of the rows and columns

corresponding to the conditions in the rules

The shaded areas correspond to decision rules Rules of the same class have the same type of

shading Empty cells correspond to combination of attribute-values not assigned to any class For

illustration collections of cells corresponding to some of the initial rules are marked R 11 R 21

R31 and R32 Rll denotes the IlfSt rule of Class T1 ie [x1=2] amp [x2=2] R21 denotes the

IlfSt rule of class T2 ie [xl= 1 v 2] amp [x2= 3 v 4] R31 the first rule of class TI ie [x1=1]

46

amp [x2 =1] and denotes R32 denotes the second rule of class T3 Le [xl=4] amp [x3=2 v 3] amp

[x4=3]

a) Decision rules b) Derived decision tree

Comparing diagrams in Figure 3-8-a and 3-8-b AQDT-2 decision tree represents a slightly more

general description of Concepts Tl TI and T3 than the original rules Let us assume that it is

very costly to know which metrics suppon the required tools In other words suppose that we

would like to select the best tools independently of the metric they suppon (this is indicated to

AQDT-l by assigning very high cost to attribute x2) The algorithm again assigns attribute xl to

the root of the tree When xl takes value 1 or 2 it is impossible to assign a specific decision

without measuring attribute x2 For the value 1 of xl the recommended tool can be either TI or

T3 (see the diagram in Figure 3-9-a) and for the value 2 of xl the recommended tool can be

either Tl or T3 However for the value 3 of xl one can make a specific decision after

measuring attribute x4 For the value 1 of x4 tool is Tl and for the value 2 of x4 the

recommended tool is TI Figure 3-9-b shows another decision tree where the type of the tool

was ignored from the data Those decision trees are called indeterminate because some of their

leaves are assigned a disjunction of two or more class names

47

xl

xl

1 12

a) Ignoring the supporting metric b) Ignoring the type of the tooL Figure 3 -9 Decision trees learned ignoring the support metric and the type of the testing tool

It is clear that the given set of rules depend highly on amount of money can be spent Let us now

suppose that the cost attribute xl which was determined as the highest ranked attribute cannot

be measured The algorithm selected x4 to be the root of the new decision tree After continuing

the algorithm the tree in Figure 3-10 is obtainedThe nodes that are assigned one class indicate

situations in which it is possible to make a specific decision without knowing the value of

attribute xl

x4

Figure 3-10 A decision tree learned without the cost attribute

34 Tailoring Decision Structures to the Decision-making situation

Decision structures are among the simplest structures for organizing a decision-making process

A decision structure specifies explicitly the order in which attributes of an object or situation

need to be evaluated in the process of determining a decision A standard way to generate a

48

decision structure is to learn it from examples of decisions Such a process usually aims at

obtaining a decision structure that has the highest prediction accuracy that is the highest rate of

assigning correct decisions to given situations There can usually be a large number of logically

equivalent decision structures (Michalski 1990) As such they may have the same predictive

accuracy but differ in the way they organize the decision process and thus may differ in the cost

of arriving at a decision To minimize the average decision cost one needs to take into

consideration the distribution of the costs of attribute evaluation and the frequency of different

decisions This report presents an approach to building such Task-oriented decision structures

which advocates that they are built not from examples but rather from decision rules Decision

rules are learned from examples using the AQ15 or AQ17 inductive learning program or are

specified by an expert An efficient algorithm and a new system AQDT-2 that transforms

decision rules to task-oriented decision structures The system is illustrated by applying it to the

problem of learning decision structures in the area of construction engineering (for determining

the best wind bracing for tall buildings) In the experiments AQDT-2 outperformed all other

programs applied to the same data

Decision-making situations can vary in several respects In some situations complete

information about a data item is available (ie values of all attributes are specified) in others the

information may be incomplete To reflect such differences the user specifies a set of parameters

that control the process of creating a decision structure from decision rules AQDT-2 provides

several features for handling different decision-making problems 1) generating a decision

structure that tries to avoid unavailable or costly attributes (tests) 2) generating unknown

leaves in situations where there is insufficient information for generating a complete decision

structure 3) providing the user with the most likely decision when performing a required test is

impossible 4) providing alternative decisions with an estimate of the likelihood of their

correctness when the needed information can not be provided

49

341 Learning Costmiddot Dependent Decision Structures

As described in Sec 2 the LEF criterion enables the system to take into consideration the cost of

measuring tests (attributes) in developing a decision structure In the default setting of LEF the

tests cost is the [lIst criterion and its tolerance is O This means that only the least expensive

attributes pass to the next step of evaluation involving other elementary criteria If an attribute

has high cost or is impossible to measure (has an infinite cost) the LEF chooses another

cheaper attribute ifpossible

342 Assigning Decision Under Insufficient Information

In decision-making situations in which one or more attributes cannot be measured the system

may not be able to assign a definite decision for some cases If no more information can be

obtained but a decision has to be made it is useful to know the probability distribution for

different candidate decisions (Smyth Goodman amp Higgins 1990) The most probable decision is

then chosen The probability distribution can be estimated from the class frequency at the given

node Let us consider a node in the structure connected to the root by the sequence bl b2 bkof

attribute-values Let P(Ci I bl b2 bk) denote the conditional probability of class Ci i=I2 m

at that node given that an example to be classified has attribute-values assigned to branches bl

b2 bull bk in the decision structure Using Bayesian formula we have

P(Ci I bl~ bull bk) = P(Ci) P(bl bk I Ci) P(bl bk) (3-9)

where P(Ci) the apriori probability of class Ci P(bl bk I Ci) is the probability that the

example has attribute-values blbull bk given that it represents decision class Ci To approximate

these probabilities let us suppose that Wi is the number of training examples of Class Ci that

passed the tests leading to this node and twi - the total number of training examples of Class

Ci Let us use these numbers to estimate the probabilities in (9) Assuming that the apriori class

probabilities correspond to the frequency of training examples from different classes we have

50

m P(Ci) = twi IL twj (3-10)

j=l

P(b1 bkl Ci) = Wi I twi (3-11)

m m P(b1 bk) =L Wj IL twj (3-12)

j=l j=1

By substituting (1O) (11) and (12) in (9) we obtain

m P(Ci I b1 bk) = wilL Wj (3-13)

j=l

A related method for handling the problem of unavailability of an attribute is described by

Quinlan (1990) Quinlans method however assigns probabilities to the most probable decision

at the node associated with such an attribute The method does not restructure the decision tree

appropriately to fit the given decision-making situation (in this case to avoid measuring xl)

AQDT -2 assigns probabilities o~ly after it first created a decision structure that suits the given

decision-making si~ation An example is presented in section 42

343 Coping with noise in training data

The proposed methodology can be easily extended to handle the problem of learning from noisy

training data This is done based on ideas of rule truncation described in (Niblett amp Bratko 1986

Bergadanoet al 1992 Cestnik amp Karalic 1991) When noise is expected in the training data

the decision rules with t-weight below a certain threshold (reflecting the expected noise level) are

removed The rule truncation method seems to result in more accurate decision structures than

decision tree pruning method because truncation decisions are solely based on the importance of

the given rule or condition for the decision-making regardless of their evaluation order (unlike

the decision tree pruning which can only prunes attributes within a subtree and thus cannot

freely chose attributes to prune) Examples are presented in section 4

ACKNOWLEDGMENT

I would like to thank Professor Ryszard S Michalski PRC Chaired Professor of Computer

Science and Engineering and my Dissertation Director for his support encouragement and

guidance I would like to thank my committee members Professor Larry Kerschburg Chair of the

Department of Information Systems and Software Engineering Professor David Rine Professor

of Information Technology and Engineering and Professor David Schum Professor of Excellence

in Information Technology and Engineering for their encomagement and help with many aspects

of my PhD

I would like to thank Professor Tomaz Arciszewki System Engineering Department for providing

me with application problems Ronny Kohavi Stanford University for discussion and providing

me with some related work on learning decision structures and decisionmiddot graphs and Professor

George Tecuci Computer Science Department for pointing some related work

I would like to thank my colleagues Nabil Al-Kharouf for reviewing my dissertation Eric

Bloedorn for reviewing an earlier draft of my dissertation and for using his program AQ17-DCI in

my experiments Srinivas Gutta for providing some application for my PhD work Mike Heib for

reviewing an earlier draft of my dissertation and helping me find relevant articles Ken Kaufman

for reviewing an earlier draft of my thesis Mark Maloof for providing me with script files which

made it easier to iteratively run AQ1Sc Halah Vafaie for working with her on application and

comparison of different aspects of my work and Janusz Wnek for using his DIAV program for

explaining my results

I would like to thank Professor Andrew P Sage Dean of the School of Information Technology

and Engineering and Professor Kenneth Bumgarner Dean of Student Services and Associate ViCe

President of George Mason University for their support and Professor Murray W Black

Associate Dean of the School of Infonnation Technology and Engineering for guidance on

preparing the PhD proposal

I would like to thank conferences organizers who supported me to attend their conferences and

present parts of my PhD work The organizers include Professor Moonis Ali Professor Frank

Anger Professor Zbigniew Ras the American Association for Artificial Intelligence and others I

would like also to thank the organizing committee of the Florida Artificial Intelligence Research

Symposium FLAIRS-95 for organizing my first workshop Thanks goes to Dr Doug Dankel Dr

Howard Hamilton Dr John Stewman and Dr Dan Tamir

I would like also to thank the many individuals who helped me in any way during my PhD Those

include Dr Ashraf Abdel-Wahab Dr Jerzy Bala Dr Alex Brodsky Dr Richard Carver Dr

Kenneth De Jong John Doulamis Tom Dybala Dean Ibrahim Farag Dr Ophir Frieder Csilla

Frakes Dr Mohamed Habib Ali Hadjarian Dr Hugo De Garis Zenon Kulpa Dr Paul Lehner

Dr George Michaels Dr Eugene Norris Dean James Palmer Mitch Potter Dr Ahmed Rafea

Jim Ribeiro Jsyshree Sarma Dr Arun Sood Dr Clifton Sutton Bradley Utz Dr Tibor Vamos

Patricia Zahra Dr Shaker Zahra and Dr Jianping Zhang

~J I yarJ I JJ I ~

U-t-oJ ~II ~) ~ oWAIl J

Dedication

To my mother my brothers and my sister

TABLE OF CONTENTS

TI1LE Page

ABS1RAcr II 1

CHAPrER 1

IN1RODUCI10N II 3

11 Motivation and Overview 3

12 The Problem Statement 6

CHAPrER2

REIAfED RESEARCFI 7

21 Learning Decision Trees from Decision Diagrams 7

22 Learning Decision Trees from Examples 10

221 Building Decision Trees Using Information-based Criteria 11

222 Building Decision Trees Using Statistics-based Criteria 16

223 Analysis of Attribute Selection Criteria 18

23 Learning Decision Structures 19

CHAPrER3

DESCRIPTION OF TIlE APPROACFI 23

31 General Methodology 23

32 A Brief Description of the AQ15 and AQ17 RuleLeaming Systems 25

33 Generating Decision Structures From Decision Rules 28

331 The AQDT-2 attribute selection method 29

332 The AQDT-2 algoridlm 37

vi

333 An example illustrating the algorithm 42

34 Tailoring Decision Structure to a decision-making situation 47

341 Learning Cost-Dependent Decision Structures 49

342 Assigning Decision Under Insufficient Infonnation 49

343 Coping with noise in training data 50

35 Analysis of the AQDT-2 Attribute Selection Criteria 51

36 Decision Structures vs Decision Trees 53

CHAPTER 4

EMPRICAL ANALYSIS AND COMPARATIVE STUDY OF TIlE MElHOD 58

41 Description of the Experimental Analysis 59

42 Experiments With Average Size Complex and Noise-Free Problems

Wind Brac-ings 60

43 Experiments With Small Size Simple and Noise-Free Problems

e MONK-I 69

44 Experiments With Small Size Complex and Noise-Free Problems

MONK-2 76

45 Experiments With Small Size Simple and Noisy Problems

MONK-3 79

46 Experiments With Large Size Complex and Noise-Free Problems

Diagnosing Breast Cancer 83

47 Experiments With Large Size Complex and Noisy Problems

Musm()()m classifications 84

48 Experiments With Small Size Structured and Noise-Free Problems

East-West Trains 85

49 Experiments With Small Size Simple and Noisy Problems

vii

Congressional Voting Records (1984) 87

410 Analysis of the Results 88

ClIAPfER 5 CONCIUSIONS 95

51 Summary 95

52 Contributions 96

REFERENCES 98

VITA 102

viii

LIST OF TABLES

No TITLE Page

2-1 An example of a decision table 9

2-2 A set of training examples used to illustrate the C45 system 15

2-3 The frequency of different attributes values to different decision classes 17

2-4 The expected values of the frequency of examples in Table 2-3 17

2-5 Attribute selection criteria and their basic evaluation measure 17

2-6 The contingency tables of Mingers example 18

2-7 Mingers results for determining the goodness of split 19

2-8 Mingers results for comaring the total accuracy and size of decision trees

provided by different attribute selection criteria from four problems 19

2-9 Comparing the AQDT approach with EDAG and HOODG approaches 22

3-1 The available tools and the factors that affect the process of testing a software 43

3-2 Calculating the disjointness of each attribute 44

3-3 Evaluation of the AQDT-2 criteria on the Quinlans example in Table 2-2 51

3-4 The data used in Mingers f11st experiments 52

3-5 The performance of AQDT-2 criteria (compare with the other criteria in Table 2-6) 52

3-6 The possible ranking domains and using condition of AQDT-2 criteria 53

3-7 Comparison between Decision Structures and Decision Tree 54

4-1 Evaluation of AQDT-2 attribute selection criteria for the wind bracing problem 62

4-2 The predictive accuracy of AQ15c and AQDT-2 for the wind bracing problem 67

4-3 Evaluation of AQDT-2 attribute selection criteria for the MONK-l problem 71

4-4 The predictive accuracy of AQ15c and AQDT-2 for the MONK-l problem 73

ix

4-5 The predictive accuracy of AQ15c and AQDT-2 for the MONK-2 problem 77

4-6 The predictive accuracy of AQ15c and AQDT-2 for the MONK-3 problem 81

4-7 The set of attributes and their values used in the trains problem 86

4-8 A tabular snmmary of the predictive accuracy of decision trees obtained

by AQDT -2 and C45 for the congressional voting data 88

4-9 Summary of the best parameter settings for the first subfunction of the approach

with different data characteristics 89

4-10 Summary of the performance of AQDT-2 and C45 on different problems 90

x

LIST OF FIGURES

No TITLE Page

2-1 An example to illustrate how attributes break rules 8

2-2 A decision tree learned from the decision table in Table 2-1 10

2-3 A decision tree learned using the gain criterion for selecting attributes 15

2-4 Decision rules and their Exception Directed Acyclic Graph (EDAG) 21

3-1 Architecture of the AQDT approach 24

3-2 A ruleset generated by AQ15 for the concept Voting pattern of

Democratic Representatives 27

3-3 Ven Diagrams ofPossible combinations of attribute values in two decision classes 33

3-4 Decision trees corresponding the Ven diagrams in Figure 3-3 33

3-5 Decision trees showing the maximum number of none leaf nodes 41

3-6 Decision rules for selecting the best tool for testing a software 43

3-7 A decision structure learned for classifying software testing tools 45

3-8 A diagrammatic visualization of the decision rules and the derived decision tree 46

3-9 Decision trees learned ignoring the support metric and the type of the testing tool 47

3-10 A decision tree learned without the cost attribute 47

3-11 Decision structures learned by AQDT-2 using different criteria 55

3-12 The Imams example Example where learning decision structures (trees)

from rules is better than learning them from examples 56

3-13 An example where decision rules are simpler than decision trees 57

4-1 Design of a complete experiment 59

Xl

4-2 Decision rules determined by AQ15c from the wind bracing data 61

4-3 A decision tree learned by C45 for the wind bracing data 63

4-4 A decision structure learned from AQ15c wind bracing rules 64

4-5 A decision structure that does not contain attribute xl 64

4-6 A decision structure without Xl with candidate decisions assigned to leaves 65

4-7 A decision structure determined from rules in Figure 4-4 under the

assumption of 10 classification error in the training data 65

4-8 Diagramatic visualization of decision trees learned for different decision

making situations for the wind bracing data 66

4-9 The accuracy of AQDT-2 and AQ15c with different parameter settings

for the wind bracing PIoblem 68

4-10 Analyzing different parameter setting of AQDT-2 using the wind bracing data 69

4-16 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-20 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-11 A comparison between AQ15c and AQDT-2 against C45 on the wind bracing data 69

4-12 A visualization diagram of the MONK-l problem 70

4-13 Decision rules learned by AQ15c for the MONK-1 problem 71

4-14 The decision tree for the MONK-1 problem generated both by AQDT-l 72

4-15 Compact decision structures generated by AQDT-2 for the MONK-1 problem 72

for the MONK-l Problem 74

4-17 Analyzing different parameter setting of AQDT-2 with the MONK-l data 75

4-18 Comparing AQ15c and AQDT-2 against C45 using the MONK-1 problem 75

4-19 A visualization diagram of the MONK-2 problem 76

for the MONK-2 Problem 78

4-21 Analyzing different parameter setting of AQDT-2 with the MONK-2 data 79

xii

4-22 Comparing AQ15c andAQDT-2 against C45 using the MONK-2 problem 79

4-24 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-35 A visualization diagram of a decision tree learned by AQDT -2 after reducing

4-23 A visualization diagram of the MONK-3 problem 80

for the MONK-3 Problem 82

4-25 Analyzing different parameter setting of AQDT-2 with the MONK-3 data 82

4-26 Comparing AQ15c and AQDT-2 against C45 using the MONK-3 problem 83

4-27 Comparing AQ15c and AQDT-2 against C45 using the breast cancer problem 84

4-28 Comparing AQ15c and AQDT-2 against C45 using the mushroom problem 85

4-29 Decision structures learned by AQDT-2 for different decision-making situations 87

4-30 Comparing decision trees for the Cong Voting-84 data learned by C45 amp AQDT -2 88

4-31 A visualization diagram of decision rules learned by AQ15c for the MONK-2 91

4-32 A visualization diagram shows the testing errors for the rules in Figure 4-31 92

4-33 A visualization diagram of the decision tree learned by AQDT-2 for the MONK-2 93

4-34 A visualization diagram shows testing errors of AQDT-2 decision tree bull 94

the generalization degree to 1 94

DERIVING TASKmiddotORIENTED DECISION STRUCTURES

FROM DECISION RULES

Ibrahim M Fahmi Imam PhD

School of Information Technology and Engineering

Geotge Mason University Fall 1995

Ryszard S Michalski Advisor

ABSTRACT

This dissenation is concerned with research on learning task-oriented decision structures from

decision rules The philosophy behind this research is that it is more appropriate to learn

knowledge and store it in a declarative form and then when a decision making situation occurs

generate from this knowledge the decision structure that is most suitable for the given decision

making situation Learning decision structures from decision rules was first introduced by

Michalski (1978) The flIst implementation of this approach was done by Imam and Michalski

(1993 ab) called AQDT-l

This approach separates the function of generating a knowledge-base from the function of using

the knowledge-base for decision-making The first function focuses on learning accurate

consistent and complete concept description expressed in a declarative form The second function

is performed whenever a new decision-making situation occurrs a task-oriented decision structure

is obtained to suit that situation Thsk-oriented knowledge is defined as knowledge that is adapted

for solving a given decision-making situation (Imam amp Michalski 1994 Michalski amp Imam

1994)

The dissertation introduces the system AQJJT-2 for learning task -oriented decision structures from

decision rules or examples Each decision making situation is defined by a set of parameters that

controls the learning process of the AQDT-2 system The extensive experiments on AQJJT-2 show

that decision structures learned by it usually outperform in terms of accuracy and average size of

the decision structures those learned from examples by other well known systems The results

show also that the system does not work very well with noisy data The system is illustrated and

compared using applications of artificial problems such as the three MONKs problems (Thrun

Mitchell amp Cheng 1991) and the East-West 1iain problem (Michie et al 1994) It was also

applied to real world problems of learning decision structures in the areas of construction

engineering (for determining the best wind bracing design for tall buildings) medical

diagnosis(for learning decision rules for recognizing breast cancer) agriculture diagnosis (for

learning classification rules for distinguishing between poisonous and non-poisonous

mushrooms) and political data (for characterizing democratic and republican voting records)

CHAPTER 1 INTRODUCTION

11 Motivation and Overview

Learning and discovery systems should be able not only to generate and store knowledge but also

to use this knowledge for decision-making The main step in the development of systems for

decision-making is the creation of a knowledge structure that characterizes the decision-making

process The form in which knowledge can be easily obtained may however differ from the form

in which it is most readily used for decision-making It is therefore important to identify the form

of knowledge representation that is most appropIiate for learning (eg due to ease of its

modification) and the form that is most convenient for decision making

A simple and effective tool for describing decision processes is a decision structure which is a

directed acyclic graph that specifies an order of tests to be appliedto an object (or a situation) to

arrive at a decision about that object The nodes of the structure are assigned individual tests

(which may correspond to a single attribute a function of attributes or a relation) the branches are

assigned possible test outcomes or ranges of outcomes and the leaves are assigned a specific

decision a set of candidate decisions with corresponding probabilities or an undetermined

decision A decision structure reduces to a familiar decision tree when each node is assigned a

single attribute and has at most one parent when the branches from each node are assigned single

values of that attribute and when leaves are assigned single definite decisions Thus the problem

of generating a decision structure is a generalization of the problem of generating a decision tree

Decision trees are typically generated from a set of examples of decisions The essential

characteristic of any such method is the attribute selection criterion used for choosing attributes to

be assigned to the nodes of the decision tree being built Such criteria include the entropy

reduction the gain and the gain ratio (Quinlan 1979 83 86) the gini index ofdiversity (Breiman

et al 1984) and others (Cestnik amp Bratko 1991 Cestnik amp Karalic 1991 Mingers 1989a)

3

4

Adecision treedecision structure representations can be an effective tool for describing a decision

process as long as all the required tests can be performed and the decision-making situations it

was designed for remain constant (eg in a doctor-patient example the doctor should detennine

that answers for all symptoms appear in the decision tree) Problems arise when these assumptions

do not hold For example in some situations measuring certain attributes may be difficult or costly

(eg in the doctor-patient example a brain or blood test is needed which is very expensive or the

tools needed are not available) In such situations it is desirable to refonnulate the decision

structure so that the inexpensive attributes are evaluated frrst (assigned to the nodes close to the

root) and the expensive attributes are evaluated only if necessary (by assignment to the nodes far

away from the root) If an attribute cannot be measured at all it is useful to either modify the

structure so that it does not contain that attribute or-when this is impossible-to indicate

alternative candidate decisions and their probabilities A restructuring is also desirable if there is a

significant change in the frequency of occurrence of different decisions (eg in the doctor-patient

example the doctor may request a decision structure expressed in a specific set of symptoms

biased to classify one or more diseases or specify a certain order of testing)

Arestructuring ofa decision structure (or a tree) in order to suit new requirements could be quite

difficult This is because a decision structure is a procedural representation that imposes an

evaluation order on the tests In contrast no evaluation order is imposed by a declarative

representation such as a set of decision rules Tests (conditions) of rules can be evaluated in any

order Thus for a given set of rules one can usually build a huge number of logically equivalent

decision structures (trees) which differ in the test ordering Due to the lack of order constraints

a declarative representation (rules) is much easier to modify to adapt to different situations than a

procedural one (a decision structure or a tree) On the other hand to apply decision rules to make a

decision one needs to decide in which order tests are evaluated and thus needs a decision

structure

An attractive solution to these opposite requirements is to acquire and store knowledge in a

declarative form and transform it to a decision structure when it is needed for decision-making

5

This method allows one to create a decision structure that is most appropriate in a given decisionshy

making situation Because the number of decision rules per decision class is usually small (each

rule is a generalization of a set of examples) generating a decision structure from decision rules

can potentially be perfonned much faster than by generation from training examples Thus this

process could be done on line without any delay noticeable to the user Such virtual decision

structures are easy to tailor to any given decision-making situation

This approach allows one to generate a decision structure that avoids or delays evaluating an

attribute that is difficult to measure in some decision-making situation or that fits well a particular

frequency distribution of decision classes In other situations it may be unnecessary to generate

complete decision structure but it may be sufficient to generate only a part of it which concerns

only with decision classes of interest Thus such an approach has many potential advantages

This dissertation presents a new system called AQflf-2 The AWf-2 system generates a taskshy

oriented decision structure (decision structure that is adapted to the given decision-making

situation) from decision rules The decision rules are learned by either rule leaming system AQ15

(Michalski et al 1986) or system AQ17-DCI which has extensive constructive induction

capabilities (Bloedorn et al 1993)

To associate the decision rules with a given decision-making task AQflf-2 provides a set of

features including 1) enabling the system to include in the decision structure nodes corresponding

to new attributes constructed during the process of learning the decision rules 2) controlling the

degree of generalization needed during the development of the decision structure 3) providing four

new criteria for selecting an attribute to be a node in the decision structure that allow the system to

generate many different but equivalent decision structures from the same set of rules 4) generating

unknown nodes in situations when there is insufficient infonnation for generating a complete

decision structure 5) learning decision structures from discriminant rules as well as

characteristic rules and 6) providing the most likely decision when the decision process stops

due to inability to evaluate an attribute associated with an intennediate node

6

To test the methooology of generating decision structures from decision rules an extensive set of

planned experiments have been designed to test different aspects of the approach The experiments

include testing different combinations of parameters for each sub-function of the approach

analyzing the relatioship between decision rules and the decision structure learned from them and

comparing decision trees learned by the AQUf2 system the well known C45 (Quinlan 1993)

system for learning decision trees from examples Different experiments were designed to examine

the new features of the AQUf approach for learning task-oriented decision structures The

experiments were applied to artificial domains as well as real-world domains including MONK-I

MONK-2 and MONK-3 (Thrun Mitchell amp Cheng 1991) East-West trains (Michie et al

1994) Engineering Design-wind bracings (Arciszewski et al 1992) Mushrooms Breast

Cancer (Mangasarian amp WolbeJg 1990) and Congressional bting Records of 1984 The

MONKs problems are concerned with learning classification rules for robot-like figures MONK-l

requires learning a DNF-type description MONK-2 requires learning a non-DNF-type description

(one that cannot be easily described as a DNF rule using the original attributes) MONK-3 concerns

learning a DNF rule from noisy data The East-West trains dataset is a structural domain that

classifies two sets of trains (Eastbound and Westbound) The Engineering Design-wind

bracing data involves learning conditions for applying different types of wind bracing for ta11

buildings The Mushrooms data is concerned with learning classification rules for distinguishing

between poisonous and non-poisonous mushrooms The Breast Cancer data involves learning

concept descriptions for recognizing breast cancer The congressional voting data includes voting

records on different issues AQUf2 outperformed C45 on the average with respect to both the

predictive accuracy and tree size for most problems Apf-2 did not work very well with noisy

data or with problems that have many rules covering very few examples

12 The Problem Statement

There are many limitations and problems accompanied using decision trees for decision-making

CHAPTER 2 RELATED RESEARCH

21 Learning Decision Trees from Decision Diagrams

The AQDT-2 method proposed here is based on the earlier work by Michalski (1978) which

introduced an algorithm for generating decision trees from decision lists The method proposed

several attribute selection criteria These criteria are of increasing power of the main criterion

order cost estimate (the nth order cost estimates n=l 2 ) Michalski also analyzed two

specific criteria MAL and DMAL for selecting the optimal attribute for a node in the tree

based on properties extracted from the decision diagram In order to better explain the method

it is necessary to define some terms These terms will be used later in the dissertation

Definition 2-1 A ~ of a decision class C is a set of rules whose union includes the set of all

examples of class C and does not include any examples of other decision classes

Definition 2-2 A ~ of a decision class C is disjoint if all rules (rules) are pairwise logically

disjoint In other words for any two rules there exists a condition with the same attribute but

with different values in each rule

Definition 2-3 An minimal cover is a disjoint cover which has the smallest number of rules

among all possible covers

Definition 2-4 A diagram for a given cover is a table constructed graphically by representing

in two dimension space of all possible combination of attributes values and locating on the

diagram all the condition parts of the given rules and marking them with the action specified

with each rule

Michalskis algorithm requires construction of minimalmiddot cover The minimal cover should be

consistent and complete The method is based on the fact that if there are n decision classes

any decision tree that correctly classifies the given rules and has n distinct leaves is a minimal

7

8

decision tree (any consistent decision tree should have at least n leaves) Michalski (1978) has

shown that if only one rule is broken by a selected attribute then instead of having one leaf

(which could potentially represent this rule or the decision class in the tree) there will have to

be at least two leaves representing this rule in the fmal decision tree

The attribute selection criterion MAL introduced in (Michalski 1978) prefers attributes that do

not break any rules or break as few as possible An attribute breaks a rule if the attribute can

divide the rule into two or more sub-rules Figure 2-1 shows two examples of two sets of rules

In the fJIst xl and x3 break at least two rules each (xl breaks the rule [x4=2] amp [x1=lv3] amp

[x3=2v3] and the rule [x4=3] amp [x1=1v2] amp [x3=1] and x3 breaks three rules the rule [x4=2]

amp [xl=2] amp [x3=2v3]the rule [x4=l] amp [xl=3] amp [x3=lv3] and the rule [x4=3] amp [xl=4] amp

[x3=2v3]) In the diagram on the right xl is the only attribute that does not break any rule

Figure 2-1 An example to illustrate how attributes break rules

One of the criteria defmed by Michalski is the first degree cost estimate which assigns to each

attribute an integer equal to the number of rules broken by that attribute This criterion is also

called the static cost estimate of an attribute or the criterion of minimizing added leaves

(MAL)

-

C3

9

The MAL criterion (Minimizing Added Leaves) seeks an attribute that minimizes the

estimated number of additional nodes in the decision tree being generated over a hypothetical

minimal decision tree When there is a tie between two attributes the attribute to be selected is

the one which breaks smaller rules (rules that cover fewer examples or more specialized

rules) AQDT-2 uses an approximate version of this criterion (the attribute dominance)

Another criterion introduced by Michalski was the DMAL criterion The DMAL criterion

(Dynamically Minimizing Added Leaves) is based on a principle similar to that of MAL but

is more complex because once an attribute is selected as a node in the tree some rules andor

parts of the broken rules at each branch are merged into one rule The DMAL ensures that the

value of the total cost estimate of an attribute is decreased by a value equal to the number of

merged rules minus one

Example Learn a decision tree from the following decision table

The minimal cover consists of the following rules

Al lt [x2=O] v [x1=0][x2=2] A2 lt [x2=I] v [xl=2][x2=2] A3 lt [xl=I][x2=2]

The evaluations ofthe MAL criterion for these attributes are 2 for xl 0 for x2 5 for x3 and 5

for x4 The attribute xl divides the two rules [x2=O] and [x2=I] so selecting it will add two

leaves to the optimal number of leaves It is clear that the attribute to be selected as a root of

10

the decision tree is x2 Then three branches are attached to the root node and the decision rules

are divided into subsets each corresponding to one branch For x2 = 0 or 1 a leave node is

generated For x2=2 another attribute is selected to be a node in the tree In this case xl has the

minimum MAL value Figure 2-2 shows the decision tree obtained using the MAL criterion

Al A3 A2

Figure 2-2 A decision tree learned from the decision table in Table 2-1

22 Learning Decision Trees from Examples

Decision tree learning is a field that concerned with generating decision tree that classifies a set

of examples according to the decision classes they belong to The essential aspect of any

inductive decision tree method is the attribute selection criterion The attribute selection

criterion measures how good the attributes are for discriminating among the given set of

decision classes The best attribute according to the selection criterion is chosen to be assigned

to a node in the tree The fIrst algorithm for generating decision trees from examples was

proposed by Hunt Marin and Stone (1966) Hunts algorithm uses a divide and conquer

algorithm for building decision trees This algorithm has been subsequently modified by

Quinlan (1979) and applied by many researchers to a variety of learning problems

Attribute selection criteria can be divided into three categories These categories are logicshy

based information-based and statistics-based The logic-based criteria for selecting attributes

use logical relationships between the attributes and the decision classes to determine the best

attribute to be a node in the decision tree such as the MAL criterion minimizing added leaves

(Michalski 1978) which uses conjunction and disjunction operators The information-based

criteria are based on the information theory These criteria measure the information conveyed

11

by dividing the training examples into subsets Examples of such criteria include the

information measure 1M the entropy reduction measure and the gain criteria (Quinlan 1979

83) the gini index of diversity (Breiman et al 1984) Gain-ratio measure (Quinlan 1986) and

others (Clark amp Niblett 1987 Bratko amp Lavrac 1987 Cestnik amp Karalic 1991) The

statistics-based criteria measure the correlation between the decision classes and the attributes

These criteria use statistical distributions for determining whether or not there is a correlation

The attribute with the highest correlation is selected to be a node in the tree Examples of

statistics-based criteria include Chi-square and G-statistic (Sokal amp Rohlf 1981 Han 1984

Mingers 1989a)

Niblett and Bratko (1986) Quinlan (1987) and Bratko and Kononeko (1987) extended the

method of learning decision trees to also handle data with noise (by pruning) Handling noise

extended the process of learning decision trees to include the creation of an initial complete

decision tree tree pruning which is done by removing subtrees with small statistical Validity

and replacing them by leaf nodes (MiDgers 1989b) More recently pruning has also been used

for simplifying decision trees even for problems without noise (Bohanec amp Bratko 1994)

Pruning decision trees improves their simplicity but reduces their predictive accuracy on the

training examples Quinlan (1990) proposed also a method to handle the unknown attributeshy

value problem by exploring probabilities of an example belonging to different classes

The rest of this section includes a brief description of the attribute selection criterion used by

C45 learning system (Quinlan 1993) C45 uses an information-based criterion for selecting

an attribute to be a node in the tree Also the section includes a brief description of the Chishy

square method for attribute selection (Mingers 1989a) The method is a statistics-based

method for selecting an attribute to be a node in the tree

221 Building Decision Trees Using htformation-based Criteria

12

This section presents a description of the inductive decision tree learning system C4S The

C4S learning system is considered to be one of the most stable accurate and fastest program

for learning decision trees from examples

Learning decision trees from examples requires a collection of examples Each example is

represented by a fixed number of attribute-value pairs C45 (Quinlan 1993) is a learning

program that induces classification decision trees from a set of given examples The C4S

learning system is descended from the learning system ID3 (Quinlan 1979) which is based on

Hunts method for constructing decision tree from a set ofcases (Hunt Marin amp Stone 1966)

The C45 system uses an attribute selection criterion called the Gain Ratio This criterion

calculates the gain in classifying information based on the residual information needed to

classify cases in a set of training examples and the information yielded by the test based on the

relative frequencies of the possible outcomes (decision classes) The gain ratio criterion is

based on an earlier criterion used by ID3 called the Gain Criterion The Gain Criterion uses

the frequency of each decision class in the given set of training examples

Once an attribute is chosen to be a node in the tree the system generates as many links as the

number of its values and classifies the set of examples based on these values If all the

examples at a certain node belong to one decision class the system generates a leaf node and

assigns it to that class Otherwise the system searches for another attribute to be a node in the

tree

The Gain Criterion The gain criterion is based on the information theory That is the

information conveyed by a message depends on its probability and can be measured in bits as

minus the logarithm (base 2) of that probability To explain the gain criterion suppose that for

a given problem Xu bullbull X are given attributes and Cu c are decision classes Suppose S is

any set of cases and T is the initial set of training cases The frequency of class C1 in the set S

is the number of examples in S that belong to class C i bull

13

freq(Cit S) = Number of examples in S belong to CI (2-1)

Suppose that lSI is the total number of examples in S the probability that an example selected

at random from S belongs to class Ci is freq(CIbullS) IISI

The information conveyed by the message that a selected example belongs to a given decision

class Cit is determined by -log1 (freq(Ch S) IISI) bits

The expected information from such a message stating class membership is given by

info(S) = -k

L (freq(C S) I lSI) log1 (freq(C S) IISI) bits (2-2)I

info(S) is known also as the entrQpy of the set S When S is the initial set of training examples

info(T) determines the average amount of information needed to identify the class of an

example in T

Suppose that we selected an attribute X to be the root of the tree and suppose that X has k

possible values The training set T will be divided into k subsets each corresponding to one of

Xs values The expected information of selecting X to partition the training set T infoz(T) can

be found as the sum over all subsets of multiplying the information conveyed by each subset

by its probability

k

infoz(T) =L (ITII 1m) infO(TIraquo (2-3) I

The information gained by partitioning the training examples T into subset using the attribute

X is given by

gain(X) =inlo(T) - inoIT) (2-4)

The attribute to be selected is the attribute with maximum gain value

The Gain Ratio Criterion This criterion indicates the proportion of information generated by

the split that appears helpful for classification Quinlan (1993) pointed out that the gain

criterion has a serious deficiency Basically it is strongly biased toward attributes with many

outcomes (values) For example for any data that contains attributes such as social security

14

number the gain criterion will select that attribute to be the root of the decision tree However

selecting such attributes increases the size of the decision tree Quinlan provided a solution to

this problem by introducing the gain ratio criterion which takes the ratio of the information that

is gained by partitioning the initial set of examples T by the attribute X to the potential

information generated by dividing T into n subsets

Following similar steps to obtain the information conveyed by dividing T into n subsets The

expected information generated by dividing T into n subsets and by analogy to equation 2-2 is

determined by

split info(T) = - Lt

(ITII lTI) 10gl (ITII m ) (2-5) I

The gain ratio is given by

gain ratio(X) = gain(X) I split info(X) (2-6)

and it expresses the proportion of information generated by the split that is useful for

classification

Example Consider the following example presented by Quinlan (1993) Table 2-2 shows the

set of training examples

First determine the amount of information gained by selecting the attribute outlook to be a

root of the decision tree This attribute divides the training examples into three subsets

sunny-with five examples two of which belong to the class Play overcast-with four

examples all of which belong to the class Play and rain-with five examples three of

which belong to the class Play To determine info(11 the average information needed to

identify the class of an example in T there are 14 training examples and two decision classes

Nine of these examples belong to the class Play and five belong to the class Dont Play

Info(11 = - 914 10gl (914) - 514 10gl (514) = 094 bits

When using outlook to divide the training examples the information becomes

15

infO(T) = 514 -25 log2 (25) - 35 10g2 (35)

+ 414 -44 10g2 (44) - 04 log2 (04)

+ 514 -35 logl (35) - 25 logl (25) = 0694 bits

By substituting in equation 2-4 the gain of information results from using the attribute

outlook to split the training examples equal to 0246 The gain information for windy is

0048

Figure 2-3 shows a decision tree learned for this problem using the gain criterion The split

information for outlook is determined as follows

split info(T) = - 514 logl (514) - 414 log2 (414) - 514 log2 (514) = 1577 bits

The gain ratio for outlook = 02461577 =0156

16

Figure 2 -3 A decision tree learned using the gain criterion for selecting attributes

The C45 system handles discrete values as well as continuous values To handle an attribute

with continuous values C45 uses a threshold to transform the continuous domain into two

intervals In other words for each continuous attribute C45 generates two branches one

where the values of that attribute is greater than the determined threshold and the other if the

value is less than or equal to the threshold

Tree pruning in C45 is a process of replacing subtrees with small classification validity by

leaves The C45 system uses Laplace ratio for determining the error rate of different subtrees

This ratio is defined as (e+l) (n+2) where n is the number of the training examples and e is

the number of misclassified examples at a given leaf

222 Building Decision Trees Using Statistics-based Criteria

The Chi-square method for selecting attributes was used by Hart (1984) and Mingers (1989a)

in building decisionmiddot trees The method uses Chi-square statistics to measure the association

between two attributes When building decision trees the method is implemented such that it

determines the association between each attribute and the decision classes The attribute to be

selected is the one with greatest value

To determine the Chi-square value for an attribute consider aij is the number of examples in

class number i where the attribute A takes value numberj In other words aij is the frequency

of the combination of decision class number i and the attribute value number j The Chi-square

value for attribute A is given by

n m

Chi-square (A) =L L [ (aij - Eijl2 Eij ] (2shy

i=1 j=1

where n is the number of decision classes m is the number ofvalues of a given attribute Also

17

Eij = (TCi TVj ) T (2shy

8)

where T Ci and T Vj are the total number of examples belong to the decision class a and the total

number of examples where the attribute A takes value vj respectively T is the total number of

examples

Consider Quinlans example in Table 2-2 Table 2-3 shows the frequency of different

combination of values between the decision class and both the outlook and the Windy

attributes Table 2-4 shows the expected values TCi and T Vj of the frequencies in Table 2-3

of different attributes values for different decision classes

To determine the association value between the decision classes and both the attribute Windy

and the attribute Outlook the observed Chi-square values are

Chi-square (Windy Class) =[(3-39)239] + [(3-21)221] + [(6-51)232] + [(2-29)232]

=021 + 039 + 025 + 025 = 11

Chi-square (Outlook Class) = [(2-32)232] + [(4-26)226] + [(3-32)232] + [(3-18118]

+ [(0-14)214] + [(2-18)2 18] =045 + 075 + 02 + 08 + 14 + 002 = 362

18

Applying the same method to the other attributes the results will favor the attribute Outlook

Once that attribute is selected to be a node in the tree the remaining set of examples are divided

into subsets and the same process is repeated on each subset

Table 2-5 shows a summary of these criteria and their basic evaluation function

Table 2-5 Attribute selection criteria and their basic evaluation measure

Info Measure (IM) Gain Gshystatistics and Gain Ratio Chi-square

II

Entropy(S) = - L (freq(~ 5) 1151) logz (freq(CIt 5) 1151 ) I

G-Statistics =2N 1M (N-No of examples) n m

Chi-square (A B) = L L [(aij - ~j)21Eij ] i=l

223 Analysis of Attribute Selection Criteria

This subsection introduces briefly the analysis of different selection criteria which was done by

Mingers (1989a) Mingers compared six attribute selection criteria that are used in decision

tree programs These criteria are the Information Measure (IM) Chi-square G statistics Gin

index of diversity Marshal correction and Gain RaJio The overall results show that the Gain

Ratio criterion had the strongest xesults

In the flISt experiment Minger tested the six criteria on a problem with ambiguous examples

(ie examples may belong to more than one decision class) to observe how the selected criteria

evaluate the given attributes The problem has two decision classes and two attributes X and

Y It was assumed that attribute X is better for classifying the examples than attribute Y The

training examples were unevenly spread between the two values of X Attribute Y has three

values and the examples were spread randomly among them Table 2-6 (a and b) shows the

contingency tables for both attributes Table 2-7 shows a summary of the goodness of split

provided by the six criteria Mingers noted that the measures that are not based on information

19

theory give radiation (attribute X here) less weighL This may be because the zero in the fIrst

row of radiation has a greater influence in the log calculation In the case of using the Chishy

square criterion the value zero adds the maximum association between any two attributes

because the Chi-square value of a zero cell is the expected value of this cell

b)

Now let us demonstrate results from another experiment done by Mingers In this experiment

Mingers used four different data sets to generate decision trees for eleven different criteria In

the final results he compared the total number of nodes and the total error rate provided by

each criterion over all given problems Table 2-8 shows the final results for five selected

criteria only

20

U lgt ~middot results for comparing the total accuracy and size of decision trees different attribute selection criteria from four VVLU

This experiment was performed on four real world data sets These data are concerned with

profiles of BA Business Studies degree students recurrence of breast cancer classifying types

of Iris and recognizing LCD display digits The data was divided randomly 70 for training

and 30 for testing For more details see Mingers (1989a)

23 Learning Decision Structures

Considering the proposed defInition of decision structures given above two related lines of

research are described in this section Gaines (1994) and Kohavi (1994) proposed two

approaches for generating decision structures that share some of the earlier ideas by Imam and

Michalski (1993b)

In the first approach Brian Gaines introduces a method for transforming decision rules or

decision trees into exceptional decision structures The method builds an Exception Directed

Acyclic Graph (EDAG) from a set of rules or decision trees The method starts by assigning

either a rule say RO or a conclusion say CO or both to the root node If it assigns a conclusion

to the root node it places it on a temporary conclusion list Then it generates a new child node

and add to it a new rule say Rl The method evaluates the new rule Rl if it satisfies the

conclusion CO in its parent node it builds a new child and repeats the processes Otherwise if

the rule Rl does not satisfy the conclusion CO it changes the temporary memory with the new

conclusion Cl which is satisfied by the rule RO The same process is repeated until all rules

that have common conditions with the rule at the root are evaluated The method then creates a

21

new child node from the root and repeat the process until all rules are evaluated In the decision

structure nodes containing rules only represent common conditions to all of its children

The main disadvantages of this approach is that it requires discriminant rules to build such a

decision structure Also such a structure is more complex than the traditional decision trees

that are used for decision-making

Figure 2-4 shows an example of set of rules and their equivalent exceptional decision structure

The decision structure can be read as follows It is Safe except if xl=1 amp x2=1 amp x3=1 and

either x4=3 or x5=I then it is Lost except if x6=1 it is Safe except if x7=1 it is Lost

The second approach introduced by Ronny Kohavi learns decision structures from examples

using a bottom-up algorithm The method uses a greedy Hill-climbing algorithm for inducing

Oblivious read-Once Decision Graphs (HOODO) A read-once decision graph is defined as a

decision graph where each attribute occurs at most once along any computational path In other

words for each path from the root to the leaves of the decision structure attributes may occur

as a node once at most However there may be more than one node with the same attribute in

the decision graph Kohavi also defined a leveled decision graph in which the nodes are

partitioned into a sequence of pairwise disjoint sets such that outgoing edges from each level

terminate at the next level An oblivious decision graph is a decision graph where all nodes at

a given level are labeled by the same attribute

22

Safe lt [xl=2] Safe lt [x2=2] Safe lt [x3=2] Safe lt [x4=1] amp [x5=2] Safe lt [x4=1] amp [x5=3] Safe lt [x6=l] amp [x7=2] Safe lt [x6=1] amp [x7=3] Safe lt [x4=2] amp [x5=2] Safe lt [x4=2] amp [x5=3]

Lost lt [x6=2] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x6=3] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x5=1] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=2] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=3] amp [x3=1] amp [x2=I] amp [xl=l]

The algorithm starts by generating a leaf node for each decision class Then it applies a

nondeterministic method to select an attribute to build new level of the decision structure This

attribute is removed from the data and the data is divided into subsets each corresponding to a

complex combination of that attributes values For each subset the process is repeated until all

the examples of a given subset belong to one decision class For example if the selected

attribute say A has two values (0 and 1) and there are two decision classes (CO and CI) The

data is divided into two subsets the fIrst subset contains examples where A takes value 0 and

belong to class CO or takes value 1 and belong to class Cl The second subset of examples is

the set where A takes value 0 and belong to class CI or takes value 1 and belong to class CO

The number of nodes of the fIrst level (after the leave nodes) is expected to be less than or

equal to ka where k is the number of decision classes and n is the number of values of the

selected attribute and it can increase exponentially before that number is reduced

exponentially to one

It is easy for the reader to fIgure out some major disadvantages of such approach including

The average size of such decision structures is estimated to be very large especially when there

23

is no similarity (ie strong patterns) or logical relationship in the data The time used to learn

such decision structure is relatively very high comparing to systems for learning decision trees

from examples and fmally it could be better to search for attribute which reduces the number

of generated subsets of the data instead of nondeterministically selecting an attribute to build a

new level of the decision structure Kohavi provided a comparison between the C45 system

and his system that can be found in (Kohavi 1994) Later Kohavi and Li (1995) used a

deterministic method for attribute selection which minimizes the width of the penultimate level

of the graph

Table 2-9 shows a comparison between the proposed approach and those two approaches The

EDAG and HOODG systems are unreleased prototype systems

or Decision structures are easy to Decision structures are Decision structures are easy to understand difficult to read understand

CHAPTER 3 DESCRIPTION OF THE APPROACH

31 General Methodology

In the proposed approach the function of learning or discovery is separated from the function of

using the discovered knowledge for decision-making The frrst function is performed by an

inductive learning program that searches for knowledge relevant to a given class of decisions

and stores the learned knowledge in the form of decision rules The second function is performed

when there is a need for assigning a decision to new data points in the database (eg bull a

classification decision) by a program that transforms obtained knowledge into a decision

structure optimized according to the given decision-making situation

The Learning Task GivenA set of training examples describing the concept to be learned Learning goal which specifies the decision classes to be learned from the training examples A background knowledge to control the learning process Determine A concept description in a declarative form of knowledge (decision rules) that satisfies the learning goal

The Decision-making Task GivenA set of decision rules in a conjunctive form A description of the new decision-making situation (eg attributes costs and order preference importance or frequency of decision classes etc) One or more examples that needed to be tested under the given decisionshymaking situation A set of parameters to control the learning process Determine A decision structure that suits the given decision-making situation

The decision rules used here are learned by either the AQ1S (Michalski et al 1986) or AQ17

(Bloedorn et aI 1993) learning systems The reasons for selecting decision rules as a form of

23

24

declarative knowledge are that they do not impose any order on the evaluation of the attributes

and due to the lack of order constraints decision rules can be evaluated in many different

ways which increases the flexibility of adapting them to the different tasks of decision-making

(Bergadano et al 1990) Since the number of rules learned per class is typically much smaller

than the number of examples per class generating a decision structure from decision rules can

potentially be done on line

Such virtual decision structures can be tailored to any given decision-making situation The

needed decision rules have to be generated only once and then they can be used many times for

generating decision structures according to changing requirements of decision-making tasks The

method uses the AQDT-2 system (Imam amp Michalski 1994) for learning decision structures

from decision rules Decision structures represent a procedural form of knowledge which makes

them easy to implement but also harder to change Consequently decision structures can be quite

effective and useful as long as they are used in decision-making situations for which they are

optimized and the attributes specified by the decision structure can be measured without much

cost Figure 3-1 shows an architecture of the proposed methodology

Data

Decision -

- Learning knowledge from database The decision making process

Figure 3 -1 Architecture of the AQDT approach

25

It is assumed that the database is not static but is regularly updated A decision-making problem

arises when there is a case or a set of cases to which the system has to assign a decision based on

the knowledge discovered Each decision-making situation is defmed by a set of attribute-values

Some attribute-values may be missing or unknown A new decision structure is obtained such

that it suits the given decision-making problem The learned decision structure associates the

new set of cases with the proper decisions

32 A Brief Description of the AQIS and AQ17 Rule Learning Programs

The decision rules are generated from examples by an AQ-type inductive learning system

specifically by AQ15 (Michalski et albull 1986) or AQI7-DCI (Bloedorn at al 1993) The rest of

this subsection includes a brief description of the inductive learning systems AQ15 and AQI7

AQ15 learns decision rules for a given set of decision classes from examples of decisions using

the STAR methodology (Michalski 1983) The simplest algorithm based on this methodology

called AQ starts with a seed example of a given decision class and generates a set of the

most general conjunctive descriptions of the seed (alternative decision rules for the seed

example) Such a set is called the star of the seed example The algorithm selects from the star

a description that optimizes a criterion reflecting the needs of the problem domain If the

criterion is not defined the program uses a default criterion that selects the description that

covers the largest number of positive examples (to minimize the total number of rules needed)

and with the second priority that involves the smallest number of attributes (to minimize the

number of attributes needed for arriving at a decision)

If the selected description does not cover all examples of a given decision class a new seed is

selected from uncovered examples and the process continues until a complete class description

is generated The algorithm can work with few examples or with many examples and can

optimize the description according to a variety of easily-modifiable hypothesis quality criteria

26

The learned descriptions are represented in the form of a set of decision rules expressed in an

attributional logic calculus called variable-valued logic 1 or VLl (Michalski 1973) A

distinctive feature of this representation is that it employs in addition to standard logic operators

the internal disjunction operator (a disjunction of values of the same attribute in a condition) and

the range operator (to express conditions involving a range of discrete or continuous values)

These operators help to simplify rules involving multi valued discrete attributes the second

operator is also used for creating logical expressions involving continuous attributes

AQ15 can generate decision rules that represent either characteristic or discriminant concept

descriptions depending on the settings of its parameters (Michalski 1983) A characteristic

description states properties that are true for all objects in the concept The simplest

characteristic concept description is in the form of a single conjunctive rule (in general it can be

a set of such rules) The most desirable is the maximal characteristic description that is a rule

with the longest condition part ie stating as many common properties of objects of the given

class as can be determined A discriminant description states properties that discriminate a given

concept from a fixed set of other concepts The most desirable is the minimal discriminant

descriptions that is a rule with the slwnest condition part For example to distinguish a given

set of tables from a set of chairs one may only need to indicate that tables have large flat top

A characteristic description of the tables would include also properties such as have four legs

have no back have four corners etc Discriminant descriptions are usually much shorter than

characteristic descriptions

Another option provided in AQ15 controls the relationship among the generated descriptions

(rulesets or covers) of different decision classes In the IC (Intersecting Covers) mode

rulesets of different classes may logically intersect over areas of the description space in which

there are no training examples In the DC (Disjoint Covers) mode descriptions of different

classes are logically disjoint The DC mode descriptions are usually more complex both in the

number of rules and the number of conditions There is also a DL mode (a Decision List mode

27

also called VL modeM-for variable-valued logic mode) in which the program generates rule sets

that are linearly ordered To assign a decision to an example using such rulesets the program

evaluates them in order If ruleset is satisfied by the example then the decision is made

otherwise the program proceeds to the evaluation of the ruleset +1 In IC and DC modes

rule sets can be evaluated in any order

Alternatively the system can use rules from the AQ17-DCI program for Data-driven

Constructive Induction AQ17-DCI differs from AQ15 mainly in that it contains a module for

generating additional attributes These attributes are various logical or mathematical

combinations the original attributes The program generates a large number of potential new

attributes and seleets from them those most promising based on an attribute quality criterion

To illustrate the format of rules generated by AQ15 (or AQ17-DCI) an exemplary ruleset is

shown in Figure 3-2 The ruleset (that can be re-represented as a disjunctive normal form

expression) describes a voting record of Democratic Representatives in the US Congress

Each rule is a conjunction of elementary conditions Each condition expresses a simple relational

statement For example the condition [State = northeast v northwest] states that the attribute

State (of the Representative) should take the value northeast or northwest to satisfy the

condition

Rl [Gas_concban = yes] amp [Soc_see_cut = no v not registered] R2 [Draft = Yes v not registered]amp[AlaslaLparks = yes v not registered] amp

[Food_stamp_cap = no] amp [State = northeast v northwest] R3 [Chrysler =yes v not registered] amp [Income =low] R4 [Education =yes] amp [Occupation = yes]

Figure 3-2 A ruleset generated by AQ15 for the concept Voting pattern of Democratic Representatives

The above rules were generated from examples of the voting records For illustration below is

an example of a voting record by a Democratic representative

Draft registration=no Ban of aid to Nicaragua=no Cut expenditure on mx missiles=yes Federal subsidy to nuclear power stations=yes Subsidy to national parks

28

in Alaska=yes Fair housing bill=yes Limit on Pac contributions=yes Limit onood stamp program=no Federal help to education=no StateFrom=north east State Population=large Occupation =unknown Cut in social security spending=no Federal help to Chrysler corp=not registered

By expressing elementary statements in the example as conditions and linking conditions by

conjunction the examples can be re-expressed as decision rules Thus decision rules and

examples formally differ only in the degree of generality

33 Generating Decision Structurestrees from Decision Rules

This section describes the AQDT-1 system for learning decision structures (trees) from decision

rules (Imam amp Michalski 1993ab) Also a description of the AQDT-2 method for learning taskshy

oriented decision structure from decision rules is included and fmally the methodology is

illustrated by two examples

Methods for learning decision trees from examples have been very popular in machine learning

due to their simplicity Decision trees built this way can be quite efficient as long as they are

used in decision-making situations for which they are optimized and these situations remain

relatively stable Problems arise when these situations significantly change and the assumptions

under which the tree was built do not hold anymore For example in some situations it may be

difficult to determine the value of the attribute assigned to some node One would like to avoid

measuring this attribute and still be able to classify the example if this is potentially possible

(Quinlan 1990) If the cost of measuring various attributes changes it is desirable to restructure

the tree so that the inexpensive attributes are evaluated f11st A tree restructuring is also

desirable if there is a significant change in the frequency of occurrence of examples from

different classes A restructuring of a decision tree to suit the above requirements is however

difficult to do The reason for this is that decision trees are a form of decision structure

representation and imposes constraints on the evaluation order of the attributes that are not

logically necessary

29

One problem in developing a method for generating decision structures from decision rules is to

design an attribute selection criterion that is based on the properties of the rules rather than of

the training examples A decision rule normally describes a number of possible examples Only

some of them are examples that have actually been observed ie training examples An attribute

selection criterion is needed to analyze the role of each attribute in the rules It cannot be based

on counting the numbers of training examples covered by each attribute-value and the

frequency of decision classes in the training examples as is done in learning decision trees from

examples because the training examples are assumed to be unavailable

Another problem in learning decision trees from decision rules stems from the fact that decision

rules constitute a more powerful knowledge representation than decision trees They can directly

represent a description in an arbitrary disjunctive normal form while decision trees can represent

directly only descriptions in the disjoint disjunctive normal form In such descriptions all

conjunctions are mutually logically disjoint Therefore when transforming a set of arbitrary

decision rules into a decision tree one faces an additional problem of handling logically

intersecting rules

The solution to both problems (attribute selection and logically intersected rules) in the AQDT-2

system is based on the earlier work by Michalski (1978) which introduced a general method for

generating decision trees from decision rules The method aimed at producing decision trees

with the minimum number of nodes or the minimum cost (where the cost was defined as the

total cost of classifying unknown examples given the cost of measuring individual attributes and

the expected probability distribution of examples of different decision classes) More

explanations are provided in the following section

331 Tbe AQDT-2attribute selection metbod

This section describes the AQDT -2 method for building a decision structure from decision rules

The method for building a single-parent decision structure is similar to that used in standard

30

methods of building a decision tree from examples The major difference is that it assigns tests

(attributes) to the nodes using criteria based on the properties of the decision rules (this includes

statistics about the examples covered by each rule in the case of learning rules from examples)

rather than statistics characterizing the frequency of training examples per decision classes per

attribute-values or per conjunctions of both Other differences are that the branches may be

assigned an internal disjunction of values (not only a single value as in a typical decision tree)

and leaves may be assigned a set of alternative decisions with probabilities Also the tests can be

attributes or names standing for logical or mathematical expressions that involve several

attributes or variables In the following we use the terms test and attribute interchangeably

(to distinguish between an attribute and a name standing for an expression the latter is called a

constructed attribute)

At each step the method chooses the test from an available set of tests that has the highest utility

(see below) for the given set of decision rules This test is assigned to the node The branches

stemming from this node are assigned test values or disjoint groups of values (in the form of

logical disjunction if such occur in the rules subsumed groups of values are removed) Each

branch is associated with a reduced set of rules determined by removing conditions in which the

selected attribute assumes value(s) assigned to this branch If all rules in the reduced ruleset

indicate the same decision class a leaf node is created and assigned this decision class The

process continues until all nodes are leaf nodes If it is not possible to reduce further the rule set

because some attribute is declared as unavailable (infmite cost) then the leaf is assigned a set of

candidate decisions with associated probabilities (see Sec 42)

The test (attribute) utility is a combination of one or more of the following elementary criteria 1)

cost which indicates the cost of using each attribute for making decision 2) disjointness which

captures the effectiveness of the test in discriminating among decision rules for different decision

classes 3) importance which determines the importance of a test in the rules 4) value

31

distribution which characterizes the distribution of the test importance over its of values and 5)

dominance which measures the test presence in the rules These criteria are dermed below

~ The cost of a test expresses the effort or cost needed to measure or apply the test

Djsjointness The disjointness of a test is defined as the sum of the class disjointness-the

disjointness of the test for each decision class Suppose decision classes are Cl C2 bull Cm and

decision rule sets for these classes have been determined Given test A let V 1 V 2 bull V m denote

sets of the values (outcomes) of A that are present in rule sets for classes C 1 C2bullbull Cm

respectively If a ruleset for some class say Ct contains a rule that does not involve test A than

Vt is the set of all possible values of A (the domain of A)

Definition 3 -1 The degree ofclass disjointness D(A Ci ) of test A for the ruleset ofclass Ci is

the sum of the degrees of disjointness D(A Cit Cj) between the ruleset for Ci and rulesets for

Cj j=l 2 m j J L The degree of disjointness between the ruleset for Ci and the ruleset for Cj is

dermed by

if Vi Vj ifVj gtVj (3-1) ifVinVj -tljgt amp -tVi amp -tVj ifVinVj=1jgt

where cjgt denotes the empty set Note that exchanging the second and third conditions of equation

(3-1) may seem to be an improved criteria However it does not clearly distinguish between both

cases (Le for both situations the disjointness will be similar) The current equation is better

because it gives higher scores to attributes that classify different subsets of the two decision

classes than attributes that classify only a subset of one decision class

Definition 3-2 The disjointness of the test A for evaluating a given set of decision rules is the

sum of the degrees of class disjointness of each decision class

32

m m Disjointness(A) = L D(A cn where D(A cn = L D(A Ci Cj) (3-2)

i=l i=l i~j

The disjointness of a test ranges from 0 when the test values in rulesets of different classes are

all the same to 3m(m-l) when every rule set of a given class contains a different set of the test

values If two tests have the same disjointness value the attribute to be selected is the one with

less number of values

Definition The A vera~ Number of Tests (ANT) required to make decision from a decision tree

is dermed as the average number of tests (attributes) to be examined from the root of the tree to

any leaf node in order to reach a decision

Definition A decision structure is a one-node-per-level decision structure if at each level there is

only one node and zero or more leaves

Such decision structure can be generated by combining together all branches that associated with

each one a set of decision rules belongs to more than one decision class

Theorem 1 Consider learning one-node-level decision structure for a database with two

decision classes The disjointness criterion ranks first attributes that add minimum number of

tests to the decision tree

Proof Consider all possible distributions of an attributes values within two decision classes Ci

and Cj There are three cases 1) subset (same as super set) 2) non-empty intersection but not

subset 3) no intersection Figure 3-3 shows all possible distributions (the case of having the

same set of values in both classes is trivial one) Consider that branches leading to one subset

with the same decision class are combined into one branch In the first case there will be two

branches only The first leads to a leaf node and the other leads to an intermediate node where

another attribute to be selected The minimum ANT in this case is 53 In the second case three

branches should be created Two branches leads to a leaf node where all values at each branch

belong to only one and different decision class The third branch leads to an intermediate node

33

where another attribute should be selected furtur classifies the decision classes The minimum

ANT in this case is 64 In the third case only two branches will be generated where each leads

to leaf node with different decision class In this case the minimum ANT is 1

Figure 3-4 shows decision trees equivalent to selecting the corresponding attribute Note that in

case of having more than one attribute-value at branches lead to leaves belonging to one decision

class they will be combined into one branch in the decision structure The symbol 1 means

that an attribute is needed to classify the two decision classes In such cases there will be at least

two additional paths

D(A Ci) = 0 D(A CJ) = 1 D(A Ci) =2 D(A Cj) = 2 D(A Ci) =3 D(A Cj) =3

Figure 3-3 Yen Diagrams of Possible combinations of attribute values in two decision classes

The average number of tests required for making a decision in each possible case is determined

in Figure 3-4 It is clear that in the case of two decision classes the disjointness criterion ranks

highly attributes that reduces the average number of tests required for decision-making The

theorem can be proved in the general case

i ~ i Cj C1middot bull Ci Cj

ANT=32middotANT=53 ANT=l

1 means at least one attribute is needed to complete the decision tree Figure 3-4 Decision trees corresponding the Yen diagrams inFigure 3-3

34

Theorem 2 The attribute with the highest non~zero disjointness is the best attribute to classify

the decision classes

Proof Suppose that the number of decision classes is n Assume also that there are two attributes

A and B where D(A) lt D(B) Since the disjointness criterion considers the mutual relationship

between any two decision classes this means that there are more decision classes where D(A

Ci) lt D(B Ci) than those where D(A Ci) gt D(B Ci) (ignoring classes with equal disjointness

for bCth attributes) Hence there are more decision classes where D(A Ci Cj) lt D(B Ci Cj)

than those where D(A Ci Cj) gt D(B Cit Cj) Let us frrst prove that if D(A Ci Cj) lt D(B Cit

Cj) then B classifies the decision classes better than A

For each pair of decision classes Ci and Cj the possible values of the disjointness of any attribute

are 0 14 or 6 For all positive values ofD(B)= 14 or 6 it is clear that attribute B should have

smaller ANT than attribute A which has lower disjointness This means that if D(A Cit Cj) lt

D(B Cit Cj) then attribute B can classify both decision classes better than attribute A

Similarly B is better for classifying more pairs of decision classes than A This implies that B is

a better classifier than A

Importance The second elementary criterion the importance of a test is based on the

importance score (IS) introduced in (Imam Michalski amp Kerschberg 1993) In the obtained

rules each test is assigned a score that represents the total number of training examples that

are covered by the rules involving this test Decision rules learned by an AQ learning program

are accompanied with information on their strength Rule strength is characterized by t-weight

and u~weight The t-weight (total-weight) of a rule for some class is the number of examples of

that class covered by the rule The importance score of a test is the aggregation of the totalshy

weights of all rules that contain that test in their condition part Given a set of decision rules for

m decision classes Cl Cm and involving n tests AlbullAn and the number of rules associated

with class Ci is denoted by tlrit The importance score is defmed as follow

35

Definition 3 -3 The importance score IS(Aj) of the test A j is detennined by

m IS(Aj)= L IS(Aj Ci) (3-31)

i=l

where ri IS(Aj Ci) = L Rik(Aj) (3-32)

k=l

and Rik the weight of a test Aj in the rule Rk of class Ci is given by

t - weight if A j belongs to rule Rik Rik (Amiddot) = (3-4)J 0 otherwise

where i=I n ik=I ri j=I m

The importance score method has been separately compared as a feature selection method with

a genetic algorithm-based method (Imam amp Vafaie 1994) The importance score method

produced an equal or higher accuracy on three real-world problems than those reported by the

GA method while selecting fewer attributes In addition the IS method was significantly faster

than the GA

yalue distribution The third elementary criterion value distribution concerns the number of

legal values of tests Given two tests with equal importance scores this criterion prefers the test

with the smaller number of legal values Experiments have shown that this criterion is especially

useful when using discriminant decision rules

Definition 34 A value distribution VD(Aj) of a test A j is defined by

VD(Aj) = IS(Aj) I Vj (3-5)

where v is the number of legal values of Aj

Dominance The fourth elementary criterion dominance prefers tests that appear in large

numbers of rules as this indicates their high relevance for discriminating among the rulesets of

given decision classes Since some conditions in the rules have values linked by internal

disjunction counting such rules directly would not reflect properly their relevance Therefore

36

for computing the dominance the rules are counted as if they were converted to rules that do not

have internal disjunction Such a conversion is done by multiplying out the condition parts of

the rules containing internal disjunction For example the condition part [x3=1 v 3]amp[x4=1] is

multiplied out to two rules with condition parts [x3=1]amp[x4=1] and [x3=3]amp[x4=I]

The above criteria are combined into one general test measure using the lexicographic

evaluation functional with tolerances (LEF) (Michalski 1973) LEF is a list of some or all of

the above elementary criteria each associated with a Ittolerance threshold It in percentage The

criteria are applied to tests in the order defined by LEF A test passes to the next criterion only if

it scores on the previous criterion within the range defined by the tolerance (from the top value)

The default LEF is

ltCost d Disjointness t2 Importance t3 Value distr t4 Dominance 0gt (3-6)

where d a t3 t4 and t5 are tolerance thresholds (in percentages) their default values are O

The default value of the cost of each test is 1

The above LEF ranks attributes this way First attributes are evaluated on the basis of their cost

If two or more attributes have the same cost they ranked by their degree of disjointness If two

or more attributes still share the same top score or their scores differ less than the assumed

tolerance threshold t2 the method evaluates these attributes using the second (Importance)

criterion If again two or more attributes share the same top score or their scores differ less than

the tolerance threshold t3 then the third criterion normalized IS is used then similarly the

fourth criterion (dominance) If there is still a tie the method selects the best attribute

randomly

If there is a non-uniform frequency distribution of examples of different classes then the

selection criterion uses a modified definition of the disjointness Namely the previously defmed

37

disjointness for each class is multiplied by the frequency of the class occurrence The class

occurrence is the expected number of future examples that are to be classified to a given class

m

Disjointness(A) = L D(A CO Frq(Ci) (3-7) i=l

where Frq(Cn is the expected frequency of examples of class Cit and it is assumed to be given

by the user The attribute ranking criteria in this case is defined by the LEF

ltCost d Disjointness t2 Importance 13 Normalized-IS 14 Dominance 15gt (3-8)

where the Cost denotes the evaluation cost of an attribute and is to be minimized while other

elementary criteria are treated the same way as in the default version

332 The AQDT-2 algorithm

The AQDT-2 algorithm constructs a decision structure from decision rules by recursively

selecting at each step the best test according to the ranking criteria described above and

assigning it to the new node The process stops when the algorithm creates terminal branches that

are assigned decision classes To facilitate such a process the system creates a special data

structure for each concept description (ruleset) This structure has fields such as the number of

rules the number of decision classes and the number of attributes present in the rules A set of

pointers connect this data structure to set of data structures each represents one decision class

The decision class structure contains fields with information on the number of rules belong to

that class the frequency of the decision class etc It is also connected to a set of data structures

representing the decision rules within each decision class The system creates independently a set

of data structures each is corresponding to one attribute Each attribute description contains the

attributes name domain type the number of legal values a list of the values the number of

rules that contain that attribute and values of that attribute for each rule The attributes are

arranged in an array in a lexicographic order flISt in the descending order of the number of rules

38

that contain that attribute and second in the ascending order of the number of the attributes

legal values

The system can work in two modes In the standard mode the system generates standard

decision trees in which each branch has a specific attribute-value assigned In the compact

mode the system builds a decision structure that may contain

A) or branches Le branches assigned an internal disjunction of attribute-values whenever it

leads to simpler structures For example if a node assigned attribute A has a branch marked by

values 1 v 2 then the control passes along this branch whenever A takes value 1 or 2 The

program creates or branches on the basis of the analysis of the value sets Vi while computing

the degree of attribute disjointness

B) nodes that are assigned derived attributes that is attributes that are certain logical or

mathematical combinations of the original attributes To produce decision structures with

derived attributes the input decision rules are generated by AQ17 (rather than AQI5) The

AQ17 rules may contain conditions involving attributes constructed by the program rather than

those originally given

To generate decision structures from rules the AQDT-2 method prefers either characteristic or

discriminant disjoint rule descriptions (given by an expert or learned by a system) Disjoint rules

are more suitable for building decision structures Assume that the description of each class is in

the form of a ruleset and that this set is the initial ruleset context The AQDT algorithm is

The AQDT2 Algorithm

Step 1 Evaluate each attribute occurring in the ruleset context using the LEF attribute ranking

measure Select the highest ranked attribute Let A represents this highest-ranked

attribute

Step 2 Create a node of the tree (initially the root afterwards a node attached to a branch) and

assign to it the attribute A In standard mode create as many branches from the node as

39

there are legal values of the attribute A and assign these values to the branches In

compact mode (decision structures) create as many branches as there are disjoint value

sets of this attribute in the decision rules and assign these sets to the branches

Step 3 For each branch associate with it a group of rules from the ruleset context that contain a

condition satisfied by the value(s) assigned to this branch For example if a branch is

assigned values i of attribute A then associate with it all rules containing condition [A=

i v ] If a branch is assigned values i v j then associate with it all rules containing

condition [A= i v j v ] Remove from the rules these conditions If there are rules in

the ruleset context that do not contain attribute A add these rules to all rule groups

associated with the branches stemming from the node assigned attribute A (This step is

justified by the consensus law [x=1] == [x=1] amp [y=a] v [x=1] amp [y=b] assuming

that a and b are the only legal values of y) All rules associated with the given branch

constitute a ruleset context for this branch

Step 4 If all the rules in a ruleset context for some branch belong to the same class create a leaf

node and assign to it that class If all branches of the trees have leaf nodes stop

Otherwise repeat steps 1 to 4 for each branch that has no leaf

To select an attribute to be a node in the decision tree (step 1 and 2 of the algorithm) the

algorithm performs two independent iterations In the first iteration it parses through all decision

rules and determine information about each attributes This information includes the importance

score of each attribute the number of rules containing a given attribute the disjoint value sets of

each attribute and the attribute values used in describing each decision class The second

iteration is only performed if the disjointness criterion is ranked first in the LEF function The

second iteration evaluates the attributes disjointness for each decision class against the other

decision classes

40

To determine the complexity of this process suppose that the maximum number of conditions

fonned by a single attribute value in one rule is s (where s ~ n the number of attributes) and r is

the total number of decision rules (in all decision classes)

m

r=LRt (m is the number of decision classes) 1=1

where Ri is the number of rules in decision class Ci bull The complexity of the frrst iteration can be

determined as

Cmpx(Itcl) =O(r s)

In the second iteration the disjointness is calculated between the decision classes and all

attributes The complexity of the second iteration can be given by

Cmpx(Itc2) = O(n m)

Assume that at each node 1 is the maximum of the number of decision classes to be classified at

this node and the number of rules associated with this node

l=max mr (3-9)

Since these two iteration are independent of each other the complexity of the AQDT algorithm

for building one node in the decision tree say Node Complexity NC(AQDT) is given by

NC(AQDT) = 0(1 n)

Usually I equals to the number of rules associated with the given node Thus the AQDT

complexity for building one node is a function of the number of attributes multiplied by the

number of rules associated with this node

At each level of the decision tree these two iterations are repeated for each non-leaf node The

Level Complexity of the AQDT algorithm LC(AQDT) can be given by

LC(AQDT) lt 0(1 n)

Which is less than the complexity of generating the root of the decision tree To explain this

consider the maximum possible number of non-leaf nodes at one level is half the number of the

initial decision rules (integer of rll) Figure 3-5-a shows an example of such situation where at

41

the lowest level each node classifies only two rules each belongs to different decision class This

decision tree could be a multimiddotvalues tree However the number of non-leaf nodes at each level

will be twice or more than the number of nonmiddot leaf nodes at the previous level Considering the

level complexity of the AQDT algorithm equal to (1 s 0) where 0 is the number of non-leaf

nodes at the given level In such cases it is either (1 0 S r) or (1 s lt r) In Figure 3middot5middota the

complexity of the AQDT algorithm at any lower level is given by

LC(AQDT) =0(2 s r(2) lt O(n I) =NC(AQDT)

a) per one level b) per one path

Figure 3 -5 Decision trees showing the maximum number of none leaf nodes

Note also that after selecting an attribute to be the root of the decision structure this attribute

and all conditions containing that attribute are removed from the data structure of the algorithm

Also if a leaf node is generated all rules belonging to the corresponding branch will not be

tested again

Since the disjointness criterion selects the attribute which minimizes the average number of tests

ANT it is true that the AQDT algorithm generates decision trees with the least number of levels

The number of levels per a decision tree is supposed to be less than or equal to the minimum of

both the number of attributes and the number of rules Consider k as the number of levels in a

given decision tree

k S min mr (3-10)

There are two cases represent the most complex situations Figure 3-5-a and 3-5-b In the fIrst

case where the decision rules were divided evenly the number of levels will be a function of the

42

logarithm of the number of rules In such case the complexity of the AQDT algorithm for

generating a decision tree from a set of decision rules is given by

Complexity(AQDT) = 0(1 n log r) (3-11)

The other situation is when the generated decision tree has the maximum number of levels The

maximum possible number of levels per a decision tree equals to one less than the number of

decision rules Figure 3-5-b Using the disjointness criterion it is not likely to get such a decision

tree because it has the maximum average number of test (ANT) that can be determined from the

same set of nodes and leaves However such a decision tree can be generated if the number of

decision classes is one less thanthe number of attributes In such case any disjQint decision rules

should have a maximum length that is less or equal to the floor of the logarithm of the number of

attributes Thus the level complexity of this decision tree is estimated as

LC(AQDT) =0(1 log n)

The maximum number of levels in such decision tree is k-l Thus the complexity of the AQDT

algorithm in such cases is given by

Complexity(AQDT) = 0(1 k log n) (3-12)

Since k is the minimum of n and r then n is the maximum of k and n Also since in (3-12)

r=n+1 then log r is greater than log n From 3-10 3-11 and 3-12 the complexity of the AQDT

algorithm is determined by

Cmplx(AQDT) = OCr k log 1) (3-13)

333 An example illustrating the algorithm

The following simple example illustrates how AQDT is used in selecting the optimal set of

testing resources for testing a software Suppose there are three tools for testing a software 1)

modeling (Tl) 2) checklist (T2) and 3) par_simul (T3) Also assume that there are four

different factors that affect the selection of any tool 1) the cost of using the tool (xl) 2) the

metrics that support the tool (x2) 3) the best phase for applying the tool (x3) and 4) type of tool

43

(automated semi-automated or manual) (x4) Table 3-1 shows these attributes and their possible

values

Suppose that the domain expert provided a set of rules to be used in testing a software Figure 3shy

6 shows a sample of these rules in AQ15c formats

Table 3-1 The available tools and the factors that affect the prclCes

Tl lt= [xl=2] amp [x2=2] v [xl=3] amp [x3=1 v 3] amp [x4=1] T2 lt= [xl=l v 2] amp [x2=3 v 4] v [xl=3] amp [x3=1 v 2] amp [x4=2] T3 lt= [xl=l] amp [x2=1] v [xl=4] amp [x3=2 v 3] amp [x4=3]

Figure 3-6 Decision rules for selecting the best tool for testing software

These rules can be interpreted as

Rule 1 Use the fltst tool for testing if you need average cost and the tool is

supported by the requirement metric

Rule 2 Use the fIrst tool for testing if you can afford high cost for testing either

in the requirement or the analysis phases and you need an automated tool

Rule 3 Use the second tool for testing if the cost limit is low or average and the

tool is supported by the system usage metric or the intractability metric

Rule 4 Use the second tool for testing if you can afford high cost for testing

either in the requirement or the design phases and you need a manual tool

Rule 5 Use the third tool for testing if the you are limited to low cost and the

tool is supported by the error rate metric

Rule 6 Use the third tool for testing if you can afford very-high cost for testing

either in the requirement or the system usage phases and you need a semishy

automated tool

44

Table 3-2 presents information on these rules and the disjointness values for all attributes For

each class the row marked Values lists values occurring in the ruleset for this class For

evaluating the disjointness of an attribute say A each rule in the ruleset above that does not

contain attribute A is characterized as having an additional condition [A= a vb ] where a b

are all legal values of A

The row Class disjointness specifies the class disjointness for each attribute The attribute xl

has the highest disjointness (11) and is assigned to the root of the tree For simplicity assume

the tolerances for each elementary criterion equal O

From the rules in Figure 3-6 we can also determine disjoint groupings of attribute-values used

for the compact mode of the algorithm This done as follows 1) determine for each attribute the

sets of values that the attribute takes in individual decision rules and remove those value sets

that subsume other value sets The remaining value sets are assigned to branches stemming from

the node marked by the given attribute For example x 1 has the following value sets in the

individual decision rules 2 3 I 2 I and 4 (Figure 3-6) Value set I 2 is

removed as it subsumes 2 and I In this case branches are assigned individual values of the

domain of x1 For attribute x2 the value sets are I 2 34 and I 2 3 4 In this case

branches are assigned value sets I 2 and 34

45

Attribute Xl ranks highest (as it is the highest disjointness) and is assigned to the root of the

tree Four branches are created each one corresponding to one of XIs possible values Since all

rules containing [x1=4] belong to class TI the branch marked by 4 is ended by a leafTI Rules

containing other values of X1 belong to more than one class This process is repeated for each

subset of rules until the decision tree is completed Figure 3-7 shows a decision structure learned

by AQDT-2 from the rules in Figure 3-6 The decision structure in Figure 3-7 can be used in

making decisions on which tools can be used for testing a given software

xl

Complexity No of nodes 4 No of leaves 7

Figure 3-7 A decision structure learned for classifying so tware testing tools

Figure 3-8-a shows the diagrammatic visualization of the decision rules and Figure 3-8-b shows

the visualization of the derived decision tree Each diagram in Figure 3-8 consists of cells

representing one combination of attribute-values Attributes and their legal values are shown on

scales surrounding the diagram (eg the horizontal scale for xl shows values 1 2 3 and 4)

Rules are represented by collections of cells in the intersection of the rows and columns

corresponding to the conditions in the rules

The shaded areas correspond to decision rules Rules of the same class have the same type of

shading Empty cells correspond to combination of attribute-values not assigned to any class For

illustration collections of cells corresponding to some of the initial rules are marked R 11 R 21

R31 and R32 Rll denotes the IlfSt rule of Class T1 ie [x1=2] amp [x2=2] R21 denotes the

IlfSt rule of class T2 ie [xl= 1 v 2] amp [x2= 3 v 4] R31 the first rule of class TI ie [x1=1]

46

amp [x2 =1] and denotes R32 denotes the second rule of class T3 Le [xl=4] amp [x3=2 v 3] amp

[x4=3]

a) Decision rules b) Derived decision tree

Comparing diagrams in Figure 3-8-a and 3-8-b AQDT-2 decision tree represents a slightly more

general description of Concepts Tl TI and T3 than the original rules Let us assume that it is

very costly to know which metrics suppon the required tools In other words suppose that we

would like to select the best tools independently of the metric they suppon (this is indicated to

AQDT-l by assigning very high cost to attribute x2) The algorithm again assigns attribute xl to

the root of the tree When xl takes value 1 or 2 it is impossible to assign a specific decision

without measuring attribute x2 For the value 1 of xl the recommended tool can be either TI or

T3 (see the diagram in Figure 3-9-a) and for the value 2 of xl the recommended tool can be

either Tl or T3 However for the value 3 of xl one can make a specific decision after

measuring attribute x4 For the value 1 of x4 tool is Tl and for the value 2 of x4 the

recommended tool is TI Figure 3-9-b shows another decision tree where the type of the tool

was ignored from the data Those decision trees are called indeterminate because some of their

leaves are assigned a disjunction of two or more class names

47

xl

xl

1 12

a) Ignoring the supporting metric b) Ignoring the type of the tooL Figure 3 -9 Decision trees learned ignoring the support metric and the type of the testing tool

It is clear that the given set of rules depend highly on amount of money can be spent Let us now

suppose that the cost attribute xl which was determined as the highest ranked attribute cannot

be measured The algorithm selected x4 to be the root of the new decision tree After continuing

the algorithm the tree in Figure 3-10 is obtainedThe nodes that are assigned one class indicate

situations in which it is possible to make a specific decision without knowing the value of

attribute xl

x4

Figure 3-10 A decision tree learned without the cost attribute

34 Tailoring Decision Structures to the Decision-making situation

Decision structures are among the simplest structures for organizing a decision-making process

A decision structure specifies explicitly the order in which attributes of an object or situation

need to be evaluated in the process of determining a decision A standard way to generate a

48

decision structure is to learn it from examples of decisions Such a process usually aims at

obtaining a decision structure that has the highest prediction accuracy that is the highest rate of

assigning correct decisions to given situations There can usually be a large number of logically

equivalent decision structures (Michalski 1990) As such they may have the same predictive

accuracy but differ in the way they organize the decision process and thus may differ in the cost

of arriving at a decision To minimize the average decision cost one needs to take into

consideration the distribution of the costs of attribute evaluation and the frequency of different

decisions This report presents an approach to building such Task-oriented decision structures

which advocates that they are built not from examples but rather from decision rules Decision

rules are learned from examples using the AQ15 or AQ17 inductive learning program or are

specified by an expert An efficient algorithm and a new system AQDT-2 that transforms

decision rules to task-oriented decision structures The system is illustrated by applying it to the

problem of learning decision structures in the area of construction engineering (for determining

the best wind bracing for tall buildings) In the experiments AQDT-2 outperformed all other

programs applied to the same data

Decision-making situations can vary in several respects In some situations complete

information about a data item is available (ie values of all attributes are specified) in others the

information may be incomplete To reflect such differences the user specifies a set of parameters

that control the process of creating a decision structure from decision rules AQDT-2 provides

several features for handling different decision-making problems 1) generating a decision

structure that tries to avoid unavailable or costly attributes (tests) 2) generating unknown

leaves in situations where there is insufficient information for generating a complete decision

structure 3) providing the user with the most likely decision when performing a required test is

impossible 4) providing alternative decisions with an estimate of the likelihood of their

correctness when the needed information can not be provided

49

341 Learning Costmiddot Dependent Decision Structures

As described in Sec 2 the LEF criterion enables the system to take into consideration the cost of

measuring tests (attributes) in developing a decision structure In the default setting of LEF the

tests cost is the [lIst criterion and its tolerance is O This means that only the least expensive

attributes pass to the next step of evaluation involving other elementary criteria If an attribute

has high cost or is impossible to measure (has an infinite cost) the LEF chooses another

cheaper attribute ifpossible

342 Assigning Decision Under Insufficient Information

In decision-making situations in which one or more attributes cannot be measured the system

may not be able to assign a definite decision for some cases If no more information can be

obtained but a decision has to be made it is useful to know the probability distribution for

different candidate decisions (Smyth Goodman amp Higgins 1990) The most probable decision is

then chosen The probability distribution can be estimated from the class frequency at the given

node Let us consider a node in the structure connected to the root by the sequence bl b2 bkof

attribute-values Let P(Ci I bl b2 bk) denote the conditional probability of class Ci i=I2 m

at that node given that an example to be classified has attribute-values assigned to branches bl

b2 bull bk in the decision structure Using Bayesian formula we have

P(Ci I bl~ bull bk) = P(Ci) P(bl bk I Ci) P(bl bk) (3-9)

where P(Ci) the apriori probability of class Ci P(bl bk I Ci) is the probability that the

example has attribute-values blbull bk given that it represents decision class Ci To approximate

these probabilities let us suppose that Wi is the number of training examples of Class Ci that

passed the tests leading to this node and twi - the total number of training examples of Class

Ci Let us use these numbers to estimate the probabilities in (9) Assuming that the apriori class

probabilities correspond to the frequency of training examples from different classes we have

50

m P(Ci) = twi IL twj (3-10)

j=l

P(b1 bkl Ci) = Wi I twi (3-11)

m m P(b1 bk) =L Wj IL twj (3-12)

j=l j=1

By substituting (1O) (11) and (12) in (9) we obtain

m P(Ci I b1 bk) = wilL Wj (3-13)

j=l

A related method for handling the problem of unavailability of an attribute is described by

Quinlan (1990) Quinlans method however assigns probabilities to the most probable decision

at the node associated with such an attribute The method does not restructure the decision tree

appropriately to fit the given decision-making situation (in this case to avoid measuring xl)

AQDT -2 assigns probabilities o~ly after it first created a decision structure that suits the given

decision-making si~ation An example is presented in section 42

343 Coping with noise in training data

The proposed methodology can be easily extended to handle the problem of learning from noisy

training data This is done based on ideas of rule truncation described in (Niblett amp Bratko 1986

Bergadanoet al 1992 Cestnik amp Karalic 1991) When noise is expected in the training data

the decision rules with t-weight below a certain threshold (reflecting the expected noise level) are

removed The rule truncation method seems to result in more accurate decision structures than

decision tree pruning method because truncation decisions are solely based on the importance of

the given rule or condition for the decision-making regardless of their evaluation order (unlike

the decision tree pruning which can only prunes attributes within a subtree and thus cannot

freely chose attributes to prune) Examples are presented in section 4

Associate Dean of the School of Infonnation Technology and Engineering for guidance on

preparing the PhD proposal

I would like to thank conferences organizers who supported me to attend their conferences and

present parts of my PhD work The organizers include Professor Moonis Ali Professor Frank

Anger Professor Zbigniew Ras the American Association for Artificial Intelligence and others I

would like also to thank the organizing committee of the Florida Artificial Intelligence Research

Symposium FLAIRS-95 for organizing my first workshop Thanks goes to Dr Doug Dankel Dr

Howard Hamilton Dr John Stewman and Dr Dan Tamir

I would like also to thank the many individuals who helped me in any way during my PhD Those

include Dr Ashraf Abdel-Wahab Dr Jerzy Bala Dr Alex Brodsky Dr Richard Carver Dr

Kenneth De Jong John Doulamis Tom Dybala Dean Ibrahim Farag Dr Ophir Frieder Csilla

Frakes Dr Mohamed Habib Ali Hadjarian Dr Hugo De Garis Zenon Kulpa Dr Paul Lehner

Dr George Michaels Dr Eugene Norris Dean James Palmer Mitch Potter Dr Ahmed Rafea

Jim Ribeiro Jsyshree Sarma Dr Arun Sood Dr Clifton Sutton Bradley Utz Dr Tibor Vamos

Patricia Zahra Dr Shaker Zahra and Dr Jianping Zhang

~J I yarJ I JJ I ~

U-t-oJ ~II ~) ~ oWAIl J

Dedication

To my mother my brothers and my sister

TABLE OF CONTENTS

TI1LE Page

ABS1RAcr II 1

CHAPrER 1

IN1RODUCI10N II 3

11 Motivation and Overview 3

12 The Problem Statement 6

CHAPrER2

REIAfED RESEARCFI 7

21 Learning Decision Trees from Decision Diagrams 7

22 Learning Decision Trees from Examples 10

221 Building Decision Trees Using Information-based Criteria 11

222 Building Decision Trees Using Statistics-based Criteria 16

223 Analysis of Attribute Selection Criteria 18

23 Learning Decision Structures 19

CHAPrER3

DESCRIPTION OF TIlE APPROACFI 23

31 General Methodology 23

32 A Brief Description of the AQ15 and AQ17 RuleLeaming Systems 25

33 Generating Decision Structures From Decision Rules 28

331 The AQDT-2 attribute selection method 29

332 The AQDT-2 algoridlm 37

vi

333 An example illustrating the algorithm 42

34 Tailoring Decision Structure to a decision-making situation 47

341 Learning Cost-Dependent Decision Structures 49

342 Assigning Decision Under Insufficient Infonnation 49

343 Coping with noise in training data 50

35 Analysis of the AQDT-2 Attribute Selection Criteria 51

36 Decision Structures vs Decision Trees 53

CHAPTER 4

EMPRICAL ANALYSIS AND COMPARATIVE STUDY OF TIlE MElHOD 58

41 Description of the Experimental Analysis 59

42 Experiments With Average Size Complex and Noise-Free Problems

Wind Brac-ings 60

43 Experiments With Small Size Simple and Noise-Free Problems

e MONK-I 69

44 Experiments With Small Size Complex and Noise-Free Problems

MONK-2 76

45 Experiments With Small Size Simple and Noisy Problems

MONK-3 79

46 Experiments With Large Size Complex and Noise-Free Problems

Diagnosing Breast Cancer 83

47 Experiments With Large Size Complex and Noisy Problems

Musm()()m classifications 84

48 Experiments With Small Size Structured and Noise-Free Problems

East-West Trains 85

49 Experiments With Small Size Simple and Noisy Problems

vii

Congressional Voting Records (1984) 87

410 Analysis of the Results 88

ClIAPfER 5 CONCIUSIONS 95

51 Summary 95

52 Contributions 96

REFERENCES 98

VITA 102

viii

LIST OF TABLES

No TITLE Page

2-1 An example of a decision table 9

2-2 A set of training examples used to illustrate the C45 system 15

2-3 The frequency of different attributes values to different decision classes 17

2-4 The expected values of the frequency of examples in Table 2-3 17

2-5 Attribute selection criteria and their basic evaluation measure 17

2-6 The contingency tables of Mingers example 18

2-7 Mingers results for determining the goodness of split 19

2-8 Mingers results for comaring the total accuracy and size of decision trees

provided by different attribute selection criteria from four problems 19

2-9 Comparing the AQDT approach with EDAG and HOODG approaches 22

3-1 The available tools and the factors that affect the process of testing a software 43

3-2 Calculating the disjointness of each attribute 44

3-3 Evaluation of the AQDT-2 criteria on the Quinlans example in Table 2-2 51

3-4 The data used in Mingers f11st experiments 52

3-5 The performance of AQDT-2 criteria (compare with the other criteria in Table 2-6) 52

3-6 The possible ranking domains and using condition of AQDT-2 criteria 53

3-7 Comparison between Decision Structures and Decision Tree 54

4-1 Evaluation of AQDT-2 attribute selection criteria for the wind bracing problem 62

4-2 The predictive accuracy of AQ15c and AQDT-2 for the wind bracing problem 67

4-3 Evaluation of AQDT-2 attribute selection criteria for the MONK-l problem 71

4-4 The predictive accuracy of AQ15c and AQDT-2 for the MONK-l problem 73

ix

4-5 The predictive accuracy of AQ15c and AQDT-2 for the MONK-2 problem 77

4-6 The predictive accuracy of AQ15c and AQDT-2 for the MONK-3 problem 81

4-7 The set of attributes and their values used in the trains problem 86

4-8 A tabular snmmary of the predictive accuracy of decision trees obtained

by AQDT -2 and C45 for the congressional voting data 88

4-9 Summary of the best parameter settings for the first subfunction of the approach

with different data characteristics 89

4-10 Summary of the performance of AQDT-2 and C45 on different problems 90

x

LIST OF FIGURES

No TITLE Page

2-1 An example to illustrate how attributes break rules 8

2-2 A decision tree learned from the decision table in Table 2-1 10

2-3 A decision tree learned using the gain criterion for selecting attributes 15

2-4 Decision rules and their Exception Directed Acyclic Graph (EDAG) 21

3-1 Architecture of the AQDT approach 24

3-2 A ruleset generated by AQ15 for the concept Voting pattern of

Democratic Representatives 27

3-3 Ven Diagrams ofPossible combinations of attribute values in two decision classes 33

3-4 Decision trees corresponding the Ven diagrams in Figure 3-3 33

3-5 Decision trees showing the maximum number of none leaf nodes 41

3-6 Decision rules for selecting the best tool for testing a software 43

3-7 A decision structure learned for classifying software testing tools 45

3-8 A diagrammatic visualization of the decision rules and the derived decision tree 46

3-9 Decision trees learned ignoring the support metric and the type of the testing tool 47

3-10 A decision tree learned without the cost attribute 47

3-11 Decision structures learned by AQDT-2 using different criteria 55

3-12 The Imams example Example where learning decision structures (trees)

from rules is better than learning them from examples 56

3-13 An example where decision rules are simpler than decision trees 57

4-1 Design of a complete experiment 59

Xl

4-2 Decision rules determined by AQ15c from the wind bracing data 61

4-3 A decision tree learned by C45 for the wind bracing data 63

4-4 A decision structure learned from AQ15c wind bracing rules 64

4-5 A decision structure that does not contain attribute xl 64

4-6 A decision structure without Xl with candidate decisions assigned to leaves 65

4-7 A decision structure determined from rules in Figure 4-4 under the

assumption of 10 classification error in the training data 65

4-8 Diagramatic visualization of decision trees learned for different decision

making situations for the wind bracing data 66

4-9 The accuracy of AQDT-2 and AQ15c with different parameter settings

for the wind bracing PIoblem 68

4-10 Analyzing different parameter setting of AQDT-2 using the wind bracing data 69

4-16 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-20 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-11 A comparison between AQ15c and AQDT-2 against C45 on the wind bracing data 69

4-12 A visualization diagram of the MONK-l problem 70

4-13 Decision rules learned by AQ15c for the MONK-1 problem 71

4-14 The decision tree for the MONK-1 problem generated both by AQDT-l 72

4-15 Compact decision structures generated by AQDT-2 for the MONK-1 problem 72

for the MONK-l Problem 74

4-17 Analyzing different parameter setting of AQDT-2 with the MONK-l data 75

4-18 Comparing AQ15c and AQDT-2 against C45 using the MONK-1 problem 75

4-19 A visualization diagram of the MONK-2 problem 76

for the MONK-2 Problem 78

4-21 Analyzing different parameter setting of AQDT-2 with the MONK-2 data 79

xii

4-22 Comparing AQ15c andAQDT-2 against C45 using the MONK-2 problem 79

4-24 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-35 A visualization diagram of a decision tree learned by AQDT -2 after reducing

4-23 A visualization diagram of the MONK-3 problem 80

for the MONK-3 Problem 82

4-25 Analyzing different parameter setting of AQDT-2 with the MONK-3 data 82

4-26 Comparing AQ15c and AQDT-2 against C45 using the MONK-3 problem 83

4-27 Comparing AQ15c and AQDT-2 against C45 using the breast cancer problem 84

4-28 Comparing AQ15c and AQDT-2 against C45 using the mushroom problem 85

4-29 Decision structures learned by AQDT-2 for different decision-making situations 87

4-30 Comparing decision trees for the Cong Voting-84 data learned by C45 amp AQDT -2 88

4-31 A visualization diagram of decision rules learned by AQ15c for the MONK-2 91

4-32 A visualization diagram shows the testing errors for the rules in Figure 4-31 92

4-33 A visualization diagram of the decision tree learned by AQDT-2 for the MONK-2 93

4-34 A visualization diagram shows testing errors of AQDT-2 decision tree bull 94

the generalization degree to 1 94

DERIVING TASKmiddotORIENTED DECISION STRUCTURES

FROM DECISION RULES

Ibrahim M Fahmi Imam PhD

School of Information Technology and Engineering

Geotge Mason University Fall 1995

Ryszard S Michalski Advisor

ABSTRACT

This dissenation is concerned with research on learning task-oriented decision structures from

decision rules The philosophy behind this research is that it is more appropriate to learn

knowledge and store it in a declarative form and then when a decision making situation occurs

generate from this knowledge the decision structure that is most suitable for the given decision

making situation Learning decision structures from decision rules was first introduced by

Michalski (1978) The flIst implementation of this approach was done by Imam and Michalski

(1993 ab) called AQDT-l

This approach separates the function of generating a knowledge-base from the function of using

the knowledge-base for decision-making The first function focuses on learning accurate

consistent and complete concept description expressed in a declarative form The second function

is performed whenever a new decision-making situation occurrs a task-oriented decision structure

is obtained to suit that situation Thsk-oriented knowledge is defined as knowledge that is adapted

for solving a given decision-making situation (Imam amp Michalski 1994 Michalski amp Imam

1994)

The dissertation introduces the system AQJJT-2 for learning task -oriented decision structures from

decision rules or examples Each decision making situation is defined by a set of parameters that

controls the learning process of the AQDT-2 system The extensive experiments on AQJJT-2 show

that decision structures learned by it usually outperform in terms of accuracy and average size of

the decision structures those learned from examples by other well known systems The results

show also that the system does not work very well with noisy data The system is illustrated and

compared using applications of artificial problems such as the three MONKs problems (Thrun

Mitchell amp Cheng 1991) and the East-West 1iain problem (Michie et al 1994) It was also

applied to real world problems of learning decision structures in the areas of construction

engineering (for determining the best wind bracing design for tall buildings) medical

diagnosis(for learning decision rules for recognizing breast cancer) agriculture diagnosis (for

learning classification rules for distinguishing between poisonous and non-poisonous

mushrooms) and political data (for characterizing democratic and republican voting records)

CHAPTER 1 INTRODUCTION

11 Motivation and Overview

Learning and discovery systems should be able not only to generate and store knowledge but also

to use this knowledge for decision-making The main step in the development of systems for

decision-making is the creation of a knowledge structure that characterizes the decision-making

process The form in which knowledge can be easily obtained may however differ from the form

in which it is most readily used for decision-making It is therefore important to identify the form

of knowledge representation that is most appropIiate for learning (eg due to ease of its

modification) and the form that is most convenient for decision making

A simple and effective tool for describing decision processes is a decision structure which is a

directed acyclic graph that specifies an order of tests to be appliedto an object (or a situation) to

arrive at a decision about that object The nodes of the structure are assigned individual tests

(which may correspond to a single attribute a function of attributes or a relation) the branches are

assigned possible test outcomes or ranges of outcomes and the leaves are assigned a specific

decision a set of candidate decisions with corresponding probabilities or an undetermined

decision A decision structure reduces to a familiar decision tree when each node is assigned a

single attribute and has at most one parent when the branches from each node are assigned single

values of that attribute and when leaves are assigned single definite decisions Thus the problem

of generating a decision structure is a generalization of the problem of generating a decision tree

Decision trees are typically generated from a set of examples of decisions The essential

characteristic of any such method is the attribute selection criterion used for choosing attributes to

be assigned to the nodes of the decision tree being built Such criteria include the entropy

reduction the gain and the gain ratio (Quinlan 1979 83 86) the gini index ofdiversity (Breiman

et al 1984) and others (Cestnik amp Bratko 1991 Cestnik amp Karalic 1991 Mingers 1989a)

3

4

Adecision treedecision structure representations can be an effective tool for describing a decision

process as long as all the required tests can be performed and the decision-making situations it

was designed for remain constant (eg in a doctor-patient example the doctor should detennine

that answers for all symptoms appear in the decision tree) Problems arise when these assumptions

do not hold For example in some situations measuring certain attributes may be difficult or costly

(eg in the doctor-patient example a brain or blood test is needed which is very expensive or the

tools needed are not available) In such situations it is desirable to refonnulate the decision

structure so that the inexpensive attributes are evaluated frrst (assigned to the nodes close to the

root) and the expensive attributes are evaluated only if necessary (by assignment to the nodes far

away from the root) If an attribute cannot be measured at all it is useful to either modify the

structure so that it does not contain that attribute or-when this is impossible-to indicate

alternative candidate decisions and their probabilities A restructuring is also desirable if there is a

significant change in the frequency of occurrence of different decisions (eg in the doctor-patient

example the doctor may request a decision structure expressed in a specific set of symptoms

biased to classify one or more diseases or specify a certain order of testing)

Arestructuring ofa decision structure (or a tree) in order to suit new requirements could be quite

difficult This is because a decision structure is a procedural representation that imposes an

evaluation order on the tests In contrast no evaluation order is imposed by a declarative

representation such as a set of decision rules Tests (conditions) of rules can be evaluated in any

order Thus for a given set of rules one can usually build a huge number of logically equivalent

decision structures (trees) which differ in the test ordering Due to the lack of order constraints

a declarative representation (rules) is much easier to modify to adapt to different situations than a

procedural one (a decision structure or a tree) On the other hand to apply decision rules to make a

decision one needs to decide in which order tests are evaluated and thus needs a decision

structure

An attractive solution to these opposite requirements is to acquire and store knowledge in a

declarative form and transform it to a decision structure when it is needed for decision-making

5

This method allows one to create a decision structure that is most appropriate in a given decisionshy

making situation Because the number of decision rules per decision class is usually small (each

rule is a generalization of a set of examples) generating a decision structure from decision rules

can potentially be perfonned much faster than by generation from training examples Thus this

process could be done on line without any delay noticeable to the user Such virtual decision

structures are easy to tailor to any given decision-making situation

This approach allows one to generate a decision structure that avoids or delays evaluating an

attribute that is difficult to measure in some decision-making situation or that fits well a particular

frequency distribution of decision classes In other situations it may be unnecessary to generate

complete decision structure but it may be sufficient to generate only a part of it which concerns

only with decision classes of interest Thus such an approach has many potential advantages

This dissertation presents a new system called AQflf-2 The AWf-2 system generates a taskshy

oriented decision structure (decision structure that is adapted to the given decision-making

situation) from decision rules The decision rules are learned by either rule leaming system AQ15

(Michalski et al 1986) or system AQ17-DCI which has extensive constructive induction

capabilities (Bloedorn et al 1993)

To associate the decision rules with a given decision-making task AQflf-2 provides a set of

features including 1) enabling the system to include in the decision structure nodes corresponding

to new attributes constructed during the process of learning the decision rules 2) controlling the

degree of generalization needed during the development of the decision structure 3) providing four

new criteria for selecting an attribute to be a node in the decision structure that allow the system to

generate many different but equivalent decision structures from the same set of rules 4) generating

unknown nodes in situations when there is insufficient infonnation for generating a complete

decision structure 5) learning decision structures from discriminant rules as well as

characteristic rules and 6) providing the most likely decision when the decision process stops

due to inability to evaluate an attribute associated with an intennediate node

6

To test the methooology of generating decision structures from decision rules an extensive set of

planned experiments have been designed to test different aspects of the approach The experiments

include testing different combinations of parameters for each sub-function of the approach

analyzing the relatioship between decision rules and the decision structure learned from them and

comparing decision trees learned by the AQUf2 system the well known C45 (Quinlan 1993)

system for learning decision trees from examples Different experiments were designed to examine

the new features of the AQUf approach for learning task-oriented decision structures The

experiments were applied to artificial domains as well as real-world domains including MONK-I

MONK-2 and MONK-3 (Thrun Mitchell amp Cheng 1991) East-West trains (Michie et al

1994) Engineering Design-wind bracings (Arciszewski et al 1992) Mushrooms Breast

Cancer (Mangasarian amp WolbeJg 1990) and Congressional bting Records of 1984 The

MONKs problems are concerned with learning classification rules for robot-like figures MONK-l

requires learning a DNF-type description MONK-2 requires learning a non-DNF-type description

(one that cannot be easily described as a DNF rule using the original attributes) MONK-3 concerns

learning a DNF rule from noisy data The East-West trains dataset is a structural domain that

classifies two sets of trains (Eastbound and Westbound) The Engineering Design-wind

bracing data involves learning conditions for applying different types of wind bracing for ta11

buildings The Mushrooms data is concerned with learning classification rules for distinguishing

between poisonous and non-poisonous mushrooms The Breast Cancer data involves learning

concept descriptions for recognizing breast cancer The congressional voting data includes voting

records on different issues AQUf2 outperformed C45 on the average with respect to both the

predictive accuracy and tree size for most problems Apf-2 did not work very well with noisy

data or with problems that have many rules covering very few examples

12 The Problem Statement

There are many limitations and problems accompanied using decision trees for decision-making

CHAPTER 2 RELATED RESEARCH

21 Learning Decision Trees from Decision Diagrams

The AQDT-2 method proposed here is based on the earlier work by Michalski (1978) which

introduced an algorithm for generating decision trees from decision lists The method proposed

several attribute selection criteria These criteria are of increasing power of the main criterion

order cost estimate (the nth order cost estimates n=l 2 ) Michalski also analyzed two

specific criteria MAL and DMAL for selecting the optimal attribute for a node in the tree

based on properties extracted from the decision diagram In order to better explain the method

it is necessary to define some terms These terms will be used later in the dissertation

Definition 2-1 A ~ of a decision class C is a set of rules whose union includes the set of all

examples of class C and does not include any examples of other decision classes

Definition 2-2 A ~ of a decision class C is disjoint if all rules (rules) are pairwise logically

disjoint In other words for any two rules there exists a condition with the same attribute but

with different values in each rule

Definition 2-3 An minimal cover is a disjoint cover which has the smallest number of rules

among all possible covers

Definition 2-4 A diagram for a given cover is a table constructed graphically by representing

in two dimension space of all possible combination of attributes values and locating on the

diagram all the condition parts of the given rules and marking them with the action specified

with each rule

Michalskis algorithm requires construction of minimalmiddot cover The minimal cover should be

consistent and complete The method is based on the fact that if there are n decision classes

any decision tree that correctly classifies the given rules and has n distinct leaves is a minimal

7

8

decision tree (any consistent decision tree should have at least n leaves) Michalski (1978) has

shown that if only one rule is broken by a selected attribute then instead of having one leaf

(which could potentially represent this rule or the decision class in the tree) there will have to

be at least two leaves representing this rule in the fmal decision tree

The attribute selection criterion MAL introduced in (Michalski 1978) prefers attributes that do

not break any rules or break as few as possible An attribute breaks a rule if the attribute can

divide the rule into two or more sub-rules Figure 2-1 shows two examples of two sets of rules

In the fJIst xl and x3 break at least two rules each (xl breaks the rule [x4=2] amp [x1=lv3] amp

[x3=2v3] and the rule [x4=3] amp [x1=1v2] amp [x3=1] and x3 breaks three rules the rule [x4=2]

amp [xl=2] amp [x3=2v3]the rule [x4=l] amp [xl=3] amp [x3=lv3] and the rule [x4=3] amp [xl=4] amp

[x3=2v3]) In the diagram on the right xl is the only attribute that does not break any rule

Figure 2-1 An example to illustrate how attributes break rules

One of the criteria defmed by Michalski is the first degree cost estimate which assigns to each

attribute an integer equal to the number of rules broken by that attribute This criterion is also

called the static cost estimate of an attribute or the criterion of minimizing added leaves

(MAL)

-

C3

9

The MAL criterion (Minimizing Added Leaves) seeks an attribute that minimizes the

estimated number of additional nodes in the decision tree being generated over a hypothetical

minimal decision tree When there is a tie between two attributes the attribute to be selected is

the one which breaks smaller rules (rules that cover fewer examples or more specialized

rules) AQDT-2 uses an approximate version of this criterion (the attribute dominance)

Another criterion introduced by Michalski was the DMAL criterion The DMAL criterion

(Dynamically Minimizing Added Leaves) is based on a principle similar to that of MAL but

is more complex because once an attribute is selected as a node in the tree some rules andor

parts of the broken rules at each branch are merged into one rule The DMAL ensures that the

value of the total cost estimate of an attribute is decreased by a value equal to the number of

merged rules minus one

Example Learn a decision tree from the following decision table

The minimal cover consists of the following rules

Al lt [x2=O] v [x1=0][x2=2] A2 lt [x2=I] v [xl=2][x2=2] A3 lt [xl=I][x2=2]

The evaluations ofthe MAL criterion for these attributes are 2 for xl 0 for x2 5 for x3 and 5

for x4 The attribute xl divides the two rules [x2=O] and [x2=I] so selecting it will add two

leaves to the optimal number of leaves It is clear that the attribute to be selected as a root of

10

the decision tree is x2 Then three branches are attached to the root node and the decision rules

are divided into subsets each corresponding to one branch For x2 = 0 or 1 a leave node is

generated For x2=2 another attribute is selected to be a node in the tree In this case xl has the

minimum MAL value Figure 2-2 shows the decision tree obtained using the MAL criterion

Al A3 A2

Figure 2-2 A decision tree learned from the decision table in Table 2-1

22 Learning Decision Trees from Examples

Decision tree learning is a field that concerned with generating decision tree that classifies a set

of examples according to the decision classes they belong to The essential aspect of any

inductive decision tree method is the attribute selection criterion The attribute selection

criterion measures how good the attributes are for discriminating among the given set of

decision classes The best attribute according to the selection criterion is chosen to be assigned

to a node in the tree The fIrst algorithm for generating decision trees from examples was

proposed by Hunt Marin and Stone (1966) Hunts algorithm uses a divide and conquer

algorithm for building decision trees This algorithm has been subsequently modified by

Quinlan (1979) and applied by many researchers to a variety of learning problems

Attribute selection criteria can be divided into three categories These categories are logicshy

based information-based and statistics-based The logic-based criteria for selecting attributes

use logical relationships between the attributes and the decision classes to determine the best

attribute to be a node in the decision tree such as the MAL criterion minimizing added leaves

(Michalski 1978) which uses conjunction and disjunction operators The information-based

criteria are based on the information theory These criteria measure the information conveyed

11

by dividing the training examples into subsets Examples of such criteria include the

information measure 1M the entropy reduction measure and the gain criteria (Quinlan 1979

83) the gini index of diversity (Breiman et al 1984) Gain-ratio measure (Quinlan 1986) and

others (Clark amp Niblett 1987 Bratko amp Lavrac 1987 Cestnik amp Karalic 1991) The

statistics-based criteria measure the correlation between the decision classes and the attributes

These criteria use statistical distributions for determining whether or not there is a correlation

The attribute with the highest correlation is selected to be a node in the tree Examples of

statistics-based criteria include Chi-square and G-statistic (Sokal amp Rohlf 1981 Han 1984

Mingers 1989a)

Niblett and Bratko (1986) Quinlan (1987) and Bratko and Kononeko (1987) extended the

method of learning decision trees to also handle data with noise (by pruning) Handling noise

extended the process of learning decision trees to include the creation of an initial complete

decision tree tree pruning which is done by removing subtrees with small statistical Validity

and replacing them by leaf nodes (MiDgers 1989b) More recently pruning has also been used

for simplifying decision trees even for problems without noise (Bohanec amp Bratko 1994)

Pruning decision trees improves their simplicity but reduces their predictive accuracy on the

training examples Quinlan (1990) proposed also a method to handle the unknown attributeshy

value problem by exploring probabilities of an example belonging to different classes

The rest of this section includes a brief description of the attribute selection criterion used by

C45 learning system (Quinlan 1993) C45 uses an information-based criterion for selecting

an attribute to be a node in the tree Also the section includes a brief description of the Chishy

square method for attribute selection (Mingers 1989a) The method is a statistics-based

method for selecting an attribute to be a node in the tree

221 Building Decision Trees Using htformation-based Criteria

12

This section presents a description of the inductive decision tree learning system C4S The

C4S learning system is considered to be one of the most stable accurate and fastest program

for learning decision trees from examples

Learning decision trees from examples requires a collection of examples Each example is

represented by a fixed number of attribute-value pairs C45 (Quinlan 1993) is a learning

program that induces classification decision trees from a set of given examples The C4S

learning system is descended from the learning system ID3 (Quinlan 1979) which is based on

Hunts method for constructing decision tree from a set ofcases (Hunt Marin amp Stone 1966)

The C45 system uses an attribute selection criterion called the Gain Ratio This criterion

calculates the gain in classifying information based on the residual information needed to

classify cases in a set of training examples and the information yielded by the test based on the

relative frequencies of the possible outcomes (decision classes) The gain ratio criterion is

based on an earlier criterion used by ID3 called the Gain Criterion The Gain Criterion uses

the frequency of each decision class in the given set of training examples

Once an attribute is chosen to be a node in the tree the system generates as many links as the

number of its values and classifies the set of examples based on these values If all the

examples at a certain node belong to one decision class the system generates a leaf node and

assigns it to that class Otherwise the system searches for another attribute to be a node in the

tree

The Gain Criterion The gain criterion is based on the information theory That is the

information conveyed by a message depends on its probability and can be measured in bits as

minus the logarithm (base 2) of that probability To explain the gain criterion suppose that for

a given problem Xu bullbull X are given attributes and Cu c are decision classes Suppose S is

any set of cases and T is the initial set of training cases The frequency of class C1 in the set S

is the number of examples in S that belong to class C i bull

13

freq(Cit S) = Number of examples in S belong to CI (2-1)

Suppose that lSI is the total number of examples in S the probability that an example selected

at random from S belongs to class Ci is freq(CIbullS) IISI

The information conveyed by the message that a selected example belongs to a given decision

class Cit is determined by -log1 (freq(Ch S) IISI) bits

The expected information from such a message stating class membership is given by

info(S) = -k

L (freq(C S) I lSI) log1 (freq(C S) IISI) bits (2-2)I

info(S) is known also as the entrQpy of the set S When S is the initial set of training examples

info(T) determines the average amount of information needed to identify the class of an

example in T

Suppose that we selected an attribute X to be the root of the tree and suppose that X has k

possible values The training set T will be divided into k subsets each corresponding to one of

Xs values The expected information of selecting X to partition the training set T infoz(T) can

be found as the sum over all subsets of multiplying the information conveyed by each subset

by its probability

k

infoz(T) =L (ITII 1m) infO(TIraquo (2-3) I

The information gained by partitioning the training examples T into subset using the attribute

X is given by

gain(X) =inlo(T) - inoIT) (2-4)

The attribute to be selected is the attribute with maximum gain value

The Gain Ratio Criterion This criterion indicates the proportion of information generated by

the split that appears helpful for classification Quinlan (1993) pointed out that the gain

criterion has a serious deficiency Basically it is strongly biased toward attributes with many

outcomes (values) For example for any data that contains attributes such as social security

14

number the gain criterion will select that attribute to be the root of the decision tree However

selecting such attributes increases the size of the decision tree Quinlan provided a solution to

this problem by introducing the gain ratio criterion which takes the ratio of the information that

is gained by partitioning the initial set of examples T by the attribute X to the potential

information generated by dividing T into n subsets

Following similar steps to obtain the information conveyed by dividing T into n subsets The

expected information generated by dividing T into n subsets and by analogy to equation 2-2 is

determined by

split info(T) = - Lt

(ITII lTI) 10gl (ITII m ) (2-5) I

The gain ratio is given by

gain ratio(X) = gain(X) I split info(X) (2-6)

and it expresses the proportion of information generated by the split that is useful for

classification

Example Consider the following example presented by Quinlan (1993) Table 2-2 shows the

set of training examples

First determine the amount of information gained by selecting the attribute outlook to be a

root of the decision tree This attribute divides the training examples into three subsets

sunny-with five examples two of which belong to the class Play overcast-with four

examples all of which belong to the class Play and rain-with five examples three of

which belong to the class Play To determine info(11 the average information needed to

identify the class of an example in T there are 14 training examples and two decision classes

Nine of these examples belong to the class Play and five belong to the class Dont Play

Info(11 = - 914 10gl (914) - 514 10gl (514) = 094 bits

When using outlook to divide the training examples the information becomes

15

infO(T) = 514 -25 log2 (25) - 35 10g2 (35)

+ 414 -44 10g2 (44) - 04 log2 (04)

+ 514 -35 logl (35) - 25 logl (25) = 0694 bits

By substituting in equation 2-4 the gain of information results from using the attribute

outlook to split the training examples equal to 0246 The gain information for windy is

0048

Figure 2-3 shows a decision tree learned for this problem using the gain criterion The split

information for outlook is determined as follows

split info(T) = - 514 logl (514) - 414 log2 (414) - 514 log2 (514) = 1577 bits

The gain ratio for outlook = 02461577 =0156

16

Figure 2 -3 A decision tree learned using the gain criterion for selecting attributes

The C45 system handles discrete values as well as continuous values To handle an attribute

with continuous values C45 uses a threshold to transform the continuous domain into two

intervals In other words for each continuous attribute C45 generates two branches one

where the values of that attribute is greater than the determined threshold and the other if the

value is less than or equal to the threshold

Tree pruning in C45 is a process of replacing subtrees with small classification validity by

leaves The C45 system uses Laplace ratio for determining the error rate of different subtrees

This ratio is defined as (e+l) (n+2) where n is the number of the training examples and e is

the number of misclassified examples at a given leaf

222 Building Decision Trees Using Statistics-based Criteria

The Chi-square method for selecting attributes was used by Hart (1984) and Mingers (1989a)

in building decisionmiddot trees The method uses Chi-square statistics to measure the association

between two attributes When building decision trees the method is implemented such that it

determines the association between each attribute and the decision classes The attribute to be

selected is the one with greatest value

To determine the Chi-square value for an attribute consider aij is the number of examples in

class number i where the attribute A takes value numberj In other words aij is the frequency

of the combination of decision class number i and the attribute value number j The Chi-square

value for attribute A is given by

n m

Chi-square (A) =L L [ (aij - Eijl2 Eij ] (2shy

i=1 j=1

where n is the number of decision classes m is the number ofvalues of a given attribute Also

17

Eij = (TCi TVj ) T (2shy

8)

where T Ci and T Vj are the total number of examples belong to the decision class a and the total

number of examples where the attribute A takes value vj respectively T is the total number of

examples

Consider Quinlans example in Table 2-2 Table 2-3 shows the frequency of different

combination of values between the decision class and both the outlook and the Windy

attributes Table 2-4 shows the expected values TCi and T Vj of the frequencies in Table 2-3

of different attributes values for different decision classes

To determine the association value between the decision classes and both the attribute Windy

and the attribute Outlook the observed Chi-square values are

Chi-square (Windy Class) =[(3-39)239] + [(3-21)221] + [(6-51)232] + [(2-29)232]

=021 + 039 + 025 + 025 = 11

Chi-square (Outlook Class) = [(2-32)232] + [(4-26)226] + [(3-32)232] + [(3-18118]

+ [(0-14)214] + [(2-18)2 18] =045 + 075 + 02 + 08 + 14 + 002 = 362

18

Applying the same method to the other attributes the results will favor the attribute Outlook

Once that attribute is selected to be a node in the tree the remaining set of examples are divided

into subsets and the same process is repeated on each subset

Table 2-5 shows a summary of these criteria and their basic evaluation function

Table 2-5 Attribute selection criteria and their basic evaluation measure

Info Measure (IM) Gain Gshystatistics and Gain Ratio Chi-square

II

Entropy(S) = - L (freq(~ 5) 1151) logz (freq(CIt 5) 1151 ) I

G-Statistics =2N 1M (N-No of examples) n m

Chi-square (A B) = L L [(aij - ~j)21Eij ] i=l

223 Analysis of Attribute Selection Criteria

This subsection introduces briefly the analysis of different selection criteria which was done by

Mingers (1989a) Mingers compared six attribute selection criteria that are used in decision

tree programs These criteria are the Information Measure (IM) Chi-square G statistics Gin

index of diversity Marshal correction and Gain RaJio The overall results show that the Gain

Ratio criterion had the strongest xesults

In the flISt experiment Minger tested the six criteria on a problem with ambiguous examples

(ie examples may belong to more than one decision class) to observe how the selected criteria

evaluate the given attributes The problem has two decision classes and two attributes X and

Y It was assumed that attribute X is better for classifying the examples than attribute Y The

training examples were unevenly spread between the two values of X Attribute Y has three

values and the examples were spread randomly among them Table 2-6 (a and b) shows the

contingency tables for both attributes Table 2-7 shows a summary of the goodness of split

provided by the six criteria Mingers noted that the measures that are not based on information

19

theory give radiation (attribute X here) less weighL This may be because the zero in the fIrst

row of radiation has a greater influence in the log calculation In the case of using the Chishy

square criterion the value zero adds the maximum association between any two attributes

because the Chi-square value of a zero cell is the expected value of this cell

b)

Now let us demonstrate results from another experiment done by Mingers In this experiment

Mingers used four different data sets to generate decision trees for eleven different criteria In

the final results he compared the total number of nodes and the total error rate provided by

each criterion over all given problems Table 2-8 shows the final results for five selected

criteria only

20

U lgt ~middot results for comparing the total accuracy and size of decision trees different attribute selection criteria from four VVLU

This experiment was performed on four real world data sets These data are concerned with

profiles of BA Business Studies degree students recurrence of breast cancer classifying types

of Iris and recognizing LCD display digits The data was divided randomly 70 for training

and 30 for testing For more details see Mingers (1989a)

23 Learning Decision Structures

Considering the proposed defInition of decision structures given above two related lines of

research are described in this section Gaines (1994) and Kohavi (1994) proposed two

approaches for generating decision structures that share some of the earlier ideas by Imam and

Michalski (1993b)

In the first approach Brian Gaines introduces a method for transforming decision rules or

decision trees into exceptional decision structures The method builds an Exception Directed

Acyclic Graph (EDAG) from a set of rules or decision trees The method starts by assigning

either a rule say RO or a conclusion say CO or both to the root node If it assigns a conclusion

to the root node it places it on a temporary conclusion list Then it generates a new child node

and add to it a new rule say Rl The method evaluates the new rule Rl if it satisfies the

conclusion CO in its parent node it builds a new child and repeats the processes Otherwise if

the rule Rl does not satisfy the conclusion CO it changes the temporary memory with the new

conclusion Cl which is satisfied by the rule RO The same process is repeated until all rules

that have common conditions with the rule at the root are evaluated The method then creates a

21

new child node from the root and repeat the process until all rules are evaluated In the decision

structure nodes containing rules only represent common conditions to all of its children

The main disadvantages of this approach is that it requires discriminant rules to build such a

decision structure Also such a structure is more complex than the traditional decision trees

that are used for decision-making

Figure 2-4 shows an example of set of rules and their equivalent exceptional decision structure

The decision structure can be read as follows It is Safe except if xl=1 amp x2=1 amp x3=1 and

either x4=3 or x5=I then it is Lost except if x6=1 it is Safe except if x7=1 it is Lost

The second approach introduced by Ronny Kohavi learns decision structures from examples

using a bottom-up algorithm The method uses a greedy Hill-climbing algorithm for inducing

Oblivious read-Once Decision Graphs (HOODO) A read-once decision graph is defined as a

decision graph where each attribute occurs at most once along any computational path In other

words for each path from the root to the leaves of the decision structure attributes may occur

as a node once at most However there may be more than one node with the same attribute in

the decision graph Kohavi also defined a leveled decision graph in which the nodes are

partitioned into a sequence of pairwise disjoint sets such that outgoing edges from each level

terminate at the next level An oblivious decision graph is a decision graph where all nodes at

a given level are labeled by the same attribute

22

Safe lt [xl=2] Safe lt [x2=2] Safe lt [x3=2] Safe lt [x4=1] amp [x5=2] Safe lt [x4=1] amp [x5=3] Safe lt [x6=l] amp [x7=2] Safe lt [x6=1] amp [x7=3] Safe lt [x4=2] amp [x5=2] Safe lt [x4=2] amp [x5=3]

Lost lt [x6=2] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x6=3] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x5=1] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=2] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=3] amp [x3=1] amp [x2=I] amp [xl=l]

The algorithm starts by generating a leaf node for each decision class Then it applies a

nondeterministic method to select an attribute to build new level of the decision structure This

attribute is removed from the data and the data is divided into subsets each corresponding to a

complex combination of that attributes values For each subset the process is repeated until all

the examples of a given subset belong to one decision class For example if the selected

attribute say A has two values (0 and 1) and there are two decision classes (CO and CI) The

data is divided into two subsets the fIrst subset contains examples where A takes value 0 and

belong to class CO or takes value 1 and belong to class Cl The second subset of examples is

the set where A takes value 0 and belong to class CI or takes value 1 and belong to class CO

The number of nodes of the fIrst level (after the leave nodes) is expected to be less than or

equal to ka where k is the number of decision classes and n is the number of values of the

selected attribute and it can increase exponentially before that number is reduced

exponentially to one

It is easy for the reader to fIgure out some major disadvantages of such approach including

The average size of such decision structures is estimated to be very large especially when there

23

is no similarity (ie strong patterns) or logical relationship in the data The time used to learn

such decision structure is relatively very high comparing to systems for learning decision trees

from examples and fmally it could be better to search for attribute which reduces the number

of generated subsets of the data instead of nondeterministically selecting an attribute to build a

new level of the decision structure Kohavi provided a comparison between the C45 system

and his system that can be found in (Kohavi 1994) Later Kohavi and Li (1995) used a

deterministic method for attribute selection which minimizes the width of the penultimate level

of the graph

Table 2-9 shows a comparison between the proposed approach and those two approaches The

EDAG and HOODG systems are unreleased prototype systems

or Decision structures are easy to Decision structures are Decision structures are easy to understand difficult to read understand

CHAPTER 3 DESCRIPTION OF THE APPROACH

31 General Methodology

In the proposed approach the function of learning or discovery is separated from the function of

using the discovered knowledge for decision-making The frrst function is performed by an

inductive learning program that searches for knowledge relevant to a given class of decisions

and stores the learned knowledge in the form of decision rules The second function is performed

when there is a need for assigning a decision to new data points in the database (eg bull a

classification decision) by a program that transforms obtained knowledge into a decision

structure optimized according to the given decision-making situation

The Learning Task GivenA set of training examples describing the concept to be learned Learning goal which specifies the decision classes to be learned from the training examples A background knowledge to control the learning process Determine A concept description in a declarative form of knowledge (decision rules) that satisfies the learning goal

The Decision-making Task GivenA set of decision rules in a conjunctive form A description of the new decision-making situation (eg attributes costs and order preference importance or frequency of decision classes etc) One or more examples that needed to be tested under the given decisionshymaking situation A set of parameters to control the learning process Determine A decision structure that suits the given decision-making situation

The decision rules used here are learned by either the AQ1S (Michalski et al 1986) or AQ17

(Bloedorn et aI 1993) learning systems The reasons for selecting decision rules as a form of

23

24

declarative knowledge are that they do not impose any order on the evaluation of the attributes

and due to the lack of order constraints decision rules can be evaluated in many different

ways which increases the flexibility of adapting them to the different tasks of decision-making

(Bergadano et al 1990) Since the number of rules learned per class is typically much smaller

than the number of examples per class generating a decision structure from decision rules can

potentially be done on line

Such virtual decision structures can be tailored to any given decision-making situation The

needed decision rules have to be generated only once and then they can be used many times for

generating decision structures according to changing requirements of decision-making tasks The

method uses the AQDT-2 system (Imam amp Michalski 1994) for learning decision structures

from decision rules Decision structures represent a procedural form of knowledge which makes

them easy to implement but also harder to change Consequently decision structures can be quite

effective and useful as long as they are used in decision-making situations for which they are

optimized and the attributes specified by the decision structure can be measured without much

cost Figure 3-1 shows an architecture of the proposed methodology

Data

Decision -

- Learning knowledge from database The decision making process

Figure 3 -1 Architecture of the AQDT approach

25

It is assumed that the database is not static but is regularly updated A decision-making problem

arises when there is a case or a set of cases to which the system has to assign a decision based on

the knowledge discovered Each decision-making situation is defmed by a set of attribute-values

Some attribute-values may be missing or unknown A new decision structure is obtained such

that it suits the given decision-making problem The learned decision structure associates the

new set of cases with the proper decisions

32 A Brief Description of the AQIS and AQ17 Rule Learning Programs

The decision rules are generated from examples by an AQ-type inductive learning system

specifically by AQ15 (Michalski et albull 1986) or AQI7-DCI (Bloedorn at al 1993) The rest of

this subsection includes a brief description of the inductive learning systems AQ15 and AQI7

AQ15 learns decision rules for a given set of decision classes from examples of decisions using

the STAR methodology (Michalski 1983) The simplest algorithm based on this methodology

called AQ starts with a seed example of a given decision class and generates a set of the

most general conjunctive descriptions of the seed (alternative decision rules for the seed

example) Such a set is called the star of the seed example The algorithm selects from the star

a description that optimizes a criterion reflecting the needs of the problem domain If the

criterion is not defined the program uses a default criterion that selects the description that

covers the largest number of positive examples (to minimize the total number of rules needed)

and with the second priority that involves the smallest number of attributes (to minimize the

number of attributes needed for arriving at a decision)

If the selected description does not cover all examples of a given decision class a new seed is

selected from uncovered examples and the process continues until a complete class description

is generated The algorithm can work with few examples or with many examples and can

optimize the description according to a variety of easily-modifiable hypothesis quality criteria

26

The learned descriptions are represented in the form of a set of decision rules expressed in an

attributional logic calculus called variable-valued logic 1 or VLl (Michalski 1973) A

distinctive feature of this representation is that it employs in addition to standard logic operators

the internal disjunction operator (a disjunction of values of the same attribute in a condition) and

the range operator (to express conditions involving a range of discrete or continuous values)

These operators help to simplify rules involving multi valued discrete attributes the second

operator is also used for creating logical expressions involving continuous attributes

AQ15 can generate decision rules that represent either characteristic or discriminant concept

descriptions depending on the settings of its parameters (Michalski 1983) A characteristic

description states properties that are true for all objects in the concept The simplest

characteristic concept description is in the form of a single conjunctive rule (in general it can be

a set of such rules) The most desirable is the maximal characteristic description that is a rule

with the longest condition part ie stating as many common properties of objects of the given

class as can be determined A discriminant description states properties that discriminate a given

concept from a fixed set of other concepts The most desirable is the minimal discriminant

descriptions that is a rule with the slwnest condition part For example to distinguish a given

set of tables from a set of chairs one may only need to indicate that tables have large flat top

A characteristic description of the tables would include also properties such as have four legs

have no back have four corners etc Discriminant descriptions are usually much shorter than

characteristic descriptions

Another option provided in AQ15 controls the relationship among the generated descriptions

(rulesets or covers) of different decision classes In the IC (Intersecting Covers) mode

rulesets of different classes may logically intersect over areas of the description space in which

there are no training examples In the DC (Disjoint Covers) mode descriptions of different

classes are logically disjoint The DC mode descriptions are usually more complex both in the

number of rules and the number of conditions There is also a DL mode (a Decision List mode

27

also called VL modeM-for variable-valued logic mode) in which the program generates rule sets

that are linearly ordered To assign a decision to an example using such rulesets the program

evaluates them in order If ruleset is satisfied by the example then the decision is made

otherwise the program proceeds to the evaluation of the ruleset +1 In IC and DC modes

rule sets can be evaluated in any order

Alternatively the system can use rules from the AQ17-DCI program for Data-driven

Constructive Induction AQ17-DCI differs from AQ15 mainly in that it contains a module for

generating additional attributes These attributes are various logical or mathematical

combinations the original attributes The program generates a large number of potential new

attributes and seleets from them those most promising based on an attribute quality criterion

To illustrate the format of rules generated by AQ15 (or AQ17-DCI) an exemplary ruleset is

shown in Figure 3-2 The ruleset (that can be re-represented as a disjunctive normal form

expression) describes a voting record of Democratic Representatives in the US Congress

Each rule is a conjunction of elementary conditions Each condition expresses a simple relational

statement For example the condition [State = northeast v northwest] states that the attribute

State (of the Representative) should take the value northeast or northwest to satisfy the

condition

Rl [Gas_concban = yes] amp [Soc_see_cut = no v not registered] R2 [Draft = Yes v not registered]amp[AlaslaLparks = yes v not registered] amp

[Food_stamp_cap = no] amp [State = northeast v northwest] R3 [Chrysler =yes v not registered] amp [Income =low] R4 [Education =yes] amp [Occupation = yes]

Figure 3-2 A ruleset generated by AQ15 for the concept Voting pattern of Democratic Representatives

The above rules were generated from examples of the voting records For illustration below is

an example of a voting record by a Democratic representative

Draft registration=no Ban of aid to Nicaragua=no Cut expenditure on mx missiles=yes Federal subsidy to nuclear power stations=yes Subsidy to national parks

28

in Alaska=yes Fair housing bill=yes Limit on Pac contributions=yes Limit onood stamp program=no Federal help to education=no StateFrom=north east State Population=large Occupation =unknown Cut in social security spending=no Federal help to Chrysler corp=not registered

By expressing elementary statements in the example as conditions and linking conditions by

conjunction the examples can be re-expressed as decision rules Thus decision rules and

examples formally differ only in the degree of generality

33 Generating Decision Structurestrees from Decision Rules

This section describes the AQDT-1 system for learning decision structures (trees) from decision

rules (Imam amp Michalski 1993ab) Also a description of the AQDT-2 method for learning taskshy

oriented decision structure from decision rules is included and fmally the methodology is

illustrated by two examples

Methods for learning decision trees from examples have been very popular in machine learning

due to their simplicity Decision trees built this way can be quite efficient as long as they are

used in decision-making situations for which they are optimized and these situations remain

relatively stable Problems arise when these situations significantly change and the assumptions

under which the tree was built do not hold anymore For example in some situations it may be

difficult to determine the value of the attribute assigned to some node One would like to avoid

measuring this attribute and still be able to classify the example if this is potentially possible

(Quinlan 1990) If the cost of measuring various attributes changes it is desirable to restructure

the tree so that the inexpensive attributes are evaluated f11st A tree restructuring is also

desirable if there is a significant change in the frequency of occurrence of examples from

different classes A restructuring of a decision tree to suit the above requirements is however

difficult to do The reason for this is that decision trees are a form of decision structure

representation and imposes constraints on the evaluation order of the attributes that are not

logically necessary

29

One problem in developing a method for generating decision structures from decision rules is to

design an attribute selection criterion that is based on the properties of the rules rather than of

the training examples A decision rule normally describes a number of possible examples Only

some of them are examples that have actually been observed ie training examples An attribute

selection criterion is needed to analyze the role of each attribute in the rules It cannot be based

on counting the numbers of training examples covered by each attribute-value and the

frequency of decision classes in the training examples as is done in learning decision trees from

examples because the training examples are assumed to be unavailable

Another problem in learning decision trees from decision rules stems from the fact that decision

rules constitute a more powerful knowledge representation than decision trees They can directly

represent a description in an arbitrary disjunctive normal form while decision trees can represent

directly only descriptions in the disjoint disjunctive normal form In such descriptions all

conjunctions are mutually logically disjoint Therefore when transforming a set of arbitrary

decision rules into a decision tree one faces an additional problem of handling logically

intersecting rules

The solution to both problems (attribute selection and logically intersected rules) in the AQDT-2

system is based on the earlier work by Michalski (1978) which introduced a general method for

generating decision trees from decision rules The method aimed at producing decision trees

with the minimum number of nodes or the minimum cost (where the cost was defined as the

total cost of classifying unknown examples given the cost of measuring individual attributes and

the expected probability distribution of examples of different decision classes) More

explanations are provided in the following section

331 Tbe AQDT-2attribute selection metbod

This section describes the AQDT -2 method for building a decision structure from decision rules

The method for building a single-parent decision structure is similar to that used in standard

30

methods of building a decision tree from examples The major difference is that it assigns tests

(attributes) to the nodes using criteria based on the properties of the decision rules (this includes

statistics about the examples covered by each rule in the case of learning rules from examples)

rather than statistics characterizing the frequency of training examples per decision classes per

attribute-values or per conjunctions of both Other differences are that the branches may be

assigned an internal disjunction of values (not only a single value as in a typical decision tree)

and leaves may be assigned a set of alternative decisions with probabilities Also the tests can be

attributes or names standing for logical or mathematical expressions that involve several

attributes or variables In the following we use the terms test and attribute interchangeably

(to distinguish between an attribute and a name standing for an expression the latter is called a

constructed attribute)

At each step the method chooses the test from an available set of tests that has the highest utility

(see below) for the given set of decision rules This test is assigned to the node The branches

stemming from this node are assigned test values or disjoint groups of values (in the form of

logical disjunction if such occur in the rules subsumed groups of values are removed) Each

branch is associated with a reduced set of rules determined by removing conditions in which the

selected attribute assumes value(s) assigned to this branch If all rules in the reduced ruleset

indicate the same decision class a leaf node is created and assigned this decision class The

process continues until all nodes are leaf nodes If it is not possible to reduce further the rule set

because some attribute is declared as unavailable (infmite cost) then the leaf is assigned a set of

candidate decisions with associated probabilities (see Sec 42)

The test (attribute) utility is a combination of one or more of the following elementary criteria 1)

cost which indicates the cost of using each attribute for making decision 2) disjointness which

captures the effectiveness of the test in discriminating among decision rules for different decision

classes 3) importance which determines the importance of a test in the rules 4) value

31

distribution which characterizes the distribution of the test importance over its of values and 5)

dominance which measures the test presence in the rules These criteria are dermed below

~ The cost of a test expresses the effort or cost needed to measure or apply the test

Djsjointness The disjointness of a test is defined as the sum of the class disjointness-the

disjointness of the test for each decision class Suppose decision classes are Cl C2 bull Cm and

decision rule sets for these classes have been determined Given test A let V 1 V 2 bull V m denote

sets of the values (outcomes) of A that are present in rule sets for classes C 1 C2bullbull Cm

respectively If a ruleset for some class say Ct contains a rule that does not involve test A than

Vt is the set of all possible values of A (the domain of A)

Definition 3 -1 The degree ofclass disjointness D(A Ci ) of test A for the ruleset ofclass Ci is

the sum of the degrees of disjointness D(A Cit Cj) between the ruleset for Ci and rulesets for

Cj j=l 2 m j J L The degree of disjointness between the ruleset for Ci and the ruleset for Cj is

dermed by

if Vi Vj ifVj gtVj (3-1) ifVinVj -tljgt amp -tVi amp -tVj ifVinVj=1jgt

where cjgt denotes the empty set Note that exchanging the second and third conditions of equation

(3-1) may seem to be an improved criteria However it does not clearly distinguish between both

cases (Le for both situations the disjointness will be similar) The current equation is better

because it gives higher scores to attributes that classify different subsets of the two decision

classes than attributes that classify only a subset of one decision class

Definition 3-2 The disjointness of the test A for evaluating a given set of decision rules is the

sum of the degrees of class disjointness of each decision class

32

m m Disjointness(A) = L D(A cn where D(A cn = L D(A Ci Cj) (3-2)

i=l i=l i~j

The disjointness of a test ranges from 0 when the test values in rulesets of different classes are

all the same to 3m(m-l) when every rule set of a given class contains a different set of the test

values If two tests have the same disjointness value the attribute to be selected is the one with

less number of values

Definition The A vera~ Number of Tests (ANT) required to make decision from a decision tree

is dermed as the average number of tests (attributes) to be examined from the root of the tree to

any leaf node in order to reach a decision

Definition A decision structure is a one-node-per-level decision structure if at each level there is

only one node and zero or more leaves

Such decision structure can be generated by combining together all branches that associated with

each one a set of decision rules belongs to more than one decision class

Theorem 1 Consider learning one-node-level decision structure for a database with two

decision classes The disjointness criterion ranks first attributes that add minimum number of

tests to the decision tree

Proof Consider all possible distributions of an attributes values within two decision classes Ci

and Cj There are three cases 1) subset (same as super set) 2) non-empty intersection but not

subset 3) no intersection Figure 3-3 shows all possible distributions (the case of having the

same set of values in both classes is trivial one) Consider that branches leading to one subset

with the same decision class are combined into one branch In the first case there will be two

branches only The first leads to a leaf node and the other leads to an intermediate node where

another attribute to be selected The minimum ANT in this case is 53 In the second case three

branches should be created Two branches leads to a leaf node where all values at each branch

belong to only one and different decision class The third branch leads to an intermediate node

33

where another attribute should be selected furtur classifies the decision classes The minimum

ANT in this case is 64 In the third case only two branches will be generated where each leads

to leaf node with different decision class In this case the minimum ANT is 1

Figure 3-4 shows decision trees equivalent to selecting the corresponding attribute Note that in

case of having more than one attribute-value at branches lead to leaves belonging to one decision

class they will be combined into one branch in the decision structure The symbol 1 means

that an attribute is needed to classify the two decision classes In such cases there will be at least

two additional paths

D(A Ci) = 0 D(A CJ) = 1 D(A Ci) =2 D(A Cj) = 2 D(A Ci) =3 D(A Cj) =3

Figure 3-3 Yen Diagrams of Possible combinations of attribute values in two decision classes

The average number of tests required for making a decision in each possible case is determined

in Figure 3-4 It is clear that in the case of two decision classes the disjointness criterion ranks

highly attributes that reduces the average number of tests required for decision-making The

theorem can be proved in the general case

i ~ i Cj C1middot bull Ci Cj

ANT=32middotANT=53 ANT=l

1 means at least one attribute is needed to complete the decision tree Figure 3-4 Decision trees corresponding the Yen diagrams inFigure 3-3

34

Theorem 2 The attribute with the highest non~zero disjointness is the best attribute to classify

the decision classes

Proof Suppose that the number of decision classes is n Assume also that there are two attributes

A and B where D(A) lt D(B) Since the disjointness criterion considers the mutual relationship

between any two decision classes this means that there are more decision classes where D(A

Ci) lt D(B Ci) than those where D(A Ci) gt D(B Ci) (ignoring classes with equal disjointness

for bCth attributes) Hence there are more decision classes where D(A Ci Cj) lt D(B Ci Cj)

than those where D(A Ci Cj) gt D(B Cit Cj) Let us frrst prove that if D(A Ci Cj) lt D(B Cit

Cj) then B classifies the decision classes better than A

For each pair of decision classes Ci and Cj the possible values of the disjointness of any attribute

are 0 14 or 6 For all positive values ofD(B)= 14 or 6 it is clear that attribute B should have

smaller ANT than attribute A which has lower disjointness This means that if D(A Cit Cj) lt

D(B Cit Cj) then attribute B can classify both decision classes better than attribute A

Similarly B is better for classifying more pairs of decision classes than A This implies that B is

a better classifier than A

Importance The second elementary criterion the importance of a test is based on the

importance score (IS) introduced in (Imam Michalski amp Kerschberg 1993) In the obtained

rules each test is assigned a score that represents the total number of training examples that

are covered by the rules involving this test Decision rules learned by an AQ learning program

are accompanied with information on their strength Rule strength is characterized by t-weight

and u~weight The t-weight (total-weight) of a rule for some class is the number of examples of

that class covered by the rule The importance score of a test is the aggregation of the totalshy

weights of all rules that contain that test in their condition part Given a set of decision rules for

m decision classes Cl Cm and involving n tests AlbullAn and the number of rules associated

with class Ci is denoted by tlrit The importance score is defmed as follow

35

Definition 3 -3 The importance score IS(Aj) of the test A j is detennined by

m IS(Aj)= L IS(Aj Ci) (3-31)

i=l

where ri IS(Aj Ci) = L Rik(Aj) (3-32)

k=l

and Rik the weight of a test Aj in the rule Rk of class Ci is given by

t - weight if A j belongs to rule Rik Rik (Amiddot) = (3-4)J 0 otherwise

where i=I n ik=I ri j=I m

The importance score method has been separately compared as a feature selection method with

a genetic algorithm-based method (Imam amp Vafaie 1994) The importance score method

produced an equal or higher accuracy on three real-world problems than those reported by the

GA method while selecting fewer attributes In addition the IS method was significantly faster

than the GA

yalue distribution The third elementary criterion value distribution concerns the number of

legal values of tests Given two tests with equal importance scores this criterion prefers the test

with the smaller number of legal values Experiments have shown that this criterion is especially

useful when using discriminant decision rules

Definition 34 A value distribution VD(Aj) of a test A j is defined by

VD(Aj) = IS(Aj) I Vj (3-5)

where v is the number of legal values of Aj

Dominance The fourth elementary criterion dominance prefers tests that appear in large

numbers of rules as this indicates their high relevance for discriminating among the rulesets of

given decision classes Since some conditions in the rules have values linked by internal

disjunction counting such rules directly would not reflect properly their relevance Therefore

36

for computing the dominance the rules are counted as if they were converted to rules that do not

have internal disjunction Such a conversion is done by multiplying out the condition parts of

the rules containing internal disjunction For example the condition part [x3=1 v 3]amp[x4=1] is

multiplied out to two rules with condition parts [x3=1]amp[x4=1] and [x3=3]amp[x4=I]

The above criteria are combined into one general test measure using the lexicographic

evaluation functional with tolerances (LEF) (Michalski 1973) LEF is a list of some or all of

the above elementary criteria each associated with a Ittolerance threshold It in percentage The

criteria are applied to tests in the order defined by LEF A test passes to the next criterion only if

it scores on the previous criterion within the range defined by the tolerance (from the top value)

The default LEF is

ltCost d Disjointness t2 Importance t3 Value distr t4 Dominance 0gt (3-6)

where d a t3 t4 and t5 are tolerance thresholds (in percentages) their default values are O

The default value of the cost of each test is 1

The above LEF ranks attributes this way First attributes are evaluated on the basis of their cost

If two or more attributes have the same cost they ranked by their degree of disjointness If two

or more attributes still share the same top score or their scores differ less than the assumed

tolerance threshold t2 the method evaluates these attributes using the second (Importance)

criterion If again two or more attributes share the same top score or their scores differ less than

the tolerance threshold t3 then the third criterion normalized IS is used then similarly the

fourth criterion (dominance) If there is still a tie the method selects the best attribute

randomly

If there is a non-uniform frequency distribution of examples of different classes then the

selection criterion uses a modified definition of the disjointness Namely the previously defmed

37

disjointness for each class is multiplied by the frequency of the class occurrence The class

occurrence is the expected number of future examples that are to be classified to a given class

m

Disjointness(A) = L D(A CO Frq(Ci) (3-7) i=l

where Frq(Cn is the expected frequency of examples of class Cit and it is assumed to be given

by the user The attribute ranking criteria in this case is defined by the LEF

ltCost d Disjointness t2 Importance 13 Normalized-IS 14 Dominance 15gt (3-8)

where the Cost denotes the evaluation cost of an attribute and is to be minimized while other

elementary criteria are treated the same way as in the default version

332 The AQDT-2 algorithm

The AQDT-2 algorithm constructs a decision structure from decision rules by recursively

selecting at each step the best test according to the ranking criteria described above and

assigning it to the new node The process stops when the algorithm creates terminal branches that

are assigned decision classes To facilitate such a process the system creates a special data

structure for each concept description (ruleset) This structure has fields such as the number of

rules the number of decision classes and the number of attributes present in the rules A set of

pointers connect this data structure to set of data structures each represents one decision class

The decision class structure contains fields with information on the number of rules belong to

that class the frequency of the decision class etc It is also connected to a set of data structures

representing the decision rules within each decision class The system creates independently a set

of data structures each is corresponding to one attribute Each attribute description contains the

attributes name domain type the number of legal values a list of the values the number of

rules that contain that attribute and values of that attribute for each rule The attributes are

arranged in an array in a lexicographic order flISt in the descending order of the number of rules

38

that contain that attribute and second in the ascending order of the number of the attributes

legal values

The system can work in two modes In the standard mode the system generates standard

decision trees in which each branch has a specific attribute-value assigned In the compact

mode the system builds a decision structure that may contain

A) or branches Le branches assigned an internal disjunction of attribute-values whenever it

leads to simpler structures For example if a node assigned attribute A has a branch marked by

values 1 v 2 then the control passes along this branch whenever A takes value 1 or 2 The

program creates or branches on the basis of the analysis of the value sets Vi while computing

the degree of attribute disjointness

B) nodes that are assigned derived attributes that is attributes that are certain logical or

mathematical combinations of the original attributes To produce decision structures with

derived attributes the input decision rules are generated by AQ17 (rather than AQI5) The

AQ17 rules may contain conditions involving attributes constructed by the program rather than

those originally given

To generate decision structures from rules the AQDT-2 method prefers either characteristic or

discriminant disjoint rule descriptions (given by an expert or learned by a system) Disjoint rules

are more suitable for building decision structures Assume that the description of each class is in

the form of a ruleset and that this set is the initial ruleset context The AQDT algorithm is

The AQDT2 Algorithm

Step 1 Evaluate each attribute occurring in the ruleset context using the LEF attribute ranking

measure Select the highest ranked attribute Let A represents this highest-ranked

attribute

Step 2 Create a node of the tree (initially the root afterwards a node attached to a branch) and

assign to it the attribute A In standard mode create as many branches from the node as

39

there are legal values of the attribute A and assign these values to the branches In

compact mode (decision structures) create as many branches as there are disjoint value

sets of this attribute in the decision rules and assign these sets to the branches

Step 3 For each branch associate with it a group of rules from the ruleset context that contain a

condition satisfied by the value(s) assigned to this branch For example if a branch is

assigned values i of attribute A then associate with it all rules containing condition [A=

i v ] If a branch is assigned values i v j then associate with it all rules containing

condition [A= i v j v ] Remove from the rules these conditions If there are rules in

the ruleset context that do not contain attribute A add these rules to all rule groups

associated with the branches stemming from the node assigned attribute A (This step is

justified by the consensus law [x=1] == [x=1] amp [y=a] v [x=1] amp [y=b] assuming

that a and b are the only legal values of y) All rules associated with the given branch

constitute a ruleset context for this branch

Step 4 If all the rules in a ruleset context for some branch belong to the same class create a leaf

node and assign to it that class If all branches of the trees have leaf nodes stop

Otherwise repeat steps 1 to 4 for each branch that has no leaf

To select an attribute to be a node in the decision tree (step 1 and 2 of the algorithm) the

algorithm performs two independent iterations In the first iteration it parses through all decision

rules and determine information about each attributes This information includes the importance

score of each attribute the number of rules containing a given attribute the disjoint value sets of

each attribute and the attribute values used in describing each decision class The second

iteration is only performed if the disjointness criterion is ranked first in the LEF function The

second iteration evaluates the attributes disjointness for each decision class against the other

decision classes

40

To determine the complexity of this process suppose that the maximum number of conditions

fonned by a single attribute value in one rule is s (where s ~ n the number of attributes) and r is

the total number of decision rules (in all decision classes)

m

r=LRt (m is the number of decision classes) 1=1

where Ri is the number of rules in decision class Ci bull The complexity of the frrst iteration can be

determined as

Cmpx(Itcl) =O(r s)

In the second iteration the disjointness is calculated between the decision classes and all

attributes The complexity of the second iteration can be given by

Cmpx(Itc2) = O(n m)

Assume that at each node 1 is the maximum of the number of decision classes to be classified at

this node and the number of rules associated with this node

l=max mr (3-9)

Since these two iteration are independent of each other the complexity of the AQDT algorithm

for building one node in the decision tree say Node Complexity NC(AQDT) is given by

NC(AQDT) = 0(1 n)

Usually I equals to the number of rules associated with the given node Thus the AQDT

complexity for building one node is a function of the number of attributes multiplied by the

number of rules associated with this node

At each level of the decision tree these two iterations are repeated for each non-leaf node The

Level Complexity of the AQDT algorithm LC(AQDT) can be given by

LC(AQDT) lt 0(1 n)

Which is less than the complexity of generating the root of the decision tree To explain this

consider the maximum possible number of non-leaf nodes at one level is half the number of the

initial decision rules (integer of rll) Figure 3-5-a shows an example of such situation where at

41

the lowest level each node classifies only two rules each belongs to different decision class This

decision tree could be a multimiddotvalues tree However the number of non-leaf nodes at each level

will be twice or more than the number of nonmiddot leaf nodes at the previous level Considering the

level complexity of the AQDT algorithm equal to (1 s 0) where 0 is the number of non-leaf

nodes at the given level In such cases it is either (1 0 S r) or (1 s lt r) In Figure 3middot5middota the

complexity of the AQDT algorithm at any lower level is given by

LC(AQDT) =0(2 s r(2) lt O(n I) =NC(AQDT)

a) per one level b) per one path

Figure 3 -5 Decision trees showing the maximum number of none leaf nodes

Note also that after selecting an attribute to be the root of the decision structure this attribute

and all conditions containing that attribute are removed from the data structure of the algorithm

Also if a leaf node is generated all rules belonging to the corresponding branch will not be

tested again

Since the disjointness criterion selects the attribute which minimizes the average number of tests

ANT it is true that the AQDT algorithm generates decision trees with the least number of levels

The number of levels per a decision tree is supposed to be less than or equal to the minimum of

both the number of attributes and the number of rules Consider k as the number of levels in a

given decision tree

k S min mr (3-10)

There are two cases represent the most complex situations Figure 3-5-a and 3-5-b In the fIrst

case where the decision rules were divided evenly the number of levels will be a function of the

42

logarithm of the number of rules In such case the complexity of the AQDT algorithm for

generating a decision tree from a set of decision rules is given by

Complexity(AQDT) = 0(1 n log r) (3-11)

The other situation is when the generated decision tree has the maximum number of levels The

maximum possible number of levels per a decision tree equals to one less than the number of

decision rules Figure 3-5-b Using the disjointness criterion it is not likely to get such a decision

tree because it has the maximum average number of test (ANT) that can be determined from the

same set of nodes and leaves However such a decision tree can be generated if the number of

decision classes is one less thanthe number of attributes In such case any disjQint decision rules

should have a maximum length that is less or equal to the floor of the logarithm of the number of

attributes Thus the level complexity of this decision tree is estimated as

LC(AQDT) =0(1 log n)

The maximum number of levels in such decision tree is k-l Thus the complexity of the AQDT

algorithm in such cases is given by

Complexity(AQDT) = 0(1 k log n) (3-12)

Since k is the minimum of n and r then n is the maximum of k and n Also since in (3-12)

r=n+1 then log r is greater than log n From 3-10 3-11 and 3-12 the complexity of the AQDT

algorithm is determined by

Cmplx(AQDT) = OCr k log 1) (3-13)

333 An example illustrating the algorithm

The following simple example illustrates how AQDT is used in selecting the optimal set of

testing resources for testing a software Suppose there are three tools for testing a software 1)

modeling (Tl) 2) checklist (T2) and 3) par_simul (T3) Also assume that there are four

different factors that affect the selection of any tool 1) the cost of using the tool (xl) 2) the

metrics that support the tool (x2) 3) the best phase for applying the tool (x3) and 4) type of tool

43

(automated semi-automated or manual) (x4) Table 3-1 shows these attributes and their possible

values

Suppose that the domain expert provided a set of rules to be used in testing a software Figure 3shy

6 shows a sample of these rules in AQ15c formats

Table 3-1 The available tools and the factors that affect the prclCes

Tl lt= [xl=2] amp [x2=2] v [xl=3] amp [x3=1 v 3] amp [x4=1] T2 lt= [xl=l v 2] amp [x2=3 v 4] v [xl=3] amp [x3=1 v 2] amp [x4=2] T3 lt= [xl=l] amp [x2=1] v [xl=4] amp [x3=2 v 3] amp [x4=3]

Figure 3-6 Decision rules for selecting the best tool for testing software

These rules can be interpreted as

Rule 1 Use the fltst tool for testing if you need average cost and the tool is

supported by the requirement metric

Rule 2 Use the fIrst tool for testing if you can afford high cost for testing either

in the requirement or the analysis phases and you need an automated tool

Rule 3 Use the second tool for testing if the cost limit is low or average and the

tool is supported by the system usage metric or the intractability metric

Rule 4 Use the second tool for testing if you can afford high cost for testing

either in the requirement or the design phases and you need a manual tool

Rule 5 Use the third tool for testing if the you are limited to low cost and the

tool is supported by the error rate metric

Rule 6 Use the third tool for testing if you can afford very-high cost for testing

either in the requirement or the system usage phases and you need a semishy

automated tool

44

Table 3-2 presents information on these rules and the disjointness values for all attributes For

each class the row marked Values lists values occurring in the ruleset for this class For

evaluating the disjointness of an attribute say A each rule in the ruleset above that does not

contain attribute A is characterized as having an additional condition [A= a vb ] where a b

are all legal values of A

The row Class disjointness specifies the class disjointness for each attribute The attribute xl

has the highest disjointness (11) and is assigned to the root of the tree For simplicity assume

the tolerances for each elementary criterion equal O

From the rules in Figure 3-6 we can also determine disjoint groupings of attribute-values used

for the compact mode of the algorithm This done as follows 1) determine for each attribute the

sets of values that the attribute takes in individual decision rules and remove those value sets

that subsume other value sets The remaining value sets are assigned to branches stemming from

the node marked by the given attribute For example x 1 has the following value sets in the

individual decision rules 2 3 I 2 I and 4 (Figure 3-6) Value set I 2 is

removed as it subsumes 2 and I In this case branches are assigned individual values of the

domain of x1 For attribute x2 the value sets are I 2 34 and I 2 3 4 In this case

branches are assigned value sets I 2 and 34

45

Attribute Xl ranks highest (as it is the highest disjointness) and is assigned to the root of the

tree Four branches are created each one corresponding to one of XIs possible values Since all

rules containing [x1=4] belong to class TI the branch marked by 4 is ended by a leafTI Rules

containing other values of X1 belong to more than one class This process is repeated for each

subset of rules until the decision tree is completed Figure 3-7 shows a decision structure learned

by AQDT-2 from the rules in Figure 3-6 The decision structure in Figure 3-7 can be used in

making decisions on which tools can be used for testing a given software

xl

Complexity No of nodes 4 No of leaves 7

Figure 3-7 A decision structure learned for classifying so tware testing tools

Figure 3-8-a shows the diagrammatic visualization of the decision rules and Figure 3-8-b shows

the visualization of the derived decision tree Each diagram in Figure 3-8 consists of cells

representing one combination of attribute-values Attributes and their legal values are shown on

scales surrounding the diagram (eg the horizontal scale for xl shows values 1 2 3 and 4)

Rules are represented by collections of cells in the intersection of the rows and columns

corresponding to the conditions in the rules

The shaded areas correspond to decision rules Rules of the same class have the same type of

shading Empty cells correspond to combination of attribute-values not assigned to any class For

illustration collections of cells corresponding to some of the initial rules are marked R 11 R 21

R31 and R32 Rll denotes the IlfSt rule of Class T1 ie [x1=2] amp [x2=2] R21 denotes the

IlfSt rule of class T2 ie [xl= 1 v 2] amp [x2= 3 v 4] R31 the first rule of class TI ie [x1=1]

46

amp [x2 =1] and denotes R32 denotes the second rule of class T3 Le [xl=4] amp [x3=2 v 3] amp

[x4=3]

a) Decision rules b) Derived decision tree

Comparing diagrams in Figure 3-8-a and 3-8-b AQDT-2 decision tree represents a slightly more

general description of Concepts Tl TI and T3 than the original rules Let us assume that it is

very costly to know which metrics suppon the required tools In other words suppose that we

would like to select the best tools independently of the metric they suppon (this is indicated to

AQDT-l by assigning very high cost to attribute x2) The algorithm again assigns attribute xl to

the root of the tree When xl takes value 1 or 2 it is impossible to assign a specific decision

without measuring attribute x2 For the value 1 of xl the recommended tool can be either TI or

T3 (see the diagram in Figure 3-9-a) and for the value 2 of xl the recommended tool can be

either Tl or T3 However for the value 3 of xl one can make a specific decision after

measuring attribute x4 For the value 1 of x4 tool is Tl and for the value 2 of x4 the

recommended tool is TI Figure 3-9-b shows another decision tree where the type of the tool

was ignored from the data Those decision trees are called indeterminate because some of their

leaves are assigned a disjunction of two or more class names

47

xl

xl

1 12

a) Ignoring the supporting metric b) Ignoring the type of the tooL Figure 3 -9 Decision trees learned ignoring the support metric and the type of the testing tool

It is clear that the given set of rules depend highly on amount of money can be spent Let us now

suppose that the cost attribute xl which was determined as the highest ranked attribute cannot

be measured The algorithm selected x4 to be the root of the new decision tree After continuing

the algorithm the tree in Figure 3-10 is obtainedThe nodes that are assigned one class indicate

situations in which it is possible to make a specific decision without knowing the value of

attribute xl

x4

Figure 3-10 A decision tree learned without the cost attribute

34 Tailoring Decision Structures to the Decision-making situation

Decision structures are among the simplest structures for organizing a decision-making process

A decision structure specifies explicitly the order in which attributes of an object or situation

need to be evaluated in the process of determining a decision A standard way to generate a

48

decision structure is to learn it from examples of decisions Such a process usually aims at

obtaining a decision structure that has the highest prediction accuracy that is the highest rate of

assigning correct decisions to given situations There can usually be a large number of logically

equivalent decision structures (Michalski 1990) As such they may have the same predictive

accuracy but differ in the way they organize the decision process and thus may differ in the cost

of arriving at a decision To minimize the average decision cost one needs to take into

consideration the distribution of the costs of attribute evaluation and the frequency of different

decisions This report presents an approach to building such Task-oriented decision structures

which advocates that they are built not from examples but rather from decision rules Decision

rules are learned from examples using the AQ15 or AQ17 inductive learning program or are

specified by an expert An efficient algorithm and a new system AQDT-2 that transforms

decision rules to task-oriented decision structures The system is illustrated by applying it to the

problem of learning decision structures in the area of construction engineering (for determining

the best wind bracing for tall buildings) In the experiments AQDT-2 outperformed all other

programs applied to the same data

Decision-making situations can vary in several respects In some situations complete

information about a data item is available (ie values of all attributes are specified) in others the

information may be incomplete To reflect such differences the user specifies a set of parameters

that control the process of creating a decision structure from decision rules AQDT-2 provides

several features for handling different decision-making problems 1) generating a decision

structure that tries to avoid unavailable or costly attributes (tests) 2) generating unknown

leaves in situations where there is insufficient information for generating a complete decision

structure 3) providing the user with the most likely decision when performing a required test is

impossible 4) providing alternative decisions with an estimate of the likelihood of their

correctness when the needed information can not be provided

49

341 Learning Costmiddot Dependent Decision Structures

As described in Sec 2 the LEF criterion enables the system to take into consideration the cost of

measuring tests (attributes) in developing a decision structure In the default setting of LEF the

tests cost is the [lIst criterion and its tolerance is O This means that only the least expensive

attributes pass to the next step of evaluation involving other elementary criteria If an attribute

has high cost or is impossible to measure (has an infinite cost) the LEF chooses another

cheaper attribute ifpossible

342 Assigning Decision Under Insufficient Information

In decision-making situations in which one or more attributes cannot be measured the system

may not be able to assign a definite decision for some cases If no more information can be

obtained but a decision has to be made it is useful to know the probability distribution for

different candidate decisions (Smyth Goodman amp Higgins 1990) The most probable decision is

then chosen The probability distribution can be estimated from the class frequency at the given

node Let us consider a node in the structure connected to the root by the sequence bl b2 bkof

attribute-values Let P(Ci I bl b2 bk) denote the conditional probability of class Ci i=I2 m

at that node given that an example to be classified has attribute-values assigned to branches bl

b2 bull bk in the decision structure Using Bayesian formula we have

P(Ci I bl~ bull bk) = P(Ci) P(bl bk I Ci) P(bl bk) (3-9)

where P(Ci) the apriori probability of class Ci P(bl bk I Ci) is the probability that the

example has attribute-values blbull bk given that it represents decision class Ci To approximate

these probabilities let us suppose that Wi is the number of training examples of Class Ci that

passed the tests leading to this node and twi - the total number of training examples of Class

Ci Let us use these numbers to estimate the probabilities in (9) Assuming that the apriori class

probabilities correspond to the frequency of training examples from different classes we have

50

m P(Ci) = twi IL twj (3-10)

j=l

P(b1 bkl Ci) = Wi I twi (3-11)

m m P(b1 bk) =L Wj IL twj (3-12)

j=l j=1

By substituting (1O) (11) and (12) in (9) we obtain

m P(Ci I b1 bk) = wilL Wj (3-13)

j=l

A related method for handling the problem of unavailability of an attribute is described by

Quinlan (1990) Quinlans method however assigns probabilities to the most probable decision

at the node associated with such an attribute The method does not restructure the decision tree

appropriately to fit the given decision-making situation (in this case to avoid measuring xl)

AQDT -2 assigns probabilities o~ly after it first created a decision structure that suits the given

decision-making si~ation An example is presented in section 42

343 Coping with noise in training data

The proposed methodology can be easily extended to handle the problem of learning from noisy

training data This is done based on ideas of rule truncation described in (Niblett amp Bratko 1986

Bergadanoet al 1992 Cestnik amp Karalic 1991) When noise is expected in the training data

the decision rules with t-weight below a certain threshold (reflecting the expected noise level) are

removed The rule truncation method seems to result in more accurate decision structures than

decision tree pruning method because truncation decisions are solely based on the importance of

the given rule or condition for the decision-making regardless of their evaluation order (unlike

the decision tree pruning which can only prunes attributes within a subtree and thus cannot

freely chose attributes to prune) Examples are presented in section 4

~J I yarJ I JJ I ~

U-t-oJ ~II ~) ~ oWAIl J

Dedication

To my mother my brothers and my sister

TABLE OF CONTENTS

TI1LE Page

ABS1RAcr II 1

CHAPrER 1

IN1RODUCI10N II 3

11 Motivation and Overview 3

12 The Problem Statement 6

CHAPrER2

REIAfED RESEARCFI 7

21 Learning Decision Trees from Decision Diagrams 7

22 Learning Decision Trees from Examples 10

221 Building Decision Trees Using Information-based Criteria 11

222 Building Decision Trees Using Statistics-based Criteria 16

223 Analysis of Attribute Selection Criteria 18

23 Learning Decision Structures 19

CHAPrER3

DESCRIPTION OF TIlE APPROACFI 23

31 General Methodology 23

32 A Brief Description of the AQ15 and AQ17 RuleLeaming Systems 25

33 Generating Decision Structures From Decision Rules 28

331 The AQDT-2 attribute selection method 29

332 The AQDT-2 algoridlm 37

vi

333 An example illustrating the algorithm 42

34 Tailoring Decision Structure to a decision-making situation 47

341 Learning Cost-Dependent Decision Structures 49

342 Assigning Decision Under Insufficient Infonnation 49

343 Coping with noise in training data 50

35 Analysis of the AQDT-2 Attribute Selection Criteria 51

36 Decision Structures vs Decision Trees 53

CHAPTER 4

EMPRICAL ANALYSIS AND COMPARATIVE STUDY OF TIlE MElHOD 58

41 Description of the Experimental Analysis 59

42 Experiments With Average Size Complex and Noise-Free Problems

Wind Brac-ings 60

43 Experiments With Small Size Simple and Noise-Free Problems

e MONK-I 69

44 Experiments With Small Size Complex and Noise-Free Problems

MONK-2 76

45 Experiments With Small Size Simple and Noisy Problems

MONK-3 79

46 Experiments With Large Size Complex and Noise-Free Problems

Diagnosing Breast Cancer 83

47 Experiments With Large Size Complex and Noisy Problems

Musm()()m classifications 84

48 Experiments With Small Size Structured and Noise-Free Problems

East-West Trains 85

49 Experiments With Small Size Simple and Noisy Problems

vii

Congressional Voting Records (1984) 87

410 Analysis of the Results 88

ClIAPfER 5 CONCIUSIONS 95

51 Summary 95

52 Contributions 96

REFERENCES 98

VITA 102

viii

LIST OF TABLES

No TITLE Page

2-1 An example of a decision table 9

2-2 A set of training examples used to illustrate the C45 system 15

2-3 The frequency of different attributes values to different decision classes 17

2-4 The expected values of the frequency of examples in Table 2-3 17

2-5 Attribute selection criteria and their basic evaluation measure 17

2-6 The contingency tables of Mingers example 18

2-7 Mingers results for determining the goodness of split 19

2-8 Mingers results for comaring the total accuracy and size of decision trees

provided by different attribute selection criteria from four problems 19

2-9 Comparing the AQDT approach with EDAG and HOODG approaches 22

3-1 The available tools and the factors that affect the process of testing a software 43

3-2 Calculating the disjointness of each attribute 44

3-3 Evaluation of the AQDT-2 criteria on the Quinlans example in Table 2-2 51

3-4 The data used in Mingers f11st experiments 52

3-5 The performance of AQDT-2 criteria (compare with the other criteria in Table 2-6) 52

3-6 The possible ranking domains and using condition of AQDT-2 criteria 53

3-7 Comparison between Decision Structures and Decision Tree 54

4-1 Evaluation of AQDT-2 attribute selection criteria for the wind bracing problem 62

4-2 The predictive accuracy of AQ15c and AQDT-2 for the wind bracing problem 67

4-3 Evaluation of AQDT-2 attribute selection criteria for the MONK-l problem 71

4-4 The predictive accuracy of AQ15c and AQDT-2 for the MONK-l problem 73

ix

4-5 The predictive accuracy of AQ15c and AQDT-2 for the MONK-2 problem 77

4-6 The predictive accuracy of AQ15c and AQDT-2 for the MONK-3 problem 81

4-7 The set of attributes and their values used in the trains problem 86

4-8 A tabular snmmary of the predictive accuracy of decision trees obtained

by AQDT -2 and C45 for the congressional voting data 88

4-9 Summary of the best parameter settings for the first subfunction of the approach

with different data characteristics 89

4-10 Summary of the performance of AQDT-2 and C45 on different problems 90

x

LIST OF FIGURES

No TITLE Page

2-1 An example to illustrate how attributes break rules 8

2-2 A decision tree learned from the decision table in Table 2-1 10

2-3 A decision tree learned using the gain criterion for selecting attributes 15

2-4 Decision rules and their Exception Directed Acyclic Graph (EDAG) 21

3-1 Architecture of the AQDT approach 24

3-2 A ruleset generated by AQ15 for the concept Voting pattern of

Democratic Representatives 27

3-3 Ven Diagrams ofPossible combinations of attribute values in two decision classes 33

3-4 Decision trees corresponding the Ven diagrams in Figure 3-3 33

3-5 Decision trees showing the maximum number of none leaf nodes 41

3-6 Decision rules for selecting the best tool for testing a software 43

3-7 A decision structure learned for classifying software testing tools 45

3-8 A diagrammatic visualization of the decision rules and the derived decision tree 46

3-9 Decision trees learned ignoring the support metric and the type of the testing tool 47

3-10 A decision tree learned without the cost attribute 47

3-11 Decision structures learned by AQDT-2 using different criteria 55

3-12 The Imams example Example where learning decision structures (trees)

from rules is better than learning them from examples 56

3-13 An example where decision rules are simpler than decision trees 57

4-1 Design of a complete experiment 59

Xl

4-2 Decision rules determined by AQ15c from the wind bracing data 61

4-3 A decision tree learned by C45 for the wind bracing data 63

4-4 A decision structure learned from AQ15c wind bracing rules 64

4-5 A decision structure that does not contain attribute xl 64

4-6 A decision structure without Xl with candidate decisions assigned to leaves 65

4-7 A decision structure determined from rules in Figure 4-4 under the

assumption of 10 classification error in the training data 65

4-8 Diagramatic visualization of decision trees learned for different decision

making situations for the wind bracing data 66

4-9 The accuracy of AQDT-2 and AQ15c with different parameter settings

for the wind bracing PIoblem 68

4-10 Analyzing different parameter setting of AQDT-2 using the wind bracing data 69

4-16 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-20 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-11 A comparison between AQ15c and AQDT-2 against C45 on the wind bracing data 69

4-12 A visualization diagram of the MONK-l problem 70

4-13 Decision rules learned by AQ15c for the MONK-1 problem 71

4-14 The decision tree for the MONK-1 problem generated both by AQDT-l 72

4-15 Compact decision structures generated by AQDT-2 for the MONK-1 problem 72

for the MONK-l Problem 74

4-17 Analyzing different parameter setting of AQDT-2 with the MONK-l data 75

4-18 Comparing AQ15c and AQDT-2 against C45 using the MONK-1 problem 75

4-19 A visualization diagram of the MONK-2 problem 76

for the MONK-2 Problem 78

4-21 Analyzing different parameter setting of AQDT-2 with the MONK-2 data 79

xii

4-22 Comparing AQ15c andAQDT-2 against C45 using the MONK-2 problem 79

4-24 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-35 A visualization diagram of a decision tree learned by AQDT -2 after reducing

4-23 A visualization diagram of the MONK-3 problem 80

for the MONK-3 Problem 82

4-25 Analyzing different parameter setting of AQDT-2 with the MONK-3 data 82

4-26 Comparing AQ15c and AQDT-2 against C45 using the MONK-3 problem 83

4-27 Comparing AQ15c and AQDT-2 against C45 using the breast cancer problem 84

4-28 Comparing AQ15c and AQDT-2 against C45 using the mushroom problem 85

4-29 Decision structures learned by AQDT-2 for different decision-making situations 87

4-30 Comparing decision trees for the Cong Voting-84 data learned by C45 amp AQDT -2 88

4-31 A visualization diagram of decision rules learned by AQ15c for the MONK-2 91

4-32 A visualization diagram shows the testing errors for the rules in Figure 4-31 92

4-33 A visualization diagram of the decision tree learned by AQDT-2 for the MONK-2 93

4-34 A visualization diagram shows testing errors of AQDT-2 decision tree bull 94

the generalization degree to 1 94

DERIVING TASKmiddotORIENTED DECISION STRUCTURES

FROM DECISION RULES

Ibrahim M Fahmi Imam PhD

School of Information Technology and Engineering

Geotge Mason University Fall 1995

Ryszard S Michalski Advisor

ABSTRACT

This dissenation is concerned with research on learning task-oriented decision structures from

decision rules The philosophy behind this research is that it is more appropriate to learn

knowledge and store it in a declarative form and then when a decision making situation occurs

generate from this knowledge the decision structure that is most suitable for the given decision

making situation Learning decision structures from decision rules was first introduced by

Michalski (1978) The flIst implementation of this approach was done by Imam and Michalski

(1993 ab) called AQDT-l

This approach separates the function of generating a knowledge-base from the function of using

the knowledge-base for decision-making The first function focuses on learning accurate

consistent and complete concept description expressed in a declarative form The second function

is performed whenever a new decision-making situation occurrs a task-oriented decision structure

is obtained to suit that situation Thsk-oriented knowledge is defined as knowledge that is adapted

for solving a given decision-making situation (Imam amp Michalski 1994 Michalski amp Imam

1994)

The dissertation introduces the system AQJJT-2 for learning task -oriented decision structures from

decision rules or examples Each decision making situation is defined by a set of parameters that

controls the learning process of the AQDT-2 system The extensive experiments on AQJJT-2 show

that decision structures learned by it usually outperform in terms of accuracy and average size of

the decision structures those learned from examples by other well known systems The results

show also that the system does not work very well with noisy data The system is illustrated and

compared using applications of artificial problems such as the three MONKs problems (Thrun

Mitchell amp Cheng 1991) and the East-West 1iain problem (Michie et al 1994) It was also

applied to real world problems of learning decision structures in the areas of construction

engineering (for determining the best wind bracing design for tall buildings) medical

diagnosis(for learning decision rules for recognizing breast cancer) agriculture diagnosis (for

learning classification rules for distinguishing between poisonous and non-poisonous

mushrooms) and political data (for characterizing democratic and republican voting records)

CHAPTER 1 INTRODUCTION

11 Motivation and Overview

Learning and discovery systems should be able not only to generate and store knowledge but also

to use this knowledge for decision-making The main step in the development of systems for

decision-making is the creation of a knowledge structure that characterizes the decision-making

process The form in which knowledge can be easily obtained may however differ from the form

in which it is most readily used for decision-making It is therefore important to identify the form

of knowledge representation that is most appropIiate for learning (eg due to ease of its

modification) and the form that is most convenient for decision making

A simple and effective tool for describing decision processes is a decision structure which is a

directed acyclic graph that specifies an order of tests to be appliedto an object (or a situation) to

arrive at a decision about that object The nodes of the structure are assigned individual tests

(which may correspond to a single attribute a function of attributes or a relation) the branches are

assigned possible test outcomes or ranges of outcomes and the leaves are assigned a specific

decision a set of candidate decisions with corresponding probabilities or an undetermined

decision A decision structure reduces to a familiar decision tree when each node is assigned a

single attribute and has at most one parent when the branches from each node are assigned single

values of that attribute and when leaves are assigned single definite decisions Thus the problem

of generating a decision structure is a generalization of the problem of generating a decision tree

Decision trees are typically generated from a set of examples of decisions The essential

characteristic of any such method is the attribute selection criterion used for choosing attributes to

be assigned to the nodes of the decision tree being built Such criteria include the entropy

reduction the gain and the gain ratio (Quinlan 1979 83 86) the gini index ofdiversity (Breiman

et al 1984) and others (Cestnik amp Bratko 1991 Cestnik amp Karalic 1991 Mingers 1989a)

3

4

Adecision treedecision structure representations can be an effective tool for describing a decision

process as long as all the required tests can be performed and the decision-making situations it

was designed for remain constant (eg in a doctor-patient example the doctor should detennine

that answers for all symptoms appear in the decision tree) Problems arise when these assumptions

do not hold For example in some situations measuring certain attributes may be difficult or costly

(eg in the doctor-patient example a brain or blood test is needed which is very expensive or the

tools needed are not available) In such situations it is desirable to refonnulate the decision

structure so that the inexpensive attributes are evaluated frrst (assigned to the nodes close to the

root) and the expensive attributes are evaluated only if necessary (by assignment to the nodes far

away from the root) If an attribute cannot be measured at all it is useful to either modify the

structure so that it does not contain that attribute or-when this is impossible-to indicate

alternative candidate decisions and their probabilities A restructuring is also desirable if there is a

significant change in the frequency of occurrence of different decisions (eg in the doctor-patient

example the doctor may request a decision structure expressed in a specific set of symptoms

biased to classify one or more diseases or specify a certain order of testing)

Arestructuring ofa decision structure (or a tree) in order to suit new requirements could be quite

difficult This is because a decision structure is a procedural representation that imposes an

evaluation order on the tests In contrast no evaluation order is imposed by a declarative

representation such as a set of decision rules Tests (conditions) of rules can be evaluated in any

order Thus for a given set of rules one can usually build a huge number of logically equivalent

decision structures (trees) which differ in the test ordering Due to the lack of order constraints

a declarative representation (rules) is much easier to modify to adapt to different situations than a

procedural one (a decision structure or a tree) On the other hand to apply decision rules to make a

decision one needs to decide in which order tests are evaluated and thus needs a decision

structure

An attractive solution to these opposite requirements is to acquire and store knowledge in a

declarative form and transform it to a decision structure when it is needed for decision-making

5

This method allows one to create a decision structure that is most appropriate in a given decisionshy

making situation Because the number of decision rules per decision class is usually small (each

rule is a generalization of a set of examples) generating a decision structure from decision rules

can potentially be perfonned much faster than by generation from training examples Thus this

process could be done on line without any delay noticeable to the user Such virtual decision

structures are easy to tailor to any given decision-making situation

This approach allows one to generate a decision structure that avoids or delays evaluating an

attribute that is difficult to measure in some decision-making situation or that fits well a particular

frequency distribution of decision classes In other situations it may be unnecessary to generate

complete decision structure but it may be sufficient to generate only a part of it which concerns

only with decision classes of interest Thus such an approach has many potential advantages

This dissertation presents a new system called AQflf-2 The AWf-2 system generates a taskshy

oriented decision structure (decision structure that is adapted to the given decision-making

situation) from decision rules The decision rules are learned by either rule leaming system AQ15

(Michalski et al 1986) or system AQ17-DCI which has extensive constructive induction

capabilities (Bloedorn et al 1993)

To associate the decision rules with a given decision-making task AQflf-2 provides a set of

features including 1) enabling the system to include in the decision structure nodes corresponding

to new attributes constructed during the process of learning the decision rules 2) controlling the

degree of generalization needed during the development of the decision structure 3) providing four

new criteria for selecting an attribute to be a node in the decision structure that allow the system to

generate many different but equivalent decision structures from the same set of rules 4) generating

unknown nodes in situations when there is insufficient infonnation for generating a complete

decision structure 5) learning decision structures from discriminant rules as well as

characteristic rules and 6) providing the most likely decision when the decision process stops

due to inability to evaluate an attribute associated with an intennediate node

6

To test the methooology of generating decision structures from decision rules an extensive set of

planned experiments have been designed to test different aspects of the approach The experiments

include testing different combinations of parameters for each sub-function of the approach

analyzing the relatioship between decision rules and the decision structure learned from them and

comparing decision trees learned by the AQUf2 system the well known C45 (Quinlan 1993)

system for learning decision trees from examples Different experiments were designed to examine

the new features of the AQUf approach for learning task-oriented decision structures The

experiments were applied to artificial domains as well as real-world domains including MONK-I

MONK-2 and MONK-3 (Thrun Mitchell amp Cheng 1991) East-West trains (Michie et al

1994) Engineering Design-wind bracings (Arciszewski et al 1992) Mushrooms Breast

Cancer (Mangasarian amp WolbeJg 1990) and Congressional bting Records of 1984 The

MONKs problems are concerned with learning classification rules for robot-like figures MONK-l

requires learning a DNF-type description MONK-2 requires learning a non-DNF-type description

(one that cannot be easily described as a DNF rule using the original attributes) MONK-3 concerns

learning a DNF rule from noisy data The East-West trains dataset is a structural domain that

classifies two sets of trains (Eastbound and Westbound) The Engineering Design-wind

bracing data involves learning conditions for applying different types of wind bracing for ta11

buildings The Mushrooms data is concerned with learning classification rules for distinguishing

between poisonous and non-poisonous mushrooms The Breast Cancer data involves learning

concept descriptions for recognizing breast cancer The congressional voting data includes voting

records on different issues AQUf2 outperformed C45 on the average with respect to both the

predictive accuracy and tree size for most problems Apf-2 did not work very well with noisy

data or with problems that have many rules covering very few examples

12 The Problem Statement

There are many limitations and problems accompanied using decision trees for decision-making

CHAPTER 2 RELATED RESEARCH

21 Learning Decision Trees from Decision Diagrams

The AQDT-2 method proposed here is based on the earlier work by Michalski (1978) which

introduced an algorithm for generating decision trees from decision lists The method proposed

several attribute selection criteria These criteria are of increasing power of the main criterion

order cost estimate (the nth order cost estimates n=l 2 ) Michalski also analyzed two

specific criteria MAL and DMAL for selecting the optimal attribute for a node in the tree

based on properties extracted from the decision diagram In order to better explain the method

it is necessary to define some terms These terms will be used later in the dissertation

Definition 2-1 A ~ of a decision class C is a set of rules whose union includes the set of all

examples of class C and does not include any examples of other decision classes

Definition 2-2 A ~ of a decision class C is disjoint if all rules (rules) are pairwise logically

disjoint In other words for any two rules there exists a condition with the same attribute but

with different values in each rule

Definition 2-3 An minimal cover is a disjoint cover which has the smallest number of rules

among all possible covers

Definition 2-4 A diagram for a given cover is a table constructed graphically by representing

in two dimension space of all possible combination of attributes values and locating on the

diagram all the condition parts of the given rules and marking them with the action specified

with each rule

Michalskis algorithm requires construction of minimalmiddot cover The minimal cover should be

consistent and complete The method is based on the fact that if there are n decision classes

any decision tree that correctly classifies the given rules and has n distinct leaves is a minimal

7

8

decision tree (any consistent decision tree should have at least n leaves) Michalski (1978) has

shown that if only one rule is broken by a selected attribute then instead of having one leaf

(which could potentially represent this rule or the decision class in the tree) there will have to

be at least two leaves representing this rule in the fmal decision tree

The attribute selection criterion MAL introduced in (Michalski 1978) prefers attributes that do

not break any rules or break as few as possible An attribute breaks a rule if the attribute can

divide the rule into two or more sub-rules Figure 2-1 shows two examples of two sets of rules

In the fJIst xl and x3 break at least two rules each (xl breaks the rule [x4=2] amp [x1=lv3] amp

[x3=2v3] and the rule [x4=3] amp [x1=1v2] amp [x3=1] and x3 breaks three rules the rule [x4=2]

amp [xl=2] amp [x3=2v3]the rule [x4=l] amp [xl=3] amp [x3=lv3] and the rule [x4=3] amp [xl=4] amp

[x3=2v3]) In the diagram on the right xl is the only attribute that does not break any rule

Figure 2-1 An example to illustrate how attributes break rules

One of the criteria defmed by Michalski is the first degree cost estimate which assigns to each

attribute an integer equal to the number of rules broken by that attribute This criterion is also

called the static cost estimate of an attribute or the criterion of minimizing added leaves

(MAL)

-

C3

9

The MAL criterion (Minimizing Added Leaves) seeks an attribute that minimizes the

estimated number of additional nodes in the decision tree being generated over a hypothetical

minimal decision tree When there is a tie between two attributes the attribute to be selected is

the one which breaks smaller rules (rules that cover fewer examples or more specialized

rules) AQDT-2 uses an approximate version of this criterion (the attribute dominance)

Another criterion introduced by Michalski was the DMAL criterion The DMAL criterion

(Dynamically Minimizing Added Leaves) is based on a principle similar to that of MAL but

is more complex because once an attribute is selected as a node in the tree some rules andor

parts of the broken rules at each branch are merged into one rule The DMAL ensures that the

value of the total cost estimate of an attribute is decreased by a value equal to the number of

merged rules minus one

Example Learn a decision tree from the following decision table

The minimal cover consists of the following rules

Al lt [x2=O] v [x1=0][x2=2] A2 lt [x2=I] v [xl=2][x2=2] A3 lt [xl=I][x2=2]

The evaluations ofthe MAL criterion for these attributes are 2 for xl 0 for x2 5 for x3 and 5

for x4 The attribute xl divides the two rules [x2=O] and [x2=I] so selecting it will add two

leaves to the optimal number of leaves It is clear that the attribute to be selected as a root of

10

the decision tree is x2 Then three branches are attached to the root node and the decision rules

are divided into subsets each corresponding to one branch For x2 = 0 or 1 a leave node is

generated For x2=2 another attribute is selected to be a node in the tree In this case xl has the

minimum MAL value Figure 2-2 shows the decision tree obtained using the MAL criterion

Al A3 A2

Figure 2-2 A decision tree learned from the decision table in Table 2-1

22 Learning Decision Trees from Examples

Decision tree learning is a field that concerned with generating decision tree that classifies a set

of examples according to the decision classes they belong to The essential aspect of any

inductive decision tree method is the attribute selection criterion The attribute selection

criterion measures how good the attributes are for discriminating among the given set of

decision classes The best attribute according to the selection criterion is chosen to be assigned

to a node in the tree The fIrst algorithm for generating decision trees from examples was

proposed by Hunt Marin and Stone (1966) Hunts algorithm uses a divide and conquer

algorithm for building decision trees This algorithm has been subsequently modified by

Quinlan (1979) and applied by many researchers to a variety of learning problems

Attribute selection criteria can be divided into three categories These categories are logicshy

based information-based and statistics-based The logic-based criteria for selecting attributes

use logical relationships between the attributes and the decision classes to determine the best

attribute to be a node in the decision tree such as the MAL criterion minimizing added leaves

(Michalski 1978) which uses conjunction and disjunction operators The information-based

criteria are based on the information theory These criteria measure the information conveyed

11

by dividing the training examples into subsets Examples of such criteria include the

information measure 1M the entropy reduction measure and the gain criteria (Quinlan 1979

83) the gini index of diversity (Breiman et al 1984) Gain-ratio measure (Quinlan 1986) and

others (Clark amp Niblett 1987 Bratko amp Lavrac 1987 Cestnik amp Karalic 1991) The

statistics-based criteria measure the correlation between the decision classes and the attributes

These criteria use statistical distributions for determining whether or not there is a correlation

The attribute with the highest correlation is selected to be a node in the tree Examples of

statistics-based criteria include Chi-square and G-statistic (Sokal amp Rohlf 1981 Han 1984

Mingers 1989a)

Niblett and Bratko (1986) Quinlan (1987) and Bratko and Kononeko (1987) extended the

method of learning decision trees to also handle data with noise (by pruning) Handling noise

extended the process of learning decision trees to include the creation of an initial complete

decision tree tree pruning which is done by removing subtrees with small statistical Validity

and replacing them by leaf nodes (MiDgers 1989b) More recently pruning has also been used

for simplifying decision trees even for problems without noise (Bohanec amp Bratko 1994)

Pruning decision trees improves their simplicity but reduces their predictive accuracy on the

training examples Quinlan (1990) proposed also a method to handle the unknown attributeshy

value problem by exploring probabilities of an example belonging to different classes

The rest of this section includes a brief description of the attribute selection criterion used by

C45 learning system (Quinlan 1993) C45 uses an information-based criterion for selecting

an attribute to be a node in the tree Also the section includes a brief description of the Chishy

square method for attribute selection (Mingers 1989a) The method is a statistics-based

method for selecting an attribute to be a node in the tree

221 Building Decision Trees Using htformation-based Criteria

12

This section presents a description of the inductive decision tree learning system C4S The

C4S learning system is considered to be one of the most stable accurate and fastest program

for learning decision trees from examples

Learning decision trees from examples requires a collection of examples Each example is

represented by a fixed number of attribute-value pairs C45 (Quinlan 1993) is a learning

program that induces classification decision trees from a set of given examples The C4S

learning system is descended from the learning system ID3 (Quinlan 1979) which is based on

Hunts method for constructing decision tree from a set ofcases (Hunt Marin amp Stone 1966)

The C45 system uses an attribute selection criterion called the Gain Ratio This criterion

calculates the gain in classifying information based on the residual information needed to

classify cases in a set of training examples and the information yielded by the test based on the

relative frequencies of the possible outcomes (decision classes) The gain ratio criterion is

based on an earlier criterion used by ID3 called the Gain Criterion The Gain Criterion uses

the frequency of each decision class in the given set of training examples

Once an attribute is chosen to be a node in the tree the system generates as many links as the

number of its values and classifies the set of examples based on these values If all the

examples at a certain node belong to one decision class the system generates a leaf node and

assigns it to that class Otherwise the system searches for another attribute to be a node in the

tree

The Gain Criterion The gain criterion is based on the information theory That is the

information conveyed by a message depends on its probability and can be measured in bits as

minus the logarithm (base 2) of that probability To explain the gain criterion suppose that for

a given problem Xu bullbull X are given attributes and Cu c are decision classes Suppose S is

any set of cases and T is the initial set of training cases The frequency of class C1 in the set S

is the number of examples in S that belong to class C i bull

13

freq(Cit S) = Number of examples in S belong to CI (2-1)

Suppose that lSI is the total number of examples in S the probability that an example selected

at random from S belongs to class Ci is freq(CIbullS) IISI

The information conveyed by the message that a selected example belongs to a given decision

class Cit is determined by -log1 (freq(Ch S) IISI) bits

The expected information from such a message stating class membership is given by

info(S) = -k

L (freq(C S) I lSI) log1 (freq(C S) IISI) bits (2-2)I

info(S) is known also as the entrQpy of the set S When S is the initial set of training examples

info(T) determines the average amount of information needed to identify the class of an

example in T

Suppose that we selected an attribute X to be the root of the tree and suppose that X has k

possible values The training set T will be divided into k subsets each corresponding to one of

Xs values The expected information of selecting X to partition the training set T infoz(T) can

be found as the sum over all subsets of multiplying the information conveyed by each subset

by its probability

k

infoz(T) =L (ITII 1m) infO(TIraquo (2-3) I

The information gained by partitioning the training examples T into subset using the attribute

X is given by

gain(X) =inlo(T) - inoIT) (2-4)

The attribute to be selected is the attribute with maximum gain value

The Gain Ratio Criterion This criterion indicates the proportion of information generated by

the split that appears helpful for classification Quinlan (1993) pointed out that the gain

criterion has a serious deficiency Basically it is strongly biased toward attributes with many

outcomes (values) For example for any data that contains attributes such as social security

14

number the gain criterion will select that attribute to be the root of the decision tree However

selecting such attributes increases the size of the decision tree Quinlan provided a solution to

this problem by introducing the gain ratio criterion which takes the ratio of the information that

is gained by partitioning the initial set of examples T by the attribute X to the potential

information generated by dividing T into n subsets

Following similar steps to obtain the information conveyed by dividing T into n subsets The

expected information generated by dividing T into n subsets and by analogy to equation 2-2 is

determined by

split info(T) = - Lt

(ITII lTI) 10gl (ITII m ) (2-5) I

The gain ratio is given by

gain ratio(X) = gain(X) I split info(X) (2-6)

and it expresses the proportion of information generated by the split that is useful for

classification

Example Consider the following example presented by Quinlan (1993) Table 2-2 shows the

set of training examples

First determine the amount of information gained by selecting the attribute outlook to be a

root of the decision tree This attribute divides the training examples into three subsets

sunny-with five examples two of which belong to the class Play overcast-with four

examples all of which belong to the class Play and rain-with five examples three of

which belong to the class Play To determine info(11 the average information needed to

identify the class of an example in T there are 14 training examples and two decision classes

Nine of these examples belong to the class Play and five belong to the class Dont Play

Info(11 = - 914 10gl (914) - 514 10gl (514) = 094 bits

When using outlook to divide the training examples the information becomes

15

infO(T) = 514 -25 log2 (25) - 35 10g2 (35)

+ 414 -44 10g2 (44) - 04 log2 (04)

+ 514 -35 logl (35) - 25 logl (25) = 0694 bits

By substituting in equation 2-4 the gain of information results from using the attribute

outlook to split the training examples equal to 0246 The gain information for windy is

0048

Figure 2-3 shows a decision tree learned for this problem using the gain criterion The split

information for outlook is determined as follows

split info(T) = - 514 logl (514) - 414 log2 (414) - 514 log2 (514) = 1577 bits

The gain ratio for outlook = 02461577 =0156

16

Figure 2 -3 A decision tree learned using the gain criterion for selecting attributes

The C45 system handles discrete values as well as continuous values To handle an attribute

with continuous values C45 uses a threshold to transform the continuous domain into two

intervals In other words for each continuous attribute C45 generates two branches one

where the values of that attribute is greater than the determined threshold and the other if the

value is less than or equal to the threshold

Tree pruning in C45 is a process of replacing subtrees with small classification validity by

leaves The C45 system uses Laplace ratio for determining the error rate of different subtrees

This ratio is defined as (e+l) (n+2) where n is the number of the training examples and e is

the number of misclassified examples at a given leaf

222 Building Decision Trees Using Statistics-based Criteria

The Chi-square method for selecting attributes was used by Hart (1984) and Mingers (1989a)

in building decisionmiddot trees The method uses Chi-square statistics to measure the association

between two attributes When building decision trees the method is implemented such that it

determines the association between each attribute and the decision classes The attribute to be

selected is the one with greatest value

To determine the Chi-square value for an attribute consider aij is the number of examples in

class number i where the attribute A takes value numberj In other words aij is the frequency

of the combination of decision class number i and the attribute value number j The Chi-square

value for attribute A is given by

n m

Chi-square (A) =L L [ (aij - Eijl2 Eij ] (2shy

i=1 j=1

where n is the number of decision classes m is the number ofvalues of a given attribute Also

17

Eij = (TCi TVj ) T (2shy

8)

where T Ci and T Vj are the total number of examples belong to the decision class a and the total

number of examples where the attribute A takes value vj respectively T is the total number of

examples

Consider Quinlans example in Table 2-2 Table 2-3 shows the frequency of different

combination of values between the decision class and both the outlook and the Windy

attributes Table 2-4 shows the expected values TCi and T Vj of the frequencies in Table 2-3

of different attributes values for different decision classes

To determine the association value between the decision classes and both the attribute Windy

and the attribute Outlook the observed Chi-square values are

Chi-square (Windy Class) =[(3-39)239] + [(3-21)221] + [(6-51)232] + [(2-29)232]

=021 + 039 + 025 + 025 = 11

Chi-square (Outlook Class) = [(2-32)232] + [(4-26)226] + [(3-32)232] + [(3-18118]

+ [(0-14)214] + [(2-18)2 18] =045 + 075 + 02 + 08 + 14 + 002 = 362

18

Applying the same method to the other attributes the results will favor the attribute Outlook

Once that attribute is selected to be a node in the tree the remaining set of examples are divided

into subsets and the same process is repeated on each subset

Table 2-5 shows a summary of these criteria and their basic evaluation function

Table 2-5 Attribute selection criteria and their basic evaluation measure

Info Measure (IM) Gain Gshystatistics and Gain Ratio Chi-square

II

Entropy(S) = - L (freq(~ 5) 1151) logz (freq(CIt 5) 1151 ) I

G-Statistics =2N 1M (N-No of examples) n m

Chi-square (A B) = L L [(aij - ~j)21Eij ] i=l

223 Analysis of Attribute Selection Criteria

This subsection introduces briefly the analysis of different selection criteria which was done by

Mingers (1989a) Mingers compared six attribute selection criteria that are used in decision

tree programs These criteria are the Information Measure (IM) Chi-square G statistics Gin

index of diversity Marshal correction and Gain RaJio The overall results show that the Gain

Ratio criterion had the strongest xesults

In the flISt experiment Minger tested the six criteria on a problem with ambiguous examples

(ie examples may belong to more than one decision class) to observe how the selected criteria

evaluate the given attributes The problem has two decision classes and two attributes X and

Y It was assumed that attribute X is better for classifying the examples than attribute Y The

training examples were unevenly spread between the two values of X Attribute Y has three

values and the examples were spread randomly among them Table 2-6 (a and b) shows the

contingency tables for both attributes Table 2-7 shows a summary of the goodness of split

provided by the six criteria Mingers noted that the measures that are not based on information

19

theory give radiation (attribute X here) less weighL This may be because the zero in the fIrst

row of radiation has a greater influence in the log calculation In the case of using the Chishy

square criterion the value zero adds the maximum association between any two attributes

because the Chi-square value of a zero cell is the expected value of this cell

b)

Now let us demonstrate results from another experiment done by Mingers In this experiment

Mingers used four different data sets to generate decision trees for eleven different criteria In

the final results he compared the total number of nodes and the total error rate provided by

each criterion over all given problems Table 2-8 shows the final results for five selected

criteria only

20

U lgt ~middot results for comparing the total accuracy and size of decision trees different attribute selection criteria from four VVLU

This experiment was performed on four real world data sets These data are concerned with

profiles of BA Business Studies degree students recurrence of breast cancer classifying types

of Iris and recognizing LCD display digits The data was divided randomly 70 for training

and 30 for testing For more details see Mingers (1989a)

23 Learning Decision Structures

Considering the proposed defInition of decision structures given above two related lines of

research are described in this section Gaines (1994) and Kohavi (1994) proposed two

approaches for generating decision structures that share some of the earlier ideas by Imam and

Michalski (1993b)

In the first approach Brian Gaines introduces a method for transforming decision rules or

decision trees into exceptional decision structures The method builds an Exception Directed

Acyclic Graph (EDAG) from a set of rules or decision trees The method starts by assigning

either a rule say RO or a conclusion say CO or both to the root node If it assigns a conclusion

to the root node it places it on a temporary conclusion list Then it generates a new child node

and add to it a new rule say Rl The method evaluates the new rule Rl if it satisfies the

conclusion CO in its parent node it builds a new child and repeats the processes Otherwise if

the rule Rl does not satisfy the conclusion CO it changes the temporary memory with the new

conclusion Cl which is satisfied by the rule RO The same process is repeated until all rules

that have common conditions with the rule at the root are evaluated The method then creates a

21

new child node from the root and repeat the process until all rules are evaluated In the decision

structure nodes containing rules only represent common conditions to all of its children

The main disadvantages of this approach is that it requires discriminant rules to build such a

decision structure Also such a structure is more complex than the traditional decision trees

that are used for decision-making

Figure 2-4 shows an example of set of rules and their equivalent exceptional decision structure

The decision structure can be read as follows It is Safe except if xl=1 amp x2=1 amp x3=1 and

either x4=3 or x5=I then it is Lost except if x6=1 it is Safe except if x7=1 it is Lost

The second approach introduced by Ronny Kohavi learns decision structures from examples

using a bottom-up algorithm The method uses a greedy Hill-climbing algorithm for inducing

Oblivious read-Once Decision Graphs (HOODO) A read-once decision graph is defined as a

decision graph where each attribute occurs at most once along any computational path In other

words for each path from the root to the leaves of the decision structure attributes may occur

as a node once at most However there may be more than one node with the same attribute in

the decision graph Kohavi also defined a leveled decision graph in which the nodes are

partitioned into a sequence of pairwise disjoint sets such that outgoing edges from each level

terminate at the next level An oblivious decision graph is a decision graph where all nodes at

a given level are labeled by the same attribute

22

Safe lt [xl=2] Safe lt [x2=2] Safe lt [x3=2] Safe lt [x4=1] amp [x5=2] Safe lt [x4=1] amp [x5=3] Safe lt [x6=l] amp [x7=2] Safe lt [x6=1] amp [x7=3] Safe lt [x4=2] amp [x5=2] Safe lt [x4=2] amp [x5=3]

Lost lt [x6=2] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x6=3] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x5=1] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=2] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=3] amp [x3=1] amp [x2=I] amp [xl=l]

The algorithm starts by generating a leaf node for each decision class Then it applies a

nondeterministic method to select an attribute to build new level of the decision structure This

attribute is removed from the data and the data is divided into subsets each corresponding to a

complex combination of that attributes values For each subset the process is repeated until all

the examples of a given subset belong to one decision class For example if the selected

attribute say A has two values (0 and 1) and there are two decision classes (CO and CI) The

data is divided into two subsets the fIrst subset contains examples where A takes value 0 and

belong to class CO or takes value 1 and belong to class Cl The second subset of examples is

the set where A takes value 0 and belong to class CI or takes value 1 and belong to class CO

The number of nodes of the fIrst level (after the leave nodes) is expected to be less than or

equal to ka where k is the number of decision classes and n is the number of values of the

selected attribute and it can increase exponentially before that number is reduced

exponentially to one

It is easy for the reader to fIgure out some major disadvantages of such approach including

The average size of such decision structures is estimated to be very large especially when there

23

is no similarity (ie strong patterns) or logical relationship in the data The time used to learn

such decision structure is relatively very high comparing to systems for learning decision trees

from examples and fmally it could be better to search for attribute which reduces the number

of generated subsets of the data instead of nondeterministically selecting an attribute to build a

new level of the decision structure Kohavi provided a comparison between the C45 system

and his system that can be found in (Kohavi 1994) Later Kohavi and Li (1995) used a

deterministic method for attribute selection which minimizes the width of the penultimate level

of the graph

Table 2-9 shows a comparison between the proposed approach and those two approaches The

EDAG and HOODG systems are unreleased prototype systems

or Decision structures are easy to Decision structures are Decision structures are easy to understand difficult to read understand

CHAPTER 3 DESCRIPTION OF THE APPROACH

31 General Methodology

In the proposed approach the function of learning or discovery is separated from the function of

using the discovered knowledge for decision-making The frrst function is performed by an

inductive learning program that searches for knowledge relevant to a given class of decisions

and stores the learned knowledge in the form of decision rules The second function is performed

when there is a need for assigning a decision to new data points in the database (eg bull a

classification decision) by a program that transforms obtained knowledge into a decision

structure optimized according to the given decision-making situation

The Learning Task GivenA set of training examples describing the concept to be learned Learning goal which specifies the decision classes to be learned from the training examples A background knowledge to control the learning process Determine A concept description in a declarative form of knowledge (decision rules) that satisfies the learning goal

The Decision-making Task GivenA set of decision rules in a conjunctive form A description of the new decision-making situation (eg attributes costs and order preference importance or frequency of decision classes etc) One or more examples that needed to be tested under the given decisionshymaking situation A set of parameters to control the learning process Determine A decision structure that suits the given decision-making situation

The decision rules used here are learned by either the AQ1S (Michalski et al 1986) or AQ17

(Bloedorn et aI 1993) learning systems The reasons for selecting decision rules as a form of

23

24

declarative knowledge are that they do not impose any order on the evaluation of the attributes

and due to the lack of order constraints decision rules can be evaluated in many different

ways which increases the flexibility of adapting them to the different tasks of decision-making

(Bergadano et al 1990) Since the number of rules learned per class is typically much smaller

than the number of examples per class generating a decision structure from decision rules can

potentially be done on line

Such virtual decision structures can be tailored to any given decision-making situation The

needed decision rules have to be generated only once and then they can be used many times for

generating decision structures according to changing requirements of decision-making tasks The

method uses the AQDT-2 system (Imam amp Michalski 1994) for learning decision structures

from decision rules Decision structures represent a procedural form of knowledge which makes

them easy to implement but also harder to change Consequently decision structures can be quite

effective and useful as long as they are used in decision-making situations for which they are

optimized and the attributes specified by the decision structure can be measured without much

cost Figure 3-1 shows an architecture of the proposed methodology

Data

Decision -

- Learning knowledge from database The decision making process

Figure 3 -1 Architecture of the AQDT approach

25

It is assumed that the database is not static but is regularly updated A decision-making problem

arises when there is a case or a set of cases to which the system has to assign a decision based on

the knowledge discovered Each decision-making situation is defmed by a set of attribute-values

Some attribute-values may be missing or unknown A new decision structure is obtained such

that it suits the given decision-making problem The learned decision structure associates the

new set of cases with the proper decisions

32 A Brief Description of the AQIS and AQ17 Rule Learning Programs

The decision rules are generated from examples by an AQ-type inductive learning system

specifically by AQ15 (Michalski et albull 1986) or AQI7-DCI (Bloedorn at al 1993) The rest of

this subsection includes a brief description of the inductive learning systems AQ15 and AQI7

AQ15 learns decision rules for a given set of decision classes from examples of decisions using

the STAR methodology (Michalski 1983) The simplest algorithm based on this methodology

called AQ starts with a seed example of a given decision class and generates a set of the

most general conjunctive descriptions of the seed (alternative decision rules for the seed

example) Such a set is called the star of the seed example The algorithm selects from the star

a description that optimizes a criterion reflecting the needs of the problem domain If the

criterion is not defined the program uses a default criterion that selects the description that

covers the largest number of positive examples (to minimize the total number of rules needed)

and with the second priority that involves the smallest number of attributes (to minimize the

number of attributes needed for arriving at a decision)

If the selected description does not cover all examples of a given decision class a new seed is

selected from uncovered examples and the process continues until a complete class description

is generated The algorithm can work with few examples or with many examples and can

optimize the description according to a variety of easily-modifiable hypothesis quality criteria

26

The learned descriptions are represented in the form of a set of decision rules expressed in an

attributional logic calculus called variable-valued logic 1 or VLl (Michalski 1973) A

distinctive feature of this representation is that it employs in addition to standard logic operators

the internal disjunction operator (a disjunction of values of the same attribute in a condition) and

the range operator (to express conditions involving a range of discrete or continuous values)

These operators help to simplify rules involving multi valued discrete attributes the second

operator is also used for creating logical expressions involving continuous attributes

AQ15 can generate decision rules that represent either characteristic or discriminant concept

descriptions depending on the settings of its parameters (Michalski 1983) A characteristic

description states properties that are true for all objects in the concept The simplest

characteristic concept description is in the form of a single conjunctive rule (in general it can be

a set of such rules) The most desirable is the maximal characteristic description that is a rule

with the longest condition part ie stating as many common properties of objects of the given

class as can be determined A discriminant description states properties that discriminate a given

concept from a fixed set of other concepts The most desirable is the minimal discriminant

descriptions that is a rule with the slwnest condition part For example to distinguish a given

set of tables from a set of chairs one may only need to indicate that tables have large flat top

A characteristic description of the tables would include also properties such as have four legs

have no back have four corners etc Discriminant descriptions are usually much shorter than

characteristic descriptions

Another option provided in AQ15 controls the relationship among the generated descriptions

(rulesets or covers) of different decision classes In the IC (Intersecting Covers) mode

rulesets of different classes may logically intersect over areas of the description space in which

there are no training examples In the DC (Disjoint Covers) mode descriptions of different

classes are logically disjoint The DC mode descriptions are usually more complex both in the

number of rules and the number of conditions There is also a DL mode (a Decision List mode

27

also called VL modeM-for variable-valued logic mode) in which the program generates rule sets

that are linearly ordered To assign a decision to an example using such rulesets the program

evaluates them in order If ruleset is satisfied by the example then the decision is made

otherwise the program proceeds to the evaluation of the ruleset +1 In IC and DC modes

rule sets can be evaluated in any order

Alternatively the system can use rules from the AQ17-DCI program for Data-driven

Constructive Induction AQ17-DCI differs from AQ15 mainly in that it contains a module for

generating additional attributes These attributes are various logical or mathematical

combinations the original attributes The program generates a large number of potential new

attributes and seleets from them those most promising based on an attribute quality criterion

To illustrate the format of rules generated by AQ15 (or AQ17-DCI) an exemplary ruleset is

shown in Figure 3-2 The ruleset (that can be re-represented as a disjunctive normal form

expression) describes a voting record of Democratic Representatives in the US Congress

Each rule is a conjunction of elementary conditions Each condition expresses a simple relational

statement For example the condition [State = northeast v northwest] states that the attribute

State (of the Representative) should take the value northeast or northwest to satisfy the

condition

Rl [Gas_concban = yes] amp [Soc_see_cut = no v not registered] R2 [Draft = Yes v not registered]amp[AlaslaLparks = yes v not registered] amp

[Food_stamp_cap = no] amp [State = northeast v northwest] R3 [Chrysler =yes v not registered] amp [Income =low] R4 [Education =yes] amp [Occupation = yes]

Figure 3-2 A ruleset generated by AQ15 for the concept Voting pattern of Democratic Representatives

The above rules were generated from examples of the voting records For illustration below is

an example of a voting record by a Democratic representative

Draft registration=no Ban of aid to Nicaragua=no Cut expenditure on mx missiles=yes Federal subsidy to nuclear power stations=yes Subsidy to national parks

28

in Alaska=yes Fair housing bill=yes Limit on Pac contributions=yes Limit onood stamp program=no Federal help to education=no StateFrom=north east State Population=large Occupation =unknown Cut in social security spending=no Federal help to Chrysler corp=not registered

By expressing elementary statements in the example as conditions and linking conditions by

conjunction the examples can be re-expressed as decision rules Thus decision rules and

examples formally differ only in the degree of generality

33 Generating Decision Structurestrees from Decision Rules

This section describes the AQDT-1 system for learning decision structures (trees) from decision

rules (Imam amp Michalski 1993ab) Also a description of the AQDT-2 method for learning taskshy

oriented decision structure from decision rules is included and fmally the methodology is

illustrated by two examples

Methods for learning decision trees from examples have been very popular in machine learning

due to their simplicity Decision trees built this way can be quite efficient as long as they are

used in decision-making situations for which they are optimized and these situations remain

relatively stable Problems arise when these situations significantly change and the assumptions

under which the tree was built do not hold anymore For example in some situations it may be

difficult to determine the value of the attribute assigned to some node One would like to avoid

measuring this attribute and still be able to classify the example if this is potentially possible

(Quinlan 1990) If the cost of measuring various attributes changes it is desirable to restructure

the tree so that the inexpensive attributes are evaluated f11st A tree restructuring is also

desirable if there is a significant change in the frequency of occurrence of examples from

different classes A restructuring of a decision tree to suit the above requirements is however

difficult to do The reason for this is that decision trees are a form of decision structure

representation and imposes constraints on the evaluation order of the attributes that are not

logically necessary

29

One problem in developing a method for generating decision structures from decision rules is to

design an attribute selection criterion that is based on the properties of the rules rather than of

the training examples A decision rule normally describes a number of possible examples Only

some of them are examples that have actually been observed ie training examples An attribute

selection criterion is needed to analyze the role of each attribute in the rules It cannot be based

on counting the numbers of training examples covered by each attribute-value and the

frequency of decision classes in the training examples as is done in learning decision trees from

examples because the training examples are assumed to be unavailable

Another problem in learning decision trees from decision rules stems from the fact that decision

rules constitute a more powerful knowledge representation than decision trees They can directly

represent a description in an arbitrary disjunctive normal form while decision trees can represent

directly only descriptions in the disjoint disjunctive normal form In such descriptions all

conjunctions are mutually logically disjoint Therefore when transforming a set of arbitrary

decision rules into a decision tree one faces an additional problem of handling logically

intersecting rules

The solution to both problems (attribute selection and logically intersected rules) in the AQDT-2

system is based on the earlier work by Michalski (1978) which introduced a general method for

generating decision trees from decision rules The method aimed at producing decision trees

with the minimum number of nodes or the minimum cost (where the cost was defined as the

total cost of classifying unknown examples given the cost of measuring individual attributes and

the expected probability distribution of examples of different decision classes) More

explanations are provided in the following section

331 Tbe AQDT-2attribute selection metbod

This section describes the AQDT -2 method for building a decision structure from decision rules

The method for building a single-parent decision structure is similar to that used in standard

30

methods of building a decision tree from examples The major difference is that it assigns tests

(attributes) to the nodes using criteria based on the properties of the decision rules (this includes

statistics about the examples covered by each rule in the case of learning rules from examples)

rather than statistics characterizing the frequency of training examples per decision classes per

attribute-values or per conjunctions of both Other differences are that the branches may be

assigned an internal disjunction of values (not only a single value as in a typical decision tree)

and leaves may be assigned a set of alternative decisions with probabilities Also the tests can be

attributes or names standing for logical or mathematical expressions that involve several

attributes or variables In the following we use the terms test and attribute interchangeably

(to distinguish between an attribute and a name standing for an expression the latter is called a

constructed attribute)

At each step the method chooses the test from an available set of tests that has the highest utility

(see below) for the given set of decision rules This test is assigned to the node The branches

stemming from this node are assigned test values or disjoint groups of values (in the form of

logical disjunction if such occur in the rules subsumed groups of values are removed) Each

branch is associated with a reduced set of rules determined by removing conditions in which the

selected attribute assumes value(s) assigned to this branch If all rules in the reduced ruleset

indicate the same decision class a leaf node is created and assigned this decision class The

process continues until all nodes are leaf nodes If it is not possible to reduce further the rule set

because some attribute is declared as unavailable (infmite cost) then the leaf is assigned a set of

candidate decisions with associated probabilities (see Sec 42)

The test (attribute) utility is a combination of one or more of the following elementary criteria 1)

cost which indicates the cost of using each attribute for making decision 2) disjointness which

captures the effectiveness of the test in discriminating among decision rules for different decision

classes 3) importance which determines the importance of a test in the rules 4) value

31

distribution which characterizes the distribution of the test importance over its of values and 5)

dominance which measures the test presence in the rules These criteria are dermed below

~ The cost of a test expresses the effort or cost needed to measure or apply the test

Djsjointness The disjointness of a test is defined as the sum of the class disjointness-the

disjointness of the test for each decision class Suppose decision classes are Cl C2 bull Cm and

decision rule sets for these classes have been determined Given test A let V 1 V 2 bull V m denote

sets of the values (outcomes) of A that are present in rule sets for classes C 1 C2bullbull Cm

respectively If a ruleset for some class say Ct contains a rule that does not involve test A than

Vt is the set of all possible values of A (the domain of A)

Definition 3 -1 The degree ofclass disjointness D(A Ci ) of test A for the ruleset ofclass Ci is

the sum of the degrees of disjointness D(A Cit Cj) between the ruleset for Ci and rulesets for

Cj j=l 2 m j J L The degree of disjointness between the ruleset for Ci and the ruleset for Cj is

dermed by

if Vi Vj ifVj gtVj (3-1) ifVinVj -tljgt amp -tVi amp -tVj ifVinVj=1jgt

where cjgt denotes the empty set Note that exchanging the second and third conditions of equation

(3-1) may seem to be an improved criteria However it does not clearly distinguish between both

cases (Le for both situations the disjointness will be similar) The current equation is better

because it gives higher scores to attributes that classify different subsets of the two decision

classes than attributes that classify only a subset of one decision class

Definition 3-2 The disjointness of the test A for evaluating a given set of decision rules is the

sum of the degrees of class disjointness of each decision class

32

m m Disjointness(A) = L D(A cn where D(A cn = L D(A Ci Cj) (3-2)

i=l i=l i~j

The disjointness of a test ranges from 0 when the test values in rulesets of different classes are

all the same to 3m(m-l) when every rule set of a given class contains a different set of the test

values If two tests have the same disjointness value the attribute to be selected is the one with

less number of values

Definition The A vera~ Number of Tests (ANT) required to make decision from a decision tree

is dermed as the average number of tests (attributes) to be examined from the root of the tree to

any leaf node in order to reach a decision

Definition A decision structure is a one-node-per-level decision structure if at each level there is

only one node and zero or more leaves

Such decision structure can be generated by combining together all branches that associated with

each one a set of decision rules belongs to more than one decision class

Theorem 1 Consider learning one-node-level decision structure for a database with two

decision classes The disjointness criterion ranks first attributes that add minimum number of

tests to the decision tree

Proof Consider all possible distributions of an attributes values within two decision classes Ci

and Cj There are three cases 1) subset (same as super set) 2) non-empty intersection but not

subset 3) no intersection Figure 3-3 shows all possible distributions (the case of having the

same set of values in both classes is trivial one) Consider that branches leading to one subset

with the same decision class are combined into one branch In the first case there will be two

branches only The first leads to a leaf node and the other leads to an intermediate node where

another attribute to be selected The minimum ANT in this case is 53 In the second case three

branches should be created Two branches leads to a leaf node where all values at each branch

belong to only one and different decision class The third branch leads to an intermediate node

33

where another attribute should be selected furtur classifies the decision classes The minimum

ANT in this case is 64 In the third case only two branches will be generated where each leads

to leaf node with different decision class In this case the minimum ANT is 1

Figure 3-4 shows decision trees equivalent to selecting the corresponding attribute Note that in

case of having more than one attribute-value at branches lead to leaves belonging to one decision

class they will be combined into one branch in the decision structure The symbol 1 means

that an attribute is needed to classify the two decision classes In such cases there will be at least

two additional paths

D(A Ci) = 0 D(A CJ) = 1 D(A Ci) =2 D(A Cj) = 2 D(A Ci) =3 D(A Cj) =3

Figure 3-3 Yen Diagrams of Possible combinations of attribute values in two decision classes

The average number of tests required for making a decision in each possible case is determined

in Figure 3-4 It is clear that in the case of two decision classes the disjointness criterion ranks

highly attributes that reduces the average number of tests required for decision-making The

theorem can be proved in the general case

i ~ i Cj C1middot bull Ci Cj

ANT=32middotANT=53 ANT=l

1 means at least one attribute is needed to complete the decision tree Figure 3-4 Decision trees corresponding the Yen diagrams inFigure 3-3

34

Theorem 2 The attribute with the highest non~zero disjointness is the best attribute to classify

the decision classes

Proof Suppose that the number of decision classes is n Assume also that there are two attributes

A and B where D(A) lt D(B) Since the disjointness criterion considers the mutual relationship

between any two decision classes this means that there are more decision classes where D(A

Ci) lt D(B Ci) than those where D(A Ci) gt D(B Ci) (ignoring classes with equal disjointness

for bCth attributes) Hence there are more decision classes where D(A Ci Cj) lt D(B Ci Cj)

than those where D(A Ci Cj) gt D(B Cit Cj) Let us frrst prove that if D(A Ci Cj) lt D(B Cit

Cj) then B classifies the decision classes better than A

For each pair of decision classes Ci and Cj the possible values of the disjointness of any attribute

are 0 14 or 6 For all positive values ofD(B)= 14 or 6 it is clear that attribute B should have

smaller ANT than attribute A which has lower disjointness This means that if D(A Cit Cj) lt

D(B Cit Cj) then attribute B can classify both decision classes better than attribute A

Similarly B is better for classifying more pairs of decision classes than A This implies that B is

a better classifier than A

Importance The second elementary criterion the importance of a test is based on the

importance score (IS) introduced in (Imam Michalski amp Kerschberg 1993) In the obtained

rules each test is assigned a score that represents the total number of training examples that

are covered by the rules involving this test Decision rules learned by an AQ learning program

are accompanied with information on their strength Rule strength is characterized by t-weight

and u~weight The t-weight (total-weight) of a rule for some class is the number of examples of

that class covered by the rule The importance score of a test is the aggregation of the totalshy

weights of all rules that contain that test in their condition part Given a set of decision rules for

m decision classes Cl Cm and involving n tests AlbullAn and the number of rules associated

with class Ci is denoted by tlrit The importance score is defmed as follow

35

Definition 3 -3 The importance score IS(Aj) of the test A j is detennined by

m IS(Aj)= L IS(Aj Ci) (3-31)

i=l

where ri IS(Aj Ci) = L Rik(Aj) (3-32)

k=l

and Rik the weight of a test Aj in the rule Rk of class Ci is given by

t - weight if A j belongs to rule Rik Rik (Amiddot) = (3-4)J 0 otherwise

where i=I n ik=I ri j=I m

The importance score method has been separately compared as a feature selection method with

a genetic algorithm-based method (Imam amp Vafaie 1994) The importance score method

produced an equal or higher accuracy on three real-world problems than those reported by the

GA method while selecting fewer attributes In addition the IS method was significantly faster

than the GA

yalue distribution The third elementary criterion value distribution concerns the number of

legal values of tests Given two tests with equal importance scores this criterion prefers the test

with the smaller number of legal values Experiments have shown that this criterion is especially

useful when using discriminant decision rules

Definition 34 A value distribution VD(Aj) of a test A j is defined by

VD(Aj) = IS(Aj) I Vj (3-5)

where v is the number of legal values of Aj

Dominance The fourth elementary criterion dominance prefers tests that appear in large

numbers of rules as this indicates their high relevance for discriminating among the rulesets of

given decision classes Since some conditions in the rules have values linked by internal

disjunction counting such rules directly would not reflect properly their relevance Therefore

36

for computing the dominance the rules are counted as if they were converted to rules that do not

have internal disjunction Such a conversion is done by multiplying out the condition parts of

the rules containing internal disjunction For example the condition part [x3=1 v 3]amp[x4=1] is

multiplied out to two rules with condition parts [x3=1]amp[x4=1] and [x3=3]amp[x4=I]

The above criteria are combined into one general test measure using the lexicographic

evaluation functional with tolerances (LEF) (Michalski 1973) LEF is a list of some or all of

the above elementary criteria each associated with a Ittolerance threshold It in percentage The

criteria are applied to tests in the order defined by LEF A test passes to the next criterion only if

it scores on the previous criterion within the range defined by the tolerance (from the top value)

The default LEF is

ltCost d Disjointness t2 Importance t3 Value distr t4 Dominance 0gt (3-6)

where d a t3 t4 and t5 are tolerance thresholds (in percentages) their default values are O

The default value of the cost of each test is 1

The above LEF ranks attributes this way First attributes are evaluated on the basis of their cost

If two or more attributes have the same cost they ranked by their degree of disjointness If two

or more attributes still share the same top score or their scores differ less than the assumed

tolerance threshold t2 the method evaluates these attributes using the second (Importance)

criterion If again two or more attributes share the same top score or their scores differ less than

the tolerance threshold t3 then the third criterion normalized IS is used then similarly the

fourth criterion (dominance) If there is still a tie the method selects the best attribute

randomly

If there is a non-uniform frequency distribution of examples of different classes then the

selection criterion uses a modified definition of the disjointness Namely the previously defmed

37

disjointness for each class is multiplied by the frequency of the class occurrence The class

occurrence is the expected number of future examples that are to be classified to a given class

m

Disjointness(A) = L D(A CO Frq(Ci) (3-7) i=l

where Frq(Cn is the expected frequency of examples of class Cit and it is assumed to be given

by the user The attribute ranking criteria in this case is defined by the LEF

ltCost d Disjointness t2 Importance 13 Normalized-IS 14 Dominance 15gt (3-8)

where the Cost denotes the evaluation cost of an attribute and is to be minimized while other

elementary criteria are treated the same way as in the default version

332 The AQDT-2 algorithm

The AQDT-2 algorithm constructs a decision structure from decision rules by recursively

selecting at each step the best test according to the ranking criteria described above and

assigning it to the new node The process stops when the algorithm creates terminal branches that

are assigned decision classes To facilitate such a process the system creates a special data

structure for each concept description (ruleset) This structure has fields such as the number of

rules the number of decision classes and the number of attributes present in the rules A set of

pointers connect this data structure to set of data structures each represents one decision class

The decision class structure contains fields with information on the number of rules belong to

that class the frequency of the decision class etc It is also connected to a set of data structures

representing the decision rules within each decision class The system creates independently a set

of data structures each is corresponding to one attribute Each attribute description contains the

attributes name domain type the number of legal values a list of the values the number of

rules that contain that attribute and values of that attribute for each rule The attributes are

arranged in an array in a lexicographic order flISt in the descending order of the number of rules

38

that contain that attribute and second in the ascending order of the number of the attributes

legal values

The system can work in two modes In the standard mode the system generates standard

decision trees in which each branch has a specific attribute-value assigned In the compact

mode the system builds a decision structure that may contain

A) or branches Le branches assigned an internal disjunction of attribute-values whenever it

leads to simpler structures For example if a node assigned attribute A has a branch marked by

values 1 v 2 then the control passes along this branch whenever A takes value 1 or 2 The

program creates or branches on the basis of the analysis of the value sets Vi while computing

the degree of attribute disjointness

B) nodes that are assigned derived attributes that is attributes that are certain logical or

mathematical combinations of the original attributes To produce decision structures with

derived attributes the input decision rules are generated by AQ17 (rather than AQI5) The

AQ17 rules may contain conditions involving attributes constructed by the program rather than

those originally given

To generate decision structures from rules the AQDT-2 method prefers either characteristic or

discriminant disjoint rule descriptions (given by an expert or learned by a system) Disjoint rules

are more suitable for building decision structures Assume that the description of each class is in

the form of a ruleset and that this set is the initial ruleset context The AQDT algorithm is

The AQDT2 Algorithm

Step 1 Evaluate each attribute occurring in the ruleset context using the LEF attribute ranking

measure Select the highest ranked attribute Let A represents this highest-ranked

attribute

Step 2 Create a node of the tree (initially the root afterwards a node attached to a branch) and

assign to it the attribute A In standard mode create as many branches from the node as

39

there are legal values of the attribute A and assign these values to the branches In

compact mode (decision structures) create as many branches as there are disjoint value

sets of this attribute in the decision rules and assign these sets to the branches

Step 3 For each branch associate with it a group of rules from the ruleset context that contain a

condition satisfied by the value(s) assigned to this branch For example if a branch is

assigned values i of attribute A then associate with it all rules containing condition [A=

i v ] If a branch is assigned values i v j then associate with it all rules containing

condition [A= i v j v ] Remove from the rules these conditions If there are rules in

the ruleset context that do not contain attribute A add these rules to all rule groups

associated with the branches stemming from the node assigned attribute A (This step is

justified by the consensus law [x=1] == [x=1] amp [y=a] v [x=1] amp [y=b] assuming

that a and b are the only legal values of y) All rules associated with the given branch

constitute a ruleset context for this branch

Step 4 If all the rules in a ruleset context for some branch belong to the same class create a leaf

node and assign to it that class If all branches of the trees have leaf nodes stop

Otherwise repeat steps 1 to 4 for each branch that has no leaf

To select an attribute to be a node in the decision tree (step 1 and 2 of the algorithm) the

algorithm performs two independent iterations In the first iteration it parses through all decision

rules and determine information about each attributes This information includes the importance

score of each attribute the number of rules containing a given attribute the disjoint value sets of

each attribute and the attribute values used in describing each decision class The second

iteration is only performed if the disjointness criterion is ranked first in the LEF function The

second iteration evaluates the attributes disjointness for each decision class against the other

decision classes

40

To determine the complexity of this process suppose that the maximum number of conditions

fonned by a single attribute value in one rule is s (where s ~ n the number of attributes) and r is

the total number of decision rules (in all decision classes)

m

r=LRt (m is the number of decision classes) 1=1

where Ri is the number of rules in decision class Ci bull The complexity of the frrst iteration can be

determined as

Cmpx(Itcl) =O(r s)

In the second iteration the disjointness is calculated between the decision classes and all

attributes The complexity of the second iteration can be given by

Cmpx(Itc2) = O(n m)

Assume that at each node 1 is the maximum of the number of decision classes to be classified at

this node and the number of rules associated with this node

l=max mr (3-9)

Since these two iteration are independent of each other the complexity of the AQDT algorithm

for building one node in the decision tree say Node Complexity NC(AQDT) is given by

NC(AQDT) = 0(1 n)

Usually I equals to the number of rules associated with the given node Thus the AQDT

complexity for building one node is a function of the number of attributes multiplied by the

number of rules associated with this node

At each level of the decision tree these two iterations are repeated for each non-leaf node The

Level Complexity of the AQDT algorithm LC(AQDT) can be given by

LC(AQDT) lt 0(1 n)

Which is less than the complexity of generating the root of the decision tree To explain this

consider the maximum possible number of non-leaf nodes at one level is half the number of the

initial decision rules (integer of rll) Figure 3-5-a shows an example of such situation where at

41

the lowest level each node classifies only two rules each belongs to different decision class This

decision tree could be a multimiddotvalues tree However the number of non-leaf nodes at each level

will be twice or more than the number of nonmiddot leaf nodes at the previous level Considering the

level complexity of the AQDT algorithm equal to (1 s 0) where 0 is the number of non-leaf

nodes at the given level In such cases it is either (1 0 S r) or (1 s lt r) In Figure 3middot5middota the

complexity of the AQDT algorithm at any lower level is given by

LC(AQDT) =0(2 s r(2) lt O(n I) =NC(AQDT)

a) per one level b) per one path

Figure 3 -5 Decision trees showing the maximum number of none leaf nodes

Note also that after selecting an attribute to be the root of the decision structure this attribute

and all conditions containing that attribute are removed from the data structure of the algorithm

Also if a leaf node is generated all rules belonging to the corresponding branch will not be

tested again

Since the disjointness criterion selects the attribute which minimizes the average number of tests

ANT it is true that the AQDT algorithm generates decision trees with the least number of levels

The number of levels per a decision tree is supposed to be less than or equal to the minimum of

both the number of attributes and the number of rules Consider k as the number of levels in a

given decision tree

k S min mr (3-10)

There are two cases represent the most complex situations Figure 3-5-a and 3-5-b In the fIrst

case where the decision rules were divided evenly the number of levels will be a function of the

42

logarithm of the number of rules In such case the complexity of the AQDT algorithm for

generating a decision tree from a set of decision rules is given by

Complexity(AQDT) = 0(1 n log r) (3-11)

The other situation is when the generated decision tree has the maximum number of levels The

maximum possible number of levels per a decision tree equals to one less than the number of

decision rules Figure 3-5-b Using the disjointness criterion it is not likely to get such a decision

tree because it has the maximum average number of test (ANT) that can be determined from the

same set of nodes and leaves However such a decision tree can be generated if the number of

decision classes is one less thanthe number of attributes In such case any disjQint decision rules

should have a maximum length that is less or equal to the floor of the logarithm of the number of

attributes Thus the level complexity of this decision tree is estimated as

LC(AQDT) =0(1 log n)

The maximum number of levels in such decision tree is k-l Thus the complexity of the AQDT

algorithm in such cases is given by

Complexity(AQDT) = 0(1 k log n) (3-12)

Since k is the minimum of n and r then n is the maximum of k and n Also since in (3-12)

r=n+1 then log r is greater than log n From 3-10 3-11 and 3-12 the complexity of the AQDT

algorithm is determined by

Cmplx(AQDT) = OCr k log 1) (3-13)

333 An example illustrating the algorithm

The following simple example illustrates how AQDT is used in selecting the optimal set of

testing resources for testing a software Suppose there are three tools for testing a software 1)

modeling (Tl) 2) checklist (T2) and 3) par_simul (T3) Also assume that there are four

different factors that affect the selection of any tool 1) the cost of using the tool (xl) 2) the

metrics that support the tool (x2) 3) the best phase for applying the tool (x3) and 4) type of tool

43

(automated semi-automated or manual) (x4) Table 3-1 shows these attributes and their possible

values

Suppose that the domain expert provided a set of rules to be used in testing a software Figure 3shy

6 shows a sample of these rules in AQ15c formats

Table 3-1 The available tools and the factors that affect the prclCes

Tl lt= [xl=2] amp [x2=2] v [xl=3] amp [x3=1 v 3] amp [x4=1] T2 lt= [xl=l v 2] amp [x2=3 v 4] v [xl=3] amp [x3=1 v 2] amp [x4=2] T3 lt= [xl=l] amp [x2=1] v [xl=4] amp [x3=2 v 3] amp [x4=3]

Figure 3-6 Decision rules for selecting the best tool for testing software

These rules can be interpreted as

Rule 1 Use the fltst tool for testing if you need average cost and the tool is

supported by the requirement metric

Rule 2 Use the fIrst tool for testing if you can afford high cost for testing either

in the requirement or the analysis phases and you need an automated tool

Rule 3 Use the second tool for testing if the cost limit is low or average and the

tool is supported by the system usage metric or the intractability metric

Rule 4 Use the second tool for testing if you can afford high cost for testing

either in the requirement or the design phases and you need a manual tool

Rule 5 Use the third tool for testing if the you are limited to low cost and the

tool is supported by the error rate metric

Rule 6 Use the third tool for testing if you can afford very-high cost for testing

either in the requirement or the system usage phases and you need a semishy

automated tool

44

Table 3-2 presents information on these rules and the disjointness values for all attributes For

each class the row marked Values lists values occurring in the ruleset for this class For

evaluating the disjointness of an attribute say A each rule in the ruleset above that does not

contain attribute A is characterized as having an additional condition [A= a vb ] where a b

are all legal values of A

The row Class disjointness specifies the class disjointness for each attribute The attribute xl

has the highest disjointness (11) and is assigned to the root of the tree For simplicity assume

the tolerances for each elementary criterion equal O

From the rules in Figure 3-6 we can also determine disjoint groupings of attribute-values used

for the compact mode of the algorithm This done as follows 1) determine for each attribute the

sets of values that the attribute takes in individual decision rules and remove those value sets

that subsume other value sets The remaining value sets are assigned to branches stemming from

the node marked by the given attribute For example x 1 has the following value sets in the

individual decision rules 2 3 I 2 I and 4 (Figure 3-6) Value set I 2 is

removed as it subsumes 2 and I In this case branches are assigned individual values of the

domain of x1 For attribute x2 the value sets are I 2 34 and I 2 3 4 In this case

branches are assigned value sets I 2 and 34

45

Attribute Xl ranks highest (as it is the highest disjointness) and is assigned to the root of the

tree Four branches are created each one corresponding to one of XIs possible values Since all

rules containing [x1=4] belong to class TI the branch marked by 4 is ended by a leafTI Rules

containing other values of X1 belong to more than one class This process is repeated for each

subset of rules until the decision tree is completed Figure 3-7 shows a decision structure learned

by AQDT-2 from the rules in Figure 3-6 The decision structure in Figure 3-7 can be used in

making decisions on which tools can be used for testing a given software

xl

Complexity No of nodes 4 No of leaves 7

Figure 3-7 A decision structure learned for classifying so tware testing tools

Figure 3-8-a shows the diagrammatic visualization of the decision rules and Figure 3-8-b shows

the visualization of the derived decision tree Each diagram in Figure 3-8 consists of cells

representing one combination of attribute-values Attributes and their legal values are shown on

scales surrounding the diagram (eg the horizontal scale for xl shows values 1 2 3 and 4)

Rules are represented by collections of cells in the intersection of the rows and columns

corresponding to the conditions in the rules

The shaded areas correspond to decision rules Rules of the same class have the same type of

shading Empty cells correspond to combination of attribute-values not assigned to any class For

illustration collections of cells corresponding to some of the initial rules are marked R 11 R 21

R31 and R32 Rll denotes the IlfSt rule of Class T1 ie [x1=2] amp [x2=2] R21 denotes the

IlfSt rule of class T2 ie [xl= 1 v 2] amp [x2= 3 v 4] R31 the first rule of class TI ie [x1=1]

46

amp [x2 =1] and denotes R32 denotes the second rule of class T3 Le [xl=4] amp [x3=2 v 3] amp

[x4=3]

a) Decision rules b) Derived decision tree

Comparing diagrams in Figure 3-8-a and 3-8-b AQDT-2 decision tree represents a slightly more

general description of Concepts Tl TI and T3 than the original rules Let us assume that it is

very costly to know which metrics suppon the required tools In other words suppose that we

would like to select the best tools independently of the metric they suppon (this is indicated to

AQDT-l by assigning very high cost to attribute x2) The algorithm again assigns attribute xl to

the root of the tree When xl takes value 1 or 2 it is impossible to assign a specific decision

without measuring attribute x2 For the value 1 of xl the recommended tool can be either TI or

T3 (see the diagram in Figure 3-9-a) and for the value 2 of xl the recommended tool can be

either Tl or T3 However for the value 3 of xl one can make a specific decision after

measuring attribute x4 For the value 1 of x4 tool is Tl and for the value 2 of x4 the

recommended tool is TI Figure 3-9-b shows another decision tree where the type of the tool

was ignored from the data Those decision trees are called indeterminate because some of their

leaves are assigned a disjunction of two or more class names

47

xl

xl

1 12

a) Ignoring the supporting metric b) Ignoring the type of the tooL Figure 3 -9 Decision trees learned ignoring the support metric and the type of the testing tool

It is clear that the given set of rules depend highly on amount of money can be spent Let us now

suppose that the cost attribute xl which was determined as the highest ranked attribute cannot

be measured The algorithm selected x4 to be the root of the new decision tree After continuing

the algorithm the tree in Figure 3-10 is obtainedThe nodes that are assigned one class indicate

situations in which it is possible to make a specific decision without knowing the value of

attribute xl

x4

Figure 3-10 A decision tree learned without the cost attribute

34 Tailoring Decision Structures to the Decision-making situation

Decision structures are among the simplest structures for organizing a decision-making process

A decision structure specifies explicitly the order in which attributes of an object or situation

need to be evaluated in the process of determining a decision A standard way to generate a

48

decision structure is to learn it from examples of decisions Such a process usually aims at

obtaining a decision structure that has the highest prediction accuracy that is the highest rate of

assigning correct decisions to given situations There can usually be a large number of logically

equivalent decision structures (Michalski 1990) As such they may have the same predictive

accuracy but differ in the way they organize the decision process and thus may differ in the cost

of arriving at a decision To minimize the average decision cost one needs to take into

consideration the distribution of the costs of attribute evaluation and the frequency of different

decisions This report presents an approach to building such Task-oriented decision structures

which advocates that they are built not from examples but rather from decision rules Decision

rules are learned from examples using the AQ15 or AQ17 inductive learning program or are

specified by an expert An efficient algorithm and a new system AQDT-2 that transforms

decision rules to task-oriented decision structures The system is illustrated by applying it to the

problem of learning decision structures in the area of construction engineering (for determining

the best wind bracing for tall buildings) In the experiments AQDT-2 outperformed all other

programs applied to the same data

Decision-making situations can vary in several respects In some situations complete

information about a data item is available (ie values of all attributes are specified) in others the

information may be incomplete To reflect such differences the user specifies a set of parameters

that control the process of creating a decision structure from decision rules AQDT-2 provides

several features for handling different decision-making problems 1) generating a decision

structure that tries to avoid unavailable or costly attributes (tests) 2) generating unknown

leaves in situations where there is insufficient information for generating a complete decision

structure 3) providing the user with the most likely decision when performing a required test is

impossible 4) providing alternative decisions with an estimate of the likelihood of their

correctness when the needed information can not be provided

49

341 Learning Costmiddot Dependent Decision Structures

As described in Sec 2 the LEF criterion enables the system to take into consideration the cost of

measuring tests (attributes) in developing a decision structure In the default setting of LEF the

tests cost is the [lIst criterion and its tolerance is O This means that only the least expensive

attributes pass to the next step of evaluation involving other elementary criteria If an attribute

has high cost or is impossible to measure (has an infinite cost) the LEF chooses another

cheaper attribute ifpossible

342 Assigning Decision Under Insufficient Information

In decision-making situations in which one or more attributes cannot be measured the system

may not be able to assign a definite decision for some cases If no more information can be

obtained but a decision has to be made it is useful to know the probability distribution for

different candidate decisions (Smyth Goodman amp Higgins 1990) The most probable decision is

then chosen The probability distribution can be estimated from the class frequency at the given

node Let us consider a node in the structure connected to the root by the sequence bl b2 bkof

attribute-values Let P(Ci I bl b2 bk) denote the conditional probability of class Ci i=I2 m

at that node given that an example to be classified has attribute-values assigned to branches bl

b2 bull bk in the decision structure Using Bayesian formula we have

P(Ci I bl~ bull bk) = P(Ci) P(bl bk I Ci) P(bl bk) (3-9)

where P(Ci) the apriori probability of class Ci P(bl bk I Ci) is the probability that the

example has attribute-values blbull bk given that it represents decision class Ci To approximate

these probabilities let us suppose that Wi is the number of training examples of Class Ci that

passed the tests leading to this node and twi - the total number of training examples of Class

Ci Let us use these numbers to estimate the probabilities in (9) Assuming that the apriori class

probabilities correspond to the frequency of training examples from different classes we have

50

m P(Ci) = twi IL twj (3-10)

j=l

P(b1 bkl Ci) = Wi I twi (3-11)

m m P(b1 bk) =L Wj IL twj (3-12)

j=l j=1

By substituting (1O) (11) and (12) in (9) we obtain

m P(Ci I b1 bk) = wilL Wj (3-13)

j=l

A related method for handling the problem of unavailability of an attribute is described by

Quinlan (1990) Quinlans method however assigns probabilities to the most probable decision

at the node associated with such an attribute The method does not restructure the decision tree

appropriately to fit the given decision-making situation (in this case to avoid measuring xl)

AQDT -2 assigns probabilities o~ly after it first created a decision structure that suits the given

decision-making si~ation An example is presented in section 42

343 Coping with noise in training data

The proposed methodology can be easily extended to handle the problem of learning from noisy

training data This is done based on ideas of rule truncation described in (Niblett amp Bratko 1986

Bergadanoet al 1992 Cestnik amp Karalic 1991) When noise is expected in the training data

the decision rules with t-weight below a certain threshold (reflecting the expected noise level) are

removed The rule truncation method seems to result in more accurate decision structures than

decision tree pruning method because truncation decisions are solely based on the importance of

the given rule or condition for the decision-making regardless of their evaluation order (unlike

the decision tree pruning which can only prunes attributes within a subtree and thus cannot

freely chose attributes to prune) Examples are presented in section 4

TABLE OF CONTENTS

TI1LE Page

ABS1RAcr II 1

CHAPrER 1

IN1RODUCI10N II 3

11 Motivation and Overview 3

12 The Problem Statement 6

CHAPrER2

REIAfED RESEARCFI 7

21 Learning Decision Trees from Decision Diagrams 7

22 Learning Decision Trees from Examples 10

221 Building Decision Trees Using Information-based Criteria 11

222 Building Decision Trees Using Statistics-based Criteria 16

223 Analysis of Attribute Selection Criteria 18

23 Learning Decision Structures 19

CHAPrER3

DESCRIPTION OF TIlE APPROACFI 23

31 General Methodology 23

32 A Brief Description of the AQ15 and AQ17 RuleLeaming Systems 25

33 Generating Decision Structures From Decision Rules 28

331 The AQDT-2 attribute selection method 29

332 The AQDT-2 algoridlm 37

vi

333 An example illustrating the algorithm 42

34 Tailoring Decision Structure to a decision-making situation 47

341 Learning Cost-Dependent Decision Structures 49

342 Assigning Decision Under Insufficient Infonnation 49

343 Coping with noise in training data 50

35 Analysis of the AQDT-2 Attribute Selection Criteria 51

36 Decision Structures vs Decision Trees 53

CHAPTER 4

EMPRICAL ANALYSIS AND COMPARATIVE STUDY OF TIlE MElHOD 58

41 Description of the Experimental Analysis 59

42 Experiments With Average Size Complex and Noise-Free Problems

Wind Brac-ings 60

43 Experiments With Small Size Simple and Noise-Free Problems

e MONK-I 69

44 Experiments With Small Size Complex and Noise-Free Problems

MONK-2 76

45 Experiments With Small Size Simple and Noisy Problems

MONK-3 79

46 Experiments With Large Size Complex and Noise-Free Problems

Diagnosing Breast Cancer 83

47 Experiments With Large Size Complex and Noisy Problems

Musm()()m classifications 84

48 Experiments With Small Size Structured and Noise-Free Problems

East-West Trains 85

49 Experiments With Small Size Simple and Noisy Problems

vii

Congressional Voting Records (1984) 87

410 Analysis of the Results 88

ClIAPfER 5 CONCIUSIONS 95

51 Summary 95

52 Contributions 96

REFERENCES 98

VITA 102

viii

LIST OF TABLES

No TITLE Page

2-1 An example of a decision table 9

2-2 A set of training examples used to illustrate the C45 system 15

2-3 The frequency of different attributes values to different decision classes 17

2-4 The expected values of the frequency of examples in Table 2-3 17

2-5 Attribute selection criteria and their basic evaluation measure 17

2-6 The contingency tables of Mingers example 18

2-7 Mingers results for determining the goodness of split 19

2-8 Mingers results for comaring the total accuracy and size of decision trees

provided by different attribute selection criteria from four problems 19

2-9 Comparing the AQDT approach with EDAG and HOODG approaches 22

3-1 The available tools and the factors that affect the process of testing a software 43

3-2 Calculating the disjointness of each attribute 44

3-3 Evaluation of the AQDT-2 criteria on the Quinlans example in Table 2-2 51

3-4 The data used in Mingers f11st experiments 52

3-5 The performance of AQDT-2 criteria (compare with the other criteria in Table 2-6) 52

3-6 The possible ranking domains and using condition of AQDT-2 criteria 53

3-7 Comparison between Decision Structures and Decision Tree 54

4-1 Evaluation of AQDT-2 attribute selection criteria for the wind bracing problem 62

4-2 The predictive accuracy of AQ15c and AQDT-2 for the wind bracing problem 67

4-3 Evaluation of AQDT-2 attribute selection criteria for the MONK-l problem 71

4-4 The predictive accuracy of AQ15c and AQDT-2 for the MONK-l problem 73

ix

4-5 The predictive accuracy of AQ15c and AQDT-2 for the MONK-2 problem 77

4-6 The predictive accuracy of AQ15c and AQDT-2 for the MONK-3 problem 81

4-7 The set of attributes and their values used in the trains problem 86

4-8 A tabular snmmary of the predictive accuracy of decision trees obtained

by AQDT -2 and C45 for the congressional voting data 88

4-9 Summary of the best parameter settings for the first subfunction of the approach

with different data characteristics 89

4-10 Summary of the performance of AQDT-2 and C45 on different problems 90

x

LIST OF FIGURES

No TITLE Page

2-1 An example to illustrate how attributes break rules 8

2-2 A decision tree learned from the decision table in Table 2-1 10

2-3 A decision tree learned using the gain criterion for selecting attributes 15

2-4 Decision rules and their Exception Directed Acyclic Graph (EDAG) 21

3-1 Architecture of the AQDT approach 24

3-2 A ruleset generated by AQ15 for the concept Voting pattern of

Democratic Representatives 27

3-3 Ven Diagrams ofPossible combinations of attribute values in two decision classes 33

3-4 Decision trees corresponding the Ven diagrams in Figure 3-3 33

3-5 Decision trees showing the maximum number of none leaf nodes 41

3-6 Decision rules for selecting the best tool for testing a software 43

3-7 A decision structure learned for classifying software testing tools 45

3-8 A diagrammatic visualization of the decision rules and the derived decision tree 46

3-9 Decision trees learned ignoring the support metric and the type of the testing tool 47

3-10 A decision tree learned without the cost attribute 47

3-11 Decision structures learned by AQDT-2 using different criteria 55

3-12 The Imams example Example where learning decision structures (trees)

from rules is better than learning them from examples 56

3-13 An example where decision rules are simpler than decision trees 57

4-1 Design of a complete experiment 59

Xl

4-2 Decision rules determined by AQ15c from the wind bracing data 61

4-3 A decision tree learned by C45 for the wind bracing data 63

4-4 A decision structure learned from AQ15c wind bracing rules 64

4-5 A decision structure that does not contain attribute xl 64

4-6 A decision structure without Xl with candidate decisions assigned to leaves 65

4-7 A decision structure determined from rules in Figure 4-4 under the

assumption of 10 classification error in the training data 65

4-8 Diagramatic visualization of decision trees learned for different decision

making situations for the wind bracing data 66

4-9 The accuracy of AQDT-2 and AQ15c with different parameter settings

for the wind bracing PIoblem 68

4-10 Analyzing different parameter setting of AQDT-2 using the wind bracing data 69

4-16 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-20 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-11 A comparison between AQ15c and AQDT-2 against C45 on the wind bracing data 69

4-12 A visualization diagram of the MONK-l problem 70

4-13 Decision rules learned by AQ15c for the MONK-1 problem 71

4-14 The decision tree for the MONK-1 problem generated both by AQDT-l 72

4-15 Compact decision structures generated by AQDT-2 for the MONK-1 problem 72

for the MONK-l Problem 74

4-17 Analyzing different parameter setting of AQDT-2 with the MONK-l data 75

4-18 Comparing AQ15c and AQDT-2 against C45 using the MONK-1 problem 75

4-19 A visualization diagram of the MONK-2 problem 76

for the MONK-2 Problem 78

4-21 Analyzing different parameter setting of AQDT-2 with the MONK-2 data 79

xii

4-22 Comparing AQ15c andAQDT-2 against C45 using the MONK-2 problem 79

4-24 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-35 A visualization diagram of a decision tree learned by AQDT -2 after reducing

4-23 A visualization diagram of the MONK-3 problem 80

for the MONK-3 Problem 82

4-25 Analyzing different parameter setting of AQDT-2 with the MONK-3 data 82

4-26 Comparing AQ15c and AQDT-2 against C45 using the MONK-3 problem 83

4-27 Comparing AQ15c and AQDT-2 against C45 using the breast cancer problem 84

4-28 Comparing AQ15c and AQDT-2 against C45 using the mushroom problem 85

4-29 Decision structures learned by AQDT-2 for different decision-making situations 87

4-30 Comparing decision trees for the Cong Voting-84 data learned by C45 amp AQDT -2 88

4-31 A visualization diagram of decision rules learned by AQ15c for the MONK-2 91

4-32 A visualization diagram shows the testing errors for the rules in Figure 4-31 92

4-33 A visualization diagram of the decision tree learned by AQDT-2 for the MONK-2 93

4-34 A visualization diagram shows testing errors of AQDT-2 decision tree bull 94

the generalization degree to 1 94

DERIVING TASKmiddotORIENTED DECISION STRUCTURES

FROM DECISION RULES

Ibrahim M Fahmi Imam PhD

School of Information Technology and Engineering

Geotge Mason University Fall 1995

Ryszard S Michalski Advisor

ABSTRACT

This dissenation is concerned with research on learning task-oriented decision structures from

decision rules The philosophy behind this research is that it is more appropriate to learn

knowledge and store it in a declarative form and then when a decision making situation occurs

generate from this knowledge the decision structure that is most suitable for the given decision

making situation Learning decision structures from decision rules was first introduced by

Michalski (1978) The flIst implementation of this approach was done by Imam and Michalski

(1993 ab) called AQDT-l

This approach separates the function of generating a knowledge-base from the function of using

the knowledge-base for decision-making The first function focuses on learning accurate

consistent and complete concept description expressed in a declarative form The second function

is performed whenever a new decision-making situation occurrs a task-oriented decision structure

is obtained to suit that situation Thsk-oriented knowledge is defined as knowledge that is adapted

for solving a given decision-making situation (Imam amp Michalski 1994 Michalski amp Imam

1994)

The dissertation introduces the system AQJJT-2 for learning task -oriented decision structures from

decision rules or examples Each decision making situation is defined by a set of parameters that

controls the learning process of the AQDT-2 system The extensive experiments on AQJJT-2 show

that decision structures learned by it usually outperform in terms of accuracy and average size of

the decision structures those learned from examples by other well known systems The results

show also that the system does not work very well with noisy data The system is illustrated and

compared using applications of artificial problems such as the three MONKs problems (Thrun

Mitchell amp Cheng 1991) and the East-West 1iain problem (Michie et al 1994) It was also

applied to real world problems of learning decision structures in the areas of construction

engineering (for determining the best wind bracing design for tall buildings) medical

diagnosis(for learning decision rules for recognizing breast cancer) agriculture diagnosis (for

learning classification rules for distinguishing between poisonous and non-poisonous

mushrooms) and political data (for characterizing democratic and republican voting records)

CHAPTER 1 INTRODUCTION

11 Motivation and Overview

Learning and discovery systems should be able not only to generate and store knowledge but also

to use this knowledge for decision-making The main step in the development of systems for

decision-making is the creation of a knowledge structure that characterizes the decision-making

process The form in which knowledge can be easily obtained may however differ from the form

in which it is most readily used for decision-making It is therefore important to identify the form

of knowledge representation that is most appropIiate for learning (eg due to ease of its

modification) and the form that is most convenient for decision making

A simple and effective tool for describing decision processes is a decision structure which is a

directed acyclic graph that specifies an order of tests to be appliedto an object (or a situation) to

arrive at a decision about that object The nodes of the structure are assigned individual tests

(which may correspond to a single attribute a function of attributes or a relation) the branches are

assigned possible test outcomes or ranges of outcomes and the leaves are assigned a specific

decision a set of candidate decisions with corresponding probabilities or an undetermined

decision A decision structure reduces to a familiar decision tree when each node is assigned a

single attribute and has at most one parent when the branches from each node are assigned single

values of that attribute and when leaves are assigned single definite decisions Thus the problem

of generating a decision structure is a generalization of the problem of generating a decision tree

Decision trees are typically generated from a set of examples of decisions The essential

characteristic of any such method is the attribute selection criterion used for choosing attributes to

be assigned to the nodes of the decision tree being built Such criteria include the entropy

reduction the gain and the gain ratio (Quinlan 1979 83 86) the gini index ofdiversity (Breiman

et al 1984) and others (Cestnik amp Bratko 1991 Cestnik amp Karalic 1991 Mingers 1989a)

3

4

Adecision treedecision structure representations can be an effective tool for describing a decision

process as long as all the required tests can be performed and the decision-making situations it

was designed for remain constant (eg in a doctor-patient example the doctor should detennine

that answers for all symptoms appear in the decision tree) Problems arise when these assumptions

do not hold For example in some situations measuring certain attributes may be difficult or costly

(eg in the doctor-patient example a brain or blood test is needed which is very expensive or the

tools needed are not available) In such situations it is desirable to refonnulate the decision

structure so that the inexpensive attributes are evaluated frrst (assigned to the nodes close to the

root) and the expensive attributes are evaluated only if necessary (by assignment to the nodes far

away from the root) If an attribute cannot be measured at all it is useful to either modify the

structure so that it does not contain that attribute or-when this is impossible-to indicate

alternative candidate decisions and their probabilities A restructuring is also desirable if there is a

significant change in the frequency of occurrence of different decisions (eg in the doctor-patient

example the doctor may request a decision structure expressed in a specific set of symptoms

biased to classify one or more diseases or specify a certain order of testing)

Arestructuring ofa decision structure (or a tree) in order to suit new requirements could be quite

difficult This is because a decision structure is a procedural representation that imposes an

evaluation order on the tests In contrast no evaluation order is imposed by a declarative

representation such as a set of decision rules Tests (conditions) of rules can be evaluated in any

order Thus for a given set of rules one can usually build a huge number of logically equivalent

decision structures (trees) which differ in the test ordering Due to the lack of order constraints

a declarative representation (rules) is much easier to modify to adapt to different situations than a

procedural one (a decision structure or a tree) On the other hand to apply decision rules to make a

decision one needs to decide in which order tests are evaluated and thus needs a decision

structure

An attractive solution to these opposite requirements is to acquire and store knowledge in a

declarative form and transform it to a decision structure when it is needed for decision-making

5

This method allows one to create a decision structure that is most appropriate in a given decisionshy

making situation Because the number of decision rules per decision class is usually small (each

rule is a generalization of a set of examples) generating a decision structure from decision rules

can potentially be perfonned much faster than by generation from training examples Thus this

process could be done on line without any delay noticeable to the user Such virtual decision

structures are easy to tailor to any given decision-making situation

This approach allows one to generate a decision structure that avoids or delays evaluating an

attribute that is difficult to measure in some decision-making situation or that fits well a particular

frequency distribution of decision classes In other situations it may be unnecessary to generate

complete decision structure but it may be sufficient to generate only a part of it which concerns

only with decision classes of interest Thus such an approach has many potential advantages

This dissertation presents a new system called AQflf-2 The AWf-2 system generates a taskshy

oriented decision structure (decision structure that is adapted to the given decision-making

situation) from decision rules The decision rules are learned by either rule leaming system AQ15

(Michalski et al 1986) or system AQ17-DCI which has extensive constructive induction

capabilities (Bloedorn et al 1993)

To associate the decision rules with a given decision-making task AQflf-2 provides a set of

features including 1) enabling the system to include in the decision structure nodes corresponding

to new attributes constructed during the process of learning the decision rules 2) controlling the

degree of generalization needed during the development of the decision structure 3) providing four

new criteria for selecting an attribute to be a node in the decision structure that allow the system to

generate many different but equivalent decision structures from the same set of rules 4) generating

unknown nodes in situations when there is insufficient infonnation for generating a complete

decision structure 5) learning decision structures from discriminant rules as well as

characteristic rules and 6) providing the most likely decision when the decision process stops

due to inability to evaluate an attribute associated with an intennediate node

6

To test the methooology of generating decision structures from decision rules an extensive set of

planned experiments have been designed to test different aspects of the approach The experiments

include testing different combinations of parameters for each sub-function of the approach

analyzing the relatioship between decision rules and the decision structure learned from them and

comparing decision trees learned by the AQUf2 system the well known C45 (Quinlan 1993)

system for learning decision trees from examples Different experiments were designed to examine

the new features of the AQUf approach for learning task-oriented decision structures The

experiments were applied to artificial domains as well as real-world domains including MONK-I

MONK-2 and MONK-3 (Thrun Mitchell amp Cheng 1991) East-West trains (Michie et al

1994) Engineering Design-wind bracings (Arciszewski et al 1992) Mushrooms Breast

Cancer (Mangasarian amp WolbeJg 1990) and Congressional bting Records of 1984 The

MONKs problems are concerned with learning classification rules for robot-like figures MONK-l

requires learning a DNF-type description MONK-2 requires learning a non-DNF-type description

(one that cannot be easily described as a DNF rule using the original attributes) MONK-3 concerns

learning a DNF rule from noisy data The East-West trains dataset is a structural domain that

classifies two sets of trains (Eastbound and Westbound) The Engineering Design-wind

bracing data involves learning conditions for applying different types of wind bracing for ta11

buildings The Mushrooms data is concerned with learning classification rules for distinguishing

between poisonous and non-poisonous mushrooms The Breast Cancer data involves learning

concept descriptions for recognizing breast cancer The congressional voting data includes voting

records on different issues AQUf2 outperformed C45 on the average with respect to both the

predictive accuracy and tree size for most problems Apf-2 did not work very well with noisy

data or with problems that have many rules covering very few examples

12 The Problem Statement

There are many limitations and problems accompanied using decision trees for decision-making

CHAPTER 2 RELATED RESEARCH

21 Learning Decision Trees from Decision Diagrams

The AQDT-2 method proposed here is based on the earlier work by Michalski (1978) which

introduced an algorithm for generating decision trees from decision lists The method proposed

several attribute selection criteria These criteria are of increasing power of the main criterion

order cost estimate (the nth order cost estimates n=l 2 ) Michalski also analyzed two

specific criteria MAL and DMAL for selecting the optimal attribute for a node in the tree

based on properties extracted from the decision diagram In order to better explain the method

it is necessary to define some terms These terms will be used later in the dissertation

Definition 2-1 A ~ of a decision class C is a set of rules whose union includes the set of all

examples of class C and does not include any examples of other decision classes

Definition 2-2 A ~ of a decision class C is disjoint if all rules (rules) are pairwise logically

disjoint In other words for any two rules there exists a condition with the same attribute but

with different values in each rule

Definition 2-3 An minimal cover is a disjoint cover which has the smallest number of rules

among all possible covers

Definition 2-4 A diagram for a given cover is a table constructed graphically by representing

in two dimension space of all possible combination of attributes values and locating on the

diagram all the condition parts of the given rules and marking them with the action specified

with each rule

Michalskis algorithm requires construction of minimalmiddot cover The minimal cover should be

consistent and complete The method is based on the fact that if there are n decision classes

any decision tree that correctly classifies the given rules and has n distinct leaves is a minimal

7

8

decision tree (any consistent decision tree should have at least n leaves) Michalski (1978) has

shown that if only one rule is broken by a selected attribute then instead of having one leaf

(which could potentially represent this rule or the decision class in the tree) there will have to

be at least two leaves representing this rule in the fmal decision tree

The attribute selection criterion MAL introduced in (Michalski 1978) prefers attributes that do

not break any rules or break as few as possible An attribute breaks a rule if the attribute can

divide the rule into two or more sub-rules Figure 2-1 shows two examples of two sets of rules

In the fJIst xl and x3 break at least two rules each (xl breaks the rule [x4=2] amp [x1=lv3] amp

[x3=2v3] and the rule [x4=3] amp [x1=1v2] amp [x3=1] and x3 breaks three rules the rule [x4=2]

amp [xl=2] amp [x3=2v3]the rule [x4=l] amp [xl=3] amp [x3=lv3] and the rule [x4=3] amp [xl=4] amp

[x3=2v3]) In the diagram on the right xl is the only attribute that does not break any rule

Figure 2-1 An example to illustrate how attributes break rules

One of the criteria defmed by Michalski is the first degree cost estimate which assigns to each

attribute an integer equal to the number of rules broken by that attribute This criterion is also

called the static cost estimate of an attribute or the criterion of minimizing added leaves

(MAL)

-

C3

9

The MAL criterion (Minimizing Added Leaves) seeks an attribute that minimizes the

estimated number of additional nodes in the decision tree being generated over a hypothetical

minimal decision tree When there is a tie between two attributes the attribute to be selected is

the one which breaks smaller rules (rules that cover fewer examples or more specialized

rules) AQDT-2 uses an approximate version of this criterion (the attribute dominance)

Another criterion introduced by Michalski was the DMAL criterion The DMAL criterion

(Dynamically Minimizing Added Leaves) is based on a principle similar to that of MAL but

is more complex because once an attribute is selected as a node in the tree some rules andor

parts of the broken rules at each branch are merged into one rule The DMAL ensures that the

value of the total cost estimate of an attribute is decreased by a value equal to the number of

merged rules minus one

Example Learn a decision tree from the following decision table

The minimal cover consists of the following rules

Al lt [x2=O] v [x1=0][x2=2] A2 lt [x2=I] v [xl=2][x2=2] A3 lt [xl=I][x2=2]

The evaluations ofthe MAL criterion for these attributes are 2 for xl 0 for x2 5 for x3 and 5

for x4 The attribute xl divides the two rules [x2=O] and [x2=I] so selecting it will add two

leaves to the optimal number of leaves It is clear that the attribute to be selected as a root of

10

the decision tree is x2 Then three branches are attached to the root node and the decision rules

are divided into subsets each corresponding to one branch For x2 = 0 or 1 a leave node is

generated For x2=2 another attribute is selected to be a node in the tree In this case xl has the

minimum MAL value Figure 2-2 shows the decision tree obtained using the MAL criterion

Al A3 A2

Figure 2-2 A decision tree learned from the decision table in Table 2-1

22 Learning Decision Trees from Examples

Decision tree learning is a field that concerned with generating decision tree that classifies a set

of examples according to the decision classes they belong to The essential aspect of any

inductive decision tree method is the attribute selection criterion The attribute selection

criterion measures how good the attributes are for discriminating among the given set of

decision classes The best attribute according to the selection criterion is chosen to be assigned

to a node in the tree The fIrst algorithm for generating decision trees from examples was

proposed by Hunt Marin and Stone (1966) Hunts algorithm uses a divide and conquer

algorithm for building decision trees This algorithm has been subsequently modified by

Quinlan (1979) and applied by many researchers to a variety of learning problems

Attribute selection criteria can be divided into three categories These categories are logicshy

based information-based and statistics-based The logic-based criteria for selecting attributes

use logical relationships between the attributes and the decision classes to determine the best

attribute to be a node in the decision tree such as the MAL criterion minimizing added leaves

(Michalski 1978) which uses conjunction and disjunction operators The information-based

criteria are based on the information theory These criteria measure the information conveyed

11

by dividing the training examples into subsets Examples of such criteria include the

information measure 1M the entropy reduction measure and the gain criteria (Quinlan 1979

83) the gini index of diversity (Breiman et al 1984) Gain-ratio measure (Quinlan 1986) and

others (Clark amp Niblett 1987 Bratko amp Lavrac 1987 Cestnik amp Karalic 1991) The

statistics-based criteria measure the correlation between the decision classes and the attributes

These criteria use statistical distributions for determining whether or not there is a correlation

The attribute with the highest correlation is selected to be a node in the tree Examples of

statistics-based criteria include Chi-square and G-statistic (Sokal amp Rohlf 1981 Han 1984

Mingers 1989a)

Niblett and Bratko (1986) Quinlan (1987) and Bratko and Kononeko (1987) extended the

method of learning decision trees to also handle data with noise (by pruning) Handling noise

extended the process of learning decision trees to include the creation of an initial complete

decision tree tree pruning which is done by removing subtrees with small statistical Validity

and replacing them by leaf nodes (MiDgers 1989b) More recently pruning has also been used

for simplifying decision trees even for problems without noise (Bohanec amp Bratko 1994)

Pruning decision trees improves their simplicity but reduces their predictive accuracy on the

training examples Quinlan (1990) proposed also a method to handle the unknown attributeshy

value problem by exploring probabilities of an example belonging to different classes

The rest of this section includes a brief description of the attribute selection criterion used by

C45 learning system (Quinlan 1993) C45 uses an information-based criterion for selecting

an attribute to be a node in the tree Also the section includes a brief description of the Chishy

square method for attribute selection (Mingers 1989a) The method is a statistics-based

method for selecting an attribute to be a node in the tree

221 Building Decision Trees Using htformation-based Criteria

12

This section presents a description of the inductive decision tree learning system C4S The

C4S learning system is considered to be one of the most stable accurate and fastest program

for learning decision trees from examples

Learning decision trees from examples requires a collection of examples Each example is

represented by a fixed number of attribute-value pairs C45 (Quinlan 1993) is a learning

program that induces classification decision trees from a set of given examples The C4S

learning system is descended from the learning system ID3 (Quinlan 1979) which is based on

Hunts method for constructing decision tree from a set ofcases (Hunt Marin amp Stone 1966)

The C45 system uses an attribute selection criterion called the Gain Ratio This criterion

calculates the gain in classifying information based on the residual information needed to

classify cases in a set of training examples and the information yielded by the test based on the

relative frequencies of the possible outcomes (decision classes) The gain ratio criterion is

based on an earlier criterion used by ID3 called the Gain Criterion The Gain Criterion uses

the frequency of each decision class in the given set of training examples

Once an attribute is chosen to be a node in the tree the system generates as many links as the

number of its values and classifies the set of examples based on these values If all the

examples at a certain node belong to one decision class the system generates a leaf node and

assigns it to that class Otherwise the system searches for another attribute to be a node in the

tree

The Gain Criterion The gain criterion is based on the information theory That is the

information conveyed by a message depends on its probability and can be measured in bits as

minus the logarithm (base 2) of that probability To explain the gain criterion suppose that for

a given problem Xu bullbull X are given attributes and Cu c are decision classes Suppose S is

any set of cases and T is the initial set of training cases The frequency of class C1 in the set S

is the number of examples in S that belong to class C i bull

13

freq(Cit S) = Number of examples in S belong to CI (2-1)

Suppose that lSI is the total number of examples in S the probability that an example selected

at random from S belongs to class Ci is freq(CIbullS) IISI

The information conveyed by the message that a selected example belongs to a given decision

class Cit is determined by -log1 (freq(Ch S) IISI) bits

The expected information from such a message stating class membership is given by

info(S) = -k

L (freq(C S) I lSI) log1 (freq(C S) IISI) bits (2-2)I

info(S) is known also as the entrQpy of the set S When S is the initial set of training examples

info(T) determines the average amount of information needed to identify the class of an

example in T

Suppose that we selected an attribute X to be the root of the tree and suppose that X has k

possible values The training set T will be divided into k subsets each corresponding to one of

Xs values The expected information of selecting X to partition the training set T infoz(T) can

be found as the sum over all subsets of multiplying the information conveyed by each subset

by its probability

k

infoz(T) =L (ITII 1m) infO(TIraquo (2-3) I

The information gained by partitioning the training examples T into subset using the attribute

X is given by

gain(X) =inlo(T) - inoIT) (2-4)

The attribute to be selected is the attribute with maximum gain value

The Gain Ratio Criterion This criterion indicates the proportion of information generated by

the split that appears helpful for classification Quinlan (1993) pointed out that the gain

criterion has a serious deficiency Basically it is strongly biased toward attributes with many

outcomes (values) For example for any data that contains attributes such as social security

14

number the gain criterion will select that attribute to be the root of the decision tree However

selecting such attributes increases the size of the decision tree Quinlan provided a solution to

this problem by introducing the gain ratio criterion which takes the ratio of the information that

is gained by partitioning the initial set of examples T by the attribute X to the potential

information generated by dividing T into n subsets

Following similar steps to obtain the information conveyed by dividing T into n subsets The

expected information generated by dividing T into n subsets and by analogy to equation 2-2 is

determined by

split info(T) = - Lt

(ITII lTI) 10gl (ITII m ) (2-5) I

The gain ratio is given by

gain ratio(X) = gain(X) I split info(X) (2-6)

and it expresses the proportion of information generated by the split that is useful for

classification

Example Consider the following example presented by Quinlan (1993) Table 2-2 shows the

set of training examples

First determine the amount of information gained by selecting the attribute outlook to be a

root of the decision tree This attribute divides the training examples into three subsets

sunny-with five examples two of which belong to the class Play overcast-with four

examples all of which belong to the class Play and rain-with five examples three of

which belong to the class Play To determine info(11 the average information needed to

identify the class of an example in T there are 14 training examples and two decision classes

Nine of these examples belong to the class Play and five belong to the class Dont Play

Info(11 = - 914 10gl (914) - 514 10gl (514) = 094 bits

When using outlook to divide the training examples the information becomes

15

infO(T) = 514 -25 log2 (25) - 35 10g2 (35)

+ 414 -44 10g2 (44) - 04 log2 (04)

+ 514 -35 logl (35) - 25 logl (25) = 0694 bits

By substituting in equation 2-4 the gain of information results from using the attribute

outlook to split the training examples equal to 0246 The gain information for windy is

0048

Figure 2-3 shows a decision tree learned for this problem using the gain criterion The split

information for outlook is determined as follows

split info(T) = - 514 logl (514) - 414 log2 (414) - 514 log2 (514) = 1577 bits

The gain ratio for outlook = 02461577 =0156

16

Figure 2 -3 A decision tree learned using the gain criterion for selecting attributes

The C45 system handles discrete values as well as continuous values To handle an attribute

with continuous values C45 uses a threshold to transform the continuous domain into two

intervals In other words for each continuous attribute C45 generates two branches one

where the values of that attribute is greater than the determined threshold and the other if the

value is less than or equal to the threshold

Tree pruning in C45 is a process of replacing subtrees with small classification validity by

leaves The C45 system uses Laplace ratio for determining the error rate of different subtrees

This ratio is defined as (e+l) (n+2) where n is the number of the training examples and e is

the number of misclassified examples at a given leaf

222 Building Decision Trees Using Statistics-based Criteria

The Chi-square method for selecting attributes was used by Hart (1984) and Mingers (1989a)

in building decisionmiddot trees The method uses Chi-square statistics to measure the association

between two attributes When building decision trees the method is implemented such that it

determines the association between each attribute and the decision classes The attribute to be

selected is the one with greatest value

To determine the Chi-square value for an attribute consider aij is the number of examples in

class number i where the attribute A takes value numberj In other words aij is the frequency

of the combination of decision class number i and the attribute value number j The Chi-square

value for attribute A is given by

n m

Chi-square (A) =L L [ (aij - Eijl2 Eij ] (2shy

i=1 j=1

where n is the number of decision classes m is the number ofvalues of a given attribute Also

17

Eij = (TCi TVj ) T (2shy

8)

where T Ci and T Vj are the total number of examples belong to the decision class a and the total

number of examples where the attribute A takes value vj respectively T is the total number of

examples

Consider Quinlans example in Table 2-2 Table 2-3 shows the frequency of different

combination of values between the decision class and both the outlook and the Windy

attributes Table 2-4 shows the expected values TCi and T Vj of the frequencies in Table 2-3

of different attributes values for different decision classes

To determine the association value between the decision classes and both the attribute Windy

and the attribute Outlook the observed Chi-square values are

Chi-square (Windy Class) =[(3-39)239] + [(3-21)221] + [(6-51)232] + [(2-29)232]

=021 + 039 + 025 + 025 = 11

Chi-square (Outlook Class) = [(2-32)232] + [(4-26)226] + [(3-32)232] + [(3-18118]

+ [(0-14)214] + [(2-18)2 18] =045 + 075 + 02 + 08 + 14 + 002 = 362

18

Applying the same method to the other attributes the results will favor the attribute Outlook

Once that attribute is selected to be a node in the tree the remaining set of examples are divided

into subsets and the same process is repeated on each subset

Table 2-5 shows a summary of these criteria and their basic evaluation function

Table 2-5 Attribute selection criteria and their basic evaluation measure

Info Measure (IM) Gain Gshystatistics and Gain Ratio Chi-square

II

Entropy(S) = - L (freq(~ 5) 1151) logz (freq(CIt 5) 1151 ) I

G-Statistics =2N 1M (N-No of examples) n m

Chi-square (A B) = L L [(aij - ~j)21Eij ] i=l

223 Analysis of Attribute Selection Criteria

This subsection introduces briefly the analysis of different selection criteria which was done by

Mingers (1989a) Mingers compared six attribute selection criteria that are used in decision

tree programs These criteria are the Information Measure (IM) Chi-square G statistics Gin

index of diversity Marshal correction and Gain RaJio The overall results show that the Gain

Ratio criterion had the strongest xesults

In the flISt experiment Minger tested the six criteria on a problem with ambiguous examples

(ie examples may belong to more than one decision class) to observe how the selected criteria

evaluate the given attributes The problem has two decision classes and two attributes X and

Y It was assumed that attribute X is better for classifying the examples than attribute Y The

training examples were unevenly spread between the two values of X Attribute Y has three

values and the examples were spread randomly among them Table 2-6 (a and b) shows the

contingency tables for both attributes Table 2-7 shows a summary of the goodness of split

provided by the six criteria Mingers noted that the measures that are not based on information

19

theory give radiation (attribute X here) less weighL This may be because the zero in the fIrst

row of radiation has a greater influence in the log calculation In the case of using the Chishy

square criterion the value zero adds the maximum association between any two attributes

because the Chi-square value of a zero cell is the expected value of this cell

b)

Now let us demonstrate results from another experiment done by Mingers In this experiment

Mingers used four different data sets to generate decision trees for eleven different criteria In

the final results he compared the total number of nodes and the total error rate provided by

each criterion over all given problems Table 2-8 shows the final results for five selected

criteria only

20

U lgt ~middot results for comparing the total accuracy and size of decision trees different attribute selection criteria from four VVLU

This experiment was performed on four real world data sets These data are concerned with

profiles of BA Business Studies degree students recurrence of breast cancer classifying types

of Iris and recognizing LCD display digits The data was divided randomly 70 for training

and 30 for testing For more details see Mingers (1989a)

23 Learning Decision Structures

Considering the proposed defInition of decision structures given above two related lines of

research are described in this section Gaines (1994) and Kohavi (1994) proposed two

approaches for generating decision structures that share some of the earlier ideas by Imam and

Michalski (1993b)

In the first approach Brian Gaines introduces a method for transforming decision rules or

decision trees into exceptional decision structures The method builds an Exception Directed

Acyclic Graph (EDAG) from a set of rules or decision trees The method starts by assigning

either a rule say RO or a conclusion say CO or both to the root node If it assigns a conclusion

to the root node it places it on a temporary conclusion list Then it generates a new child node

and add to it a new rule say Rl The method evaluates the new rule Rl if it satisfies the

conclusion CO in its parent node it builds a new child and repeats the processes Otherwise if

the rule Rl does not satisfy the conclusion CO it changes the temporary memory with the new

conclusion Cl which is satisfied by the rule RO The same process is repeated until all rules

that have common conditions with the rule at the root are evaluated The method then creates a

21

new child node from the root and repeat the process until all rules are evaluated In the decision

structure nodes containing rules only represent common conditions to all of its children

The main disadvantages of this approach is that it requires discriminant rules to build such a

decision structure Also such a structure is more complex than the traditional decision trees

that are used for decision-making

Figure 2-4 shows an example of set of rules and their equivalent exceptional decision structure

The decision structure can be read as follows It is Safe except if xl=1 amp x2=1 amp x3=1 and

either x4=3 or x5=I then it is Lost except if x6=1 it is Safe except if x7=1 it is Lost

The second approach introduced by Ronny Kohavi learns decision structures from examples

using a bottom-up algorithm The method uses a greedy Hill-climbing algorithm for inducing

Oblivious read-Once Decision Graphs (HOODO) A read-once decision graph is defined as a

decision graph where each attribute occurs at most once along any computational path In other

words for each path from the root to the leaves of the decision structure attributes may occur

as a node once at most However there may be more than one node with the same attribute in

the decision graph Kohavi also defined a leveled decision graph in which the nodes are

partitioned into a sequence of pairwise disjoint sets such that outgoing edges from each level

terminate at the next level An oblivious decision graph is a decision graph where all nodes at

a given level are labeled by the same attribute

22

Safe lt [xl=2] Safe lt [x2=2] Safe lt [x3=2] Safe lt [x4=1] amp [x5=2] Safe lt [x4=1] amp [x5=3] Safe lt [x6=l] amp [x7=2] Safe lt [x6=1] amp [x7=3] Safe lt [x4=2] amp [x5=2] Safe lt [x4=2] amp [x5=3]

Lost lt [x6=2] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x6=3] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x5=1] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=2] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=3] amp [x3=1] amp [x2=I] amp [xl=l]

The algorithm starts by generating a leaf node for each decision class Then it applies a

nondeterministic method to select an attribute to build new level of the decision structure This

attribute is removed from the data and the data is divided into subsets each corresponding to a

complex combination of that attributes values For each subset the process is repeated until all

the examples of a given subset belong to one decision class For example if the selected

attribute say A has two values (0 and 1) and there are two decision classes (CO and CI) The

data is divided into two subsets the fIrst subset contains examples where A takes value 0 and

belong to class CO or takes value 1 and belong to class Cl The second subset of examples is

the set where A takes value 0 and belong to class CI or takes value 1 and belong to class CO

The number of nodes of the fIrst level (after the leave nodes) is expected to be less than or

equal to ka where k is the number of decision classes and n is the number of values of the

selected attribute and it can increase exponentially before that number is reduced

exponentially to one

It is easy for the reader to fIgure out some major disadvantages of such approach including

The average size of such decision structures is estimated to be very large especially when there

23

is no similarity (ie strong patterns) or logical relationship in the data The time used to learn

such decision structure is relatively very high comparing to systems for learning decision trees

from examples and fmally it could be better to search for attribute which reduces the number

of generated subsets of the data instead of nondeterministically selecting an attribute to build a

new level of the decision structure Kohavi provided a comparison between the C45 system

and his system that can be found in (Kohavi 1994) Later Kohavi and Li (1995) used a

deterministic method for attribute selection which minimizes the width of the penultimate level

of the graph

Table 2-9 shows a comparison between the proposed approach and those two approaches The

EDAG and HOODG systems are unreleased prototype systems

or Decision structures are easy to Decision structures are Decision structures are easy to understand difficult to read understand

CHAPTER 3 DESCRIPTION OF THE APPROACH

31 General Methodology

In the proposed approach the function of learning or discovery is separated from the function of

using the discovered knowledge for decision-making The frrst function is performed by an

inductive learning program that searches for knowledge relevant to a given class of decisions

and stores the learned knowledge in the form of decision rules The second function is performed

when there is a need for assigning a decision to new data points in the database (eg bull a

classification decision) by a program that transforms obtained knowledge into a decision

structure optimized according to the given decision-making situation

The Learning Task GivenA set of training examples describing the concept to be learned Learning goal which specifies the decision classes to be learned from the training examples A background knowledge to control the learning process Determine A concept description in a declarative form of knowledge (decision rules) that satisfies the learning goal

The Decision-making Task GivenA set of decision rules in a conjunctive form A description of the new decision-making situation (eg attributes costs and order preference importance or frequency of decision classes etc) One or more examples that needed to be tested under the given decisionshymaking situation A set of parameters to control the learning process Determine A decision structure that suits the given decision-making situation

The decision rules used here are learned by either the AQ1S (Michalski et al 1986) or AQ17

(Bloedorn et aI 1993) learning systems The reasons for selecting decision rules as a form of

23

24

declarative knowledge are that they do not impose any order on the evaluation of the attributes

and due to the lack of order constraints decision rules can be evaluated in many different

ways which increases the flexibility of adapting them to the different tasks of decision-making

(Bergadano et al 1990) Since the number of rules learned per class is typically much smaller

than the number of examples per class generating a decision structure from decision rules can

potentially be done on line

Such virtual decision structures can be tailored to any given decision-making situation The

needed decision rules have to be generated only once and then they can be used many times for

generating decision structures according to changing requirements of decision-making tasks The

method uses the AQDT-2 system (Imam amp Michalski 1994) for learning decision structures

from decision rules Decision structures represent a procedural form of knowledge which makes

them easy to implement but also harder to change Consequently decision structures can be quite

effective and useful as long as they are used in decision-making situations for which they are

optimized and the attributes specified by the decision structure can be measured without much

cost Figure 3-1 shows an architecture of the proposed methodology

Data

Decision -

- Learning knowledge from database The decision making process

Figure 3 -1 Architecture of the AQDT approach

25

It is assumed that the database is not static but is regularly updated A decision-making problem

arises when there is a case or a set of cases to which the system has to assign a decision based on

the knowledge discovered Each decision-making situation is defmed by a set of attribute-values

Some attribute-values may be missing or unknown A new decision structure is obtained such

that it suits the given decision-making problem The learned decision structure associates the

new set of cases with the proper decisions

32 A Brief Description of the AQIS and AQ17 Rule Learning Programs

The decision rules are generated from examples by an AQ-type inductive learning system

specifically by AQ15 (Michalski et albull 1986) or AQI7-DCI (Bloedorn at al 1993) The rest of

this subsection includes a brief description of the inductive learning systems AQ15 and AQI7

AQ15 learns decision rules for a given set of decision classes from examples of decisions using

the STAR methodology (Michalski 1983) The simplest algorithm based on this methodology

called AQ starts with a seed example of a given decision class and generates a set of the

most general conjunctive descriptions of the seed (alternative decision rules for the seed

example) Such a set is called the star of the seed example The algorithm selects from the star

a description that optimizes a criterion reflecting the needs of the problem domain If the

criterion is not defined the program uses a default criterion that selects the description that

covers the largest number of positive examples (to minimize the total number of rules needed)

and with the second priority that involves the smallest number of attributes (to minimize the

number of attributes needed for arriving at a decision)

If the selected description does not cover all examples of a given decision class a new seed is

selected from uncovered examples and the process continues until a complete class description

is generated The algorithm can work with few examples or with many examples and can

optimize the description according to a variety of easily-modifiable hypothesis quality criteria

26

The learned descriptions are represented in the form of a set of decision rules expressed in an

attributional logic calculus called variable-valued logic 1 or VLl (Michalski 1973) A

distinctive feature of this representation is that it employs in addition to standard logic operators

the internal disjunction operator (a disjunction of values of the same attribute in a condition) and

the range operator (to express conditions involving a range of discrete or continuous values)

These operators help to simplify rules involving multi valued discrete attributes the second

operator is also used for creating logical expressions involving continuous attributes

AQ15 can generate decision rules that represent either characteristic or discriminant concept

descriptions depending on the settings of its parameters (Michalski 1983) A characteristic

description states properties that are true for all objects in the concept The simplest

characteristic concept description is in the form of a single conjunctive rule (in general it can be

a set of such rules) The most desirable is the maximal characteristic description that is a rule

with the longest condition part ie stating as many common properties of objects of the given

class as can be determined A discriminant description states properties that discriminate a given

concept from a fixed set of other concepts The most desirable is the minimal discriminant

descriptions that is a rule with the slwnest condition part For example to distinguish a given

set of tables from a set of chairs one may only need to indicate that tables have large flat top

A characteristic description of the tables would include also properties such as have four legs

have no back have four corners etc Discriminant descriptions are usually much shorter than

characteristic descriptions

Another option provided in AQ15 controls the relationship among the generated descriptions

(rulesets or covers) of different decision classes In the IC (Intersecting Covers) mode

rulesets of different classes may logically intersect over areas of the description space in which

there are no training examples In the DC (Disjoint Covers) mode descriptions of different

classes are logically disjoint The DC mode descriptions are usually more complex both in the

number of rules and the number of conditions There is also a DL mode (a Decision List mode

27

also called VL modeM-for variable-valued logic mode) in which the program generates rule sets

that are linearly ordered To assign a decision to an example using such rulesets the program

evaluates them in order If ruleset is satisfied by the example then the decision is made

otherwise the program proceeds to the evaluation of the ruleset +1 In IC and DC modes

rule sets can be evaluated in any order

Alternatively the system can use rules from the AQ17-DCI program for Data-driven

Constructive Induction AQ17-DCI differs from AQ15 mainly in that it contains a module for

generating additional attributes These attributes are various logical or mathematical

combinations the original attributes The program generates a large number of potential new

attributes and seleets from them those most promising based on an attribute quality criterion

To illustrate the format of rules generated by AQ15 (or AQ17-DCI) an exemplary ruleset is

shown in Figure 3-2 The ruleset (that can be re-represented as a disjunctive normal form

expression) describes a voting record of Democratic Representatives in the US Congress

Each rule is a conjunction of elementary conditions Each condition expresses a simple relational

statement For example the condition [State = northeast v northwest] states that the attribute

State (of the Representative) should take the value northeast or northwest to satisfy the

condition

Rl [Gas_concban = yes] amp [Soc_see_cut = no v not registered] R2 [Draft = Yes v not registered]amp[AlaslaLparks = yes v not registered] amp

[Food_stamp_cap = no] amp [State = northeast v northwest] R3 [Chrysler =yes v not registered] amp [Income =low] R4 [Education =yes] amp [Occupation = yes]

Figure 3-2 A ruleset generated by AQ15 for the concept Voting pattern of Democratic Representatives

The above rules were generated from examples of the voting records For illustration below is

an example of a voting record by a Democratic representative

Draft registration=no Ban of aid to Nicaragua=no Cut expenditure on mx missiles=yes Federal subsidy to nuclear power stations=yes Subsidy to national parks

28

in Alaska=yes Fair housing bill=yes Limit on Pac contributions=yes Limit onood stamp program=no Federal help to education=no StateFrom=north east State Population=large Occupation =unknown Cut in social security spending=no Federal help to Chrysler corp=not registered

By expressing elementary statements in the example as conditions and linking conditions by

conjunction the examples can be re-expressed as decision rules Thus decision rules and

examples formally differ only in the degree of generality

33 Generating Decision Structurestrees from Decision Rules

This section describes the AQDT-1 system for learning decision structures (trees) from decision

rules (Imam amp Michalski 1993ab) Also a description of the AQDT-2 method for learning taskshy

oriented decision structure from decision rules is included and fmally the methodology is

illustrated by two examples

Methods for learning decision trees from examples have been very popular in machine learning

due to their simplicity Decision trees built this way can be quite efficient as long as they are

used in decision-making situations for which they are optimized and these situations remain

relatively stable Problems arise when these situations significantly change and the assumptions

under which the tree was built do not hold anymore For example in some situations it may be

difficult to determine the value of the attribute assigned to some node One would like to avoid

measuring this attribute and still be able to classify the example if this is potentially possible

(Quinlan 1990) If the cost of measuring various attributes changes it is desirable to restructure

the tree so that the inexpensive attributes are evaluated f11st A tree restructuring is also

desirable if there is a significant change in the frequency of occurrence of examples from

different classes A restructuring of a decision tree to suit the above requirements is however

difficult to do The reason for this is that decision trees are a form of decision structure

representation and imposes constraints on the evaluation order of the attributes that are not

logically necessary

29

One problem in developing a method for generating decision structures from decision rules is to

design an attribute selection criterion that is based on the properties of the rules rather than of

the training examples A decision rule normally describes a number of possible examples Only

some of them are examples that have actually been observed ie training examples An attribute

selection criterion is needed to analyze the role of each attribute in the rules It cannot be based

on counting the numbers of training examples covered by each attribute-value and the

frequency of decision classes in the training examples as is done in learning decision trees from

examples because the training examples are assumed to be unavailable

Another problem in learning decision trees from decision rules stems from the fact that decision

rules constitute a more powerful knowledge representation than decision trees They can directly

represent a description in an arbitrary disjunctive normal form while decision trees can represent

directly only descriptions in the disjoint disjunctive normal form In such descriptions all

conjunctions are mutually logically disjoint Therefore when transforming a set of arbitrary

decision rules into a decision tree one faces an additional problem of handling logically

intersecting rules

The solution to both problems (attribute selection and logically intersected rules) in the AQDT-2

system is based on the earlier work by Michalski (1978) which introduced a general method for

generating decision trees from decision rules The method aimed at producing decision trees

with the minimum number of nodes or the minimum cost (where the cost was defined as the

total cost of classifying unknown examples given the cost of measuring individual attributes and

the expected probability distribution of examples of different decision classes) More

explanations are provided in the following section

331 Tbe AQDT-2attribute selection metbod

This section describes the AQDT -2 method for building a decision structure from decision rules

The method for building a single-parent decision structure is similar to that used in standard

30

methods of building a decision tree from examples The major difference is that it assigns tests

(attributes) to the nodes using criteria based on the properties of the decision rules (this includes

statistics about the examples covered by each rule in the case of learning rules from examples)

rather than statistics characterizing the frequency of training examples per decision classes per

attribute-values or per conjunctions of both Other differences are that the branches may be

assigned an internal disjunction of values (not only a single value as in a typical decision tree)

and leaves may be assigned a set of alternative decisions with probabilities Also the tests can be

attributes or names standing for logical or mathematical expressions that involve several

attributes or variables In the following we use the terms test and attribute interchangeably

(to distinguish between an attribute and a name standing for an expression the latter is called a

constructed attribute)

At each step the method chooses the test from an available set of tests that has the highest utility

(see below) for the given set of decision rules This test is assigned to the node The branches

stemming from this node are assigned test values or disjoint groups of values (in the form of

logical disjunction if such occur in the rules subsumed groups of values are removed) Each

branch is associated with a reduced set of rules determined by removing conditions in which the

selected attribute assumes value(s) assigned to this branch If all rules in the reduced ruleset

indicate the same decision class a leaf node is created and assigned this decision class The

process continues until all nodes are leaf nodes If it is not possible to reduce further the rule set

because some attribute is declared as unavailable (infmite cost) then the leaf is assigned a set of

candidate decisions with associated probabilities (see Sec 42)

The test (attribute) utility is a combination of one or more of the following elementary criteria 1)

cost which indicates the cost of using each attribute for making decision 2) disjointness which

captures the effectiveness of the test in discriminating among decision rules for different decision

classes 3) importance which determines the importance of a test in the rules 4) value

31

distribution which characterizes the distribution of the test importance over its of values and 5)

dominance which measures the test presence in the rules These criteria are dermed below

~ The cost of a test expresses the effort or cost needed to measure or apply the test

Djsjointness The disjointness of a test is defined as the sum of the class disjointness-the

disjointness of the test for each decision class Suppose decision classes are Cl C2 bull Cm and

decision rule sets for these classes have been determined Given test A let V 1 V 2 bull V m denote

sets of the values (outcomes) of A that are present in rule sets for classes C 1 C2bullbull Cm

respectively If a ruleset for some class say Ct contains a rule that does not involve test A than

Vt is the set of all possible values of A (the domain of A)

Definition 3 -1 The degree ofclass disjointness D(A Ci ) of test A for the ruleset ofclass Ci is

the sum of the degrees of disjointness D(A Cit Cj) between the ruleset for Ci and rulesets for

Cj j=l 2 m j J L The degree of disjointness between the ruleset for Ci and the ruleset for Cj is

dermed by

if Vi Vj ifVj gtVj (3-1) ifVinVj -tljgt amp -tVi amp -tVj ifVinVj=1jgt

where cjgt denotes the empty set Note that exchanging the second and third conditions of equation

(3-1) may seem to be an improved criteria However it does not clearly distinguish between both

cases (Le for both situations the disjointness will be similar) The current equation is better

because it gives higher scores to attributes that classify different subsets of the two decision

classes than attributes that classify only a subset of one decision class

Definition 3-2 The disjointness of the test A for evaluating a given set of decision rules is the

sum of the degrees of class disjointness of each decision class

32

m m Disjointness(A) = L D(A cn where D(A cn = L D(A Ci Cj) (3-2)

i=l i=l i~j

The disjointness of a test ranges from 0 when the test values in rulesets of different classes are

all the same to 3m(m-l) when every rule set of a given class contains a different set of the test

values If two tests have the same disjointness value the attribute to be selected is the one with

less number of values

Definition The A vera~ Number of Tests (ANT) required to make decision from a decision tree

is dermed as the average number of tests (attributes) to be examined from the root of the tree to

any leaf node in order to reach a decision

Definition A decision structure is a one-node-per-level decision structure if at each level there is

only one node and zero or more leaves

Such decision structure can be generated by combining together all branches that associated with

each one a set of decision rules belongs to more than one decision class

Theorem 1 Consider learning one-node-level decision structure for a database with two

decision classes The disjointness criterion ranks first attributes that add minimum number of

tests to the decision tree

Proof Consider all possible distributions of an attributes values within two decision classes Ci

and Cj There are three cases 1) subset (same as super set) 2) non-empty intersection but not

subset 3) no intersection Figure 3-3 shows all possible distributions (the case of having the

same set of values in both classes is trivial one) Consider that branches leading to one subset

with the same decision class are combined into one branch In the first case there will be two

branches only The first leads to a leaf node and the other leads to an intermediate node where

another attribute to be selected The minimum ANT in this case is 53 In the second case three

branches should be created Two branches leads to a leaf node where all values at each branch

belong to only one and different decision class The third branch leads to an intermediate node

33

where another attribute should be selected furtur classifies the decision classes The minimum

ANT in this case is 64 In the third case only two branches will be generated where each leads

to leaf node with different decision class In this case the minimum ANT is 1

Figure 3-4 shows decision trees equivalent to selecting the corresponding attribute Note that in

case of having more than one attribute-value at branches lead to leaves belonging to one decision

class they will be combined into one branch in the decision structure The symbol 1 means

that an attribute is needed to classify the two decision classes In such cases there will be at least

two additional paths

D(A Ci) = 0 D(A CJ) = 1 D(A Ci) =2 D(A Cj) = 2 D(A Ci) =3 D(A Cj) =3

Figure 3-3 Yen Diagrams of Possible combinations of attribute values in two decision classes

The average number of tests required for making a decision in each possible case is determined

in Figure 3-4 It is clear that in the case of two decision classes the disjointness criterion ranks

highly attributes that reduces the average number of tests required for decision-making The

theorem can be proved in the general case

i ~ i Cj C1middot bull Ci Cj

ANT=32middotANT=53 ANT=l

1 means at least one attribute is needed to complete the decision tree Figure 3-4 Decision trees corresponding the Yen diagrams inFigure 3-3

34

Theorem 2 The attribute with the highest non~zero disjointness is the best attribute to classify

the decision classes

Proof Suppose that the number of decision classes is n Assume also that there are two attributes

A and B where D(A) lt D(B) Since the disjointness criterion considers the mutual relationship

between any two decision classes this means that there are more decision classes where D(A

Ci) lt D(B Ci) than those where D(A Ci) gt D(B Ci) (ignoring classes with equal disjointness

for bCth attributes) Hence there are more decision classes where D(A Ci Cj) lt D(B Ci Cj)

than those where D(A Ci Cj) gt D(B Cit Cj) Let us frrst prove that if D(A Ci Cj) lt D(B Cit

Cj) then B classifies the decision classes better than A

For each pair of decision classes Ci and Cj the possible values of the disjointness of any attribute

are 0 14 or 6 For all positive values ofD(B)= 14 or 6 it is clear that attribute B should have

smaller ANT than attribute A which has lower disjointness This means that if D(A Cit Cj) lt

D(B Cit Cj) then attribute B can classify both decision classes better than attribute A

Similarly B is better for classifying more pairs of decision classes than A This implies that B is

a better classifier than A

Importance The second elementary criterion the importance of a test is based on the

importance score (IS) introduced in (Imam Michalski amp Kerschberg 1993) In the obtained

rules each test is assigned a score that represents the total number of training examples that

are covered by the rules involving this test Decision rules learned by an AQ learning program

are accompanied with information on their strength Rule strength is characterized by t-weight

and u~weight The t-weight (total-weight) of a rule for some class is the number of examples of

that class covered by the rule The importance score of a test is the aggregation of the totalshy

weights of all rules that contain that test in their condition part Given a set of decision rules for

m decision classes Cl Cm and involving n tests AlbullAn and the number of rules associated

with class Ci is denoted by tlrit The importance score is defmed as follow

35

Definition 3 -3 The importance score IS(Aj) of the test A j is detennined by

m IS(Aj)= L IS(Aj Ci) (3-31)

i=l

where ri IS(Aj Ci) = L Rik(Aj) (3-32)

k=l

and Rik the weight of a test Aj in the rule Rk of class Ci is given by

t - weight if A j belongs to rule Rik Rik (Amiddot) = (3-4)J 0 otherwise

where i=I n ik=I ri j=I m

The importance score method has been separately compared as a feature selection method with

a genetic algorithm-based method (Imam amp Vafaie 1994) The importance score method

produced an equal or higher accuracy on three real-world problems than those reported by the

GA method while selecting fewer attributes In addition the IS method was significantly faster

than the GA

yalue distribution The third elementary criterion value distribution concerns the number of

legal values of tests Given two tests with equal importance scores this criterion prefers the test

with the smaller number of legal values Experiments have shown that this criterion is especially

useful when using discriminant decision rules

Definition 34 A value distribution VD(Aj) of a test A j is defined by

VD(Aj) = IS(Aj) I Vj (3-5)

where v is the number of legal values of Aj

Dominance The fourth elementary criterion dominance prefers tests that appear in large

numbers of rules as this indicates their high relevance for discriminating among the rulesets of

given decision classes Since some conditions in the rules have values linked by internal

disjunction counting such rules directly would not reflect properly their relevance Therefore

36

for computing the dominance the rules are counted as if they were converted to rules that do not

have internal disjunction Such a conversion is done by multiplying out the condition parts of

the rules containing internal disjunction For example the condition part [x3=1 v 3]amp[x4=1] is

multiplied out to two rules with condition parts [x3=1]amp[x4=1] and [x3=3]amp[x4=I]

The above criteria are combined into one general test measure using the lexicographic

evaluation functional with tolerances (LEF) (Michalski 1973) LEF is a list of some or all of

the above elementary criteria each associated with a Ittolerance threshold It in percentage The

criteria are applied to tests in the order defined by LEF A test passes to the next criterion only if

it scores on the previous criterion within the range defined by the tolerance (from the top value)

The default LEF is

ltCost d Disjointness t2 Importance t3 Value distr t4 Dominance 0gt (3-6)

where d a t3 t4 and t5 are tolerance thresholds (in percentages) their default values are O

The default value of the cost of each test is 1

The above LEF ranks attributes this way First attributes are evaluated on the basis of their cost

If two or more attributes have the same cost they ranked by their degree of disjointness If two

or more attributes still share the same top score or their scores differ less than the assumed

tolerance threshold t2 the method evaluates these attributes using the second (Importance)

criterion If again two or more attributes share the same top score or their scores differ less than

the tolerance threshold t3 then the third criterion normalized IS is used then similarly the

fourth criterion (dominance) If there is still a tie the method selects the best attribute

randomly

If there is a non-uniform frequency distribution of examples of different classes then the

selection criterion uses a modified definition of the disjointness Namely the previously defmed

37

disjointness for each class is multiplied by the frequency of the class occurrence The class

occurrence is the expected number of future examples that are to be classified to a given class

m

Disjointness(A) = L D(A CO Frq(Ci) (3-7) i=l

where Frq(Cn is the expected frequency of examples of class Cit and it is assumed to be given

by the user The attribute ranking criteria in this case is defined by the LEF

ltCost d Disjointness t2 Importance 13 Normalized-IS 14 Dominance 15gt (3-8)

where the Cost denotes the evaluation cost of an attribute and is to be minimized while other

elementary criteria are treated the same way as in the default version

332 The AQDT-2 algorithm

The AQDT-2 algorithm constructs a decision structure from decision rules by recursively

selecting at each step the best test according to the ranking criteria described above and

assigning it to the new node The process stops when the algorithm creates terminal branches that

are assigned decision classes To facilitate such a process the system creates a special data

structure for each concept description (ruleset) This structure has fields such as the number of

rules the number of decision classes and the number of attributes present in the rules A set of

pointers connect this data structure to set of data structures each represents one decision class

The decision class structure contains fields with information on the number of rules belong to

that class the frequency of the decision class etc It is also connected to a set of data structures

representing the decision rules within each decision class The system creates independently a set

of data structures each is corresponding to one attribute Each attribute description contains the

attributes name domain type the number of legal values a list of the values the number of

rules that contain that attribute and values of that attribute for each rule The attributes are

arranged in an array in a lexicographic order flISt in the descending order of the number of rules

38

that contain that attribute and second in the ascending order of the number of the attributes

legal values

The system can work in two modes In the standard mode the system generates standard

decision trees in which each branch has a specific attribute-value assigned In the compact

mode the system builds a decision structure that may contain

A) or branches Le branches assigned an internal disjunction of attribute-values whenever it

leads to simpler structures For example if a node assigned attribute A has a branch marked by

values 1 v 2 then the control passes along this branch whenever A takes value 1 or 2 The

program creates or branches on the basis of the analysis of the value sets Vi while computing

the degree of attribute disjointness

B) nodes that are assigned derived attributes that is attributes that are certain logical or

mathematical combinations of the original attributes To produce decision structures with

derived attributes the input decision rules are generated by AQ17 (rather than AQI5) The

AQ17 rules may contain conditions involving attributes constructed by the program rather than

those originally given

To generate decision structures from rules the AQDT-2 method prefers either characteristic or

discriminant disjoint rule descriptions (given by an expert or learned by a system) Disjoint rules

are more suitable for building decision structures Assume that the description of each class is in

the form of a ruleset and that this set is the initial ruleset context The AQDT algorithm is

The AQDT2 Algorithm

Step 1 Evaluate each attribute occurring in the ruleset context using the LEF attribute ranking

measure Select the highest ranked attribute Let A represents this highest-ranked

attribute

Step 2 Create a node of the tree (initially the root afterwards a node attached to a branch) and

assign to it the attribute A In standard mode create as many branches from the node as

39

there are legal values of the attribute A and assign these values to the branches In

compact mode (decision structures) create as many branches as there are disjoint value

sets of this attribute in the decision rules and assign these sets to the branches

Step 3 For each branch associate with it a group of rules from the ruleset context that contain a

condition satisfied by the value(s) assigned to this branch For example if a branch is

assigned values i of attribute A then associate with it all rules containing condition [A=

i v ] If a branch is assigned values i v j then associate with it all rules containing

condition [A= i v j v ] Remove from the rules these conditions If there are rules in

the ruleset context that do not contain attribute A add these rules to all rule groups

associated with the branches stemming from the node assigned attribute A (This step is

justified by the consensus law [x=1] == [x=1] amp [y=a] v [x=1] amp [y=b] assuming

that a and b are the only legal values of y) All rules associated with the given branch

constitute a ruleset context for this branch

Step 4 If all the rules in a ruleset context for some branch belong to the same class create a leaf

node and assign to it that class If all branches of the trees have leaf nodes stop

Otherwise repeat steps 1 to 4 for each branch that has no leaf

To select an attribute to be a node in the decision tree (step 1 and 2 of the algorithm) the

algorithm performs two independent iterations In the first iteration it parses through all decision

rules and determine information about each attributes This information includes the importance

score of each attribute the number of rules containing a given attribute the disjoint value sets of

each attribute and the attribute values used in describing each decision class The second

iteration is only performed if the disjointness criterion is ranked first in the LEF function The

second iteration evaluates the attributes disjointness for each decision class against the other

decision classes

40

To determine the complexity of this process suppose that the maximum number of conditions

fonned by a single attribute value in one rule is s (where s ~ n the number of attributes) and r is

the total number of decision rules (in all decision classes)

m

r=LRt (m is the number of decision classes) 1=1

where Ri is the number of rules in decision class Ci bull The complexity of the frrst iteration can be

determined as

Cmpx(Itcl) =O(r s)

In the second iteration the disjointness is calculated between the decision classes and all

attributes The complexity of the second iteration can be given by

Cmpx(Itc2) = O(n m)

Assume that at each node 1 is the maximum of the number of decision classes to be classified at

this node and the number of rules associated with this node

l=max mr (3-9)

Since these two iteration are independent of each other the complexity of the AQDT algorithm

for building one node in the decision tree say Node Complexity NC(AQDT) is given by

NC(AQDT) = 0(1 n)

Usually I equals to the number of rules associated with the given node Thus the AQDT

complexity for building one node is a function of the number of attributes multiplied by the

number of rules associated with this node

At each level of the decision tree these two iterations are repeated for each non-leaf node The

Level Complexity of the AQDT algorithm LC(AQDT) can be given by

LC(AQDT) lt 0(1 n)

Which is less than the complexity of generating the root of the decision tree To explain this

consider the maximum possible number of non-leaf nodes at one level is half the number of the

initial decision rules (integer of rll) Figure 3-5-a shows an example of such situation where at

41

the lowest level each node classifies only two rules each belongs to different decision class This

decision tree could be a multimiddotvalues tree However the number of non-leaf nodes at each level

will be twice or more than the number of nonmiddot leaf nodes at the previous level Considering the

level complexity of the AQDT algorithm equal to (1 s 0) where 0 is the number of non-leaf

nodes at the given level In such cases it is either (1 0 S r) or (1 s lt r) In Figure 3middot5middota the

complexity of the AQDT algorithm at any lower level is given by

LC(AQDT) =0(2 s r(2) lt O(n I) =NC(AQDT)

a) per one level b) per one path

Figure 3 -5 Decision trees showing the maximum number of none leaf nodes

Note also that after selecting an attribute to be the root of the decision structure this attribute

and all conditions containing that attribute are removed from the data structure of the algorithm

Also if a leaf node is generated all rules belonging to the corresponding branch will not be

tested again

Since the disjointness criterion selects the attribute which minimizes the average number of tests

ANT it is true that the AQDT algorithm generates decision trees with the least number of levels

The number of levels per a decision tree is supposed to be less than or equal to the minimum of

both the number of attributes and the number of rules Consider k as the number of levels in a

given decision tree

k S min mr (3-10)

There are two cases represent the most complex situations Figure 3-5-a and 3-5-b In the fIrst

case where the decision rules were divided evenly the number of levels will be a function of the

42

logarithm of the number of rules In such case the complexity of the AQDT algorithm for

generating a decision tree from a set of decision rules is given by

Complexity(AQDT) = 0(1 n log r) (3-11)

The other situation is when the generated decision tree has the maximum number of levels The

maximum possible number of levels per a decision tree equals to one less than the number of

decision rules Figure 3-5-b Using the disjointness criterion it is not likely to get such a decision

tree because it has the maximum average number of test (ANT) that can be determined from the

same set of nodes and leaves However such a decision tree can be generated if the number of

decision classes is one less thanthe number of attributes In such case any disjQint decision rules

should have a maximum length that is less or equal to the floor of the logarithm of the number of

attributes Thus the level complexity of this decision tree is estimated as

LC(AQDT) =0(1 log n)

The maximum number of levels in such decision tree is k-l Thus the complexity of the AQDT

algorithm in such cases is given by

Complexity(AQDT) = 0(1 k log n) (3-12)

Since k is the minimum of n and r then n is the maximum of k and n Also since in (3-12)

r=n+1 then log r is greater than log n From 3-10 3-11 and 3-12 the complexity of the AQDT

algorithm is determined by

Cmplx(AQDT) = OCr k log 1) (3-13)

333 An example illustrating the algorithm

The following simple example illustrates how AQDT is used in selecting the optimal set of

testing resources for testing a software Suppose there are three tools for testing a software 1)

modeling (Tl) 2) checklist (T2) and 3) par_simul (T3) Also assume that there are four

different factors that affect the selection of any tool 1) the cost of using the tool (xl) 2) the

metrics that support the tool (x2) 3) the best phase for applying the tool (x3) and 4) type of tool

43

(automated semi-automated or manual) (x4) Table 3-1 shows these attributes and their possible

values

Suppose that the domain expert provided a set of rules to be used in testing a software Figure 3shy

6 shows a sample of these rules in AQ15c formats

Table 3-1 The available tools and the factors that affect the prclCes

Tl lt= [xl=2] amp [x2=2] v [xl=3] amp [x3=1 v 3] amp [x4=1] T2 lt= [xl=l v 2] amp [x2=3 v 4] v [xl=3] amp [x3=1 v 2] amp [x4=2] T3 lt= [xl=l] amp [x2=1] v [xl=4] amp [x3=2 v 3] amp [x4=3]

Figure 3-6 Decision rules for selecting the best tool for testing software

These rules can be interpreted as

Rule 1 Use the fltst tool for testing if you need average cost and the tool is

supported by the requirement metric

Rule 2 Use the fIrst tool for testing if you can afford high cost for testing either

in the requirement or the analysis phases and you need an automated tool

Rule 3 Use the second tool for testing if the cost limit is low or average and the

tool is supported by the system usage metric or the intractability metric

Rule 4 Use the second tool for testing if you can afford high cost for testing

either in the requirement or the design phases and you need a manual tool

Rule 5 Use the third tool for testing if the you are limited to low cost and the

tool is supported by the error rate metric

Rule 6 Use the third tool for testing if you can afford very-high cost for testing

either in the requirement or the system usage phases and you need a semishy

automated tool

44

Table 3-2 presents information on these rules and the disjointness values for all attributes For

each class the row marked Values lists values occurring in the ruleset for this class For

evaluating the disjointness of an attribute say A each rule in the ruleset above that does not

contain attribute A is characterized as having an additional condition [A= a vb ] where a b

are all legal values of A

The row Class disjointness specifies the class disjointness for each attribute The attribute xl

has the highest disjointness (11) and is assigned to the root of the tree For simplicity assume

the tolerances for each elementary criterion equal O

From the rules in Figure 3-6 we can also determine disjoint groupings of attribute-values used

for the compact mode of the algorithm This done as follows 1) determine for each attribute the

sets of values that the attribute takes in individual decision rules and remove those value sets

that subsume other value sets The remaining value sets are assigned to branches stemming from

the node marked by the given attribute For example x 1 has the following value sets in the

individual decision rules 2 3 I 2 I and 4 (Figure 3-6) Value set I 2 is

removed as it subsumes 2 and I In this case branches are assigned individual values of the

domain of x1 For attribute x2 the value sets are I 2 34 and I 2 3 4 In this case

branches are assigned value sets I 2 and 34

45

Attribute Xl ranks highest (as it is the highest disjointness) and is assigned to the root of the

tree Four branches are created each one corresponding to one of XIs possible values Since all

rules containing [x1=4] belong to class TI the branch marked by 4 is ended by a leafTI Rules

containing other values of X1 belong to more than one class This process is repeated for each

subset of rules until the decision tree is completed Figure 3-7 shows a decision structure learned

by AQDT-2 from the rules in Figure 3-6 The decision structure in Figure 3-7 can be used in

making decisions on which tools can be used for testing a given software

xl

Complexity No of nodes 4 No of leaves 7

Figure 3-7 A decision structure learned for classifying so tware testing tools

Figure 3-8-a shows the diagrammatic visualization of the decision rules and Figure 3-8-b shows

the visualization of the derived decision tree Each diagram in Figure 3-8 consists of cells

representing one combination of attribute-values Attributes and their legal values are shown on

scales surrounding the diagram (eg the horizontal scale for xl shows values 1 2 3 and 4)

Rules are represented by collections of cells in the intersection of the rows and columns

corresponding to the conditions in the rules

The shaded areas correspond to decision rules Rules of the same class have the same type of

shading Empty cells correspond to combination of attribute-values not assigned to any class For

illustration collections of cells corresponding to some of the initial rules are marked R 11 R 21

R31 and R32 Rll denotes the IlfSt rule of Class T1 ie [x1=2] amp [x2=2] R21 denotes the

IlfSt rule of class T2 ie [xl= 1 v 2] amp [x2= 3 v 4] R31 the first rule of class TI ie [x1=1]

46

amp [x2 =1] and denotes R32 denotes the second rule of class T3 Le [xl=4] amp [x3=2 v 3] amp

[x4=3]

a) Decision rules b) Derived decision tree

Comparing diagrams in Figure 3-8-a and 3-8-b AQDT-2 decision tree represents a slightly more

general description of Concepts Tl TI and T3 than the original rules Let us assume that it is

very costly to know which metrics suppon the required tools In other words suppose that we

would like to select the best tools independently of the metric they suppon (this is indicated to

AQDT-l by assigning very high cost to attribute x2) The algorithm again assigns attribute xl to

the root of the tree When xl takes value 1 or 2 it is impossible to assign a specific decision

without measuring attribute x2 For the value 1 of xl the recommended tool can be either TI or

T3 (see the diagram in Figure 3-9-a) and for the value 2 of xl the recommended tool can be

either Tl or T3 However for the value 3 of xl one can make a specific decision after

measuring attribute x4 For the value 1 of x4 tool is Tl and for the value 2 of x4 the

recommended tool is TI Figure 3-9-b shows another decision tree where the type of the tool

was ignored from the data Those decision trees are called indeterminate because some of their

leaves are assigned a disjunction of two or more class names

47

xl

xl

1 12

a) Ignoring the supporting metric b) Ignoring the type of the tooL Figure 3 -9 Decision trees learned ignoring the support metric and the type of the testing tool

It is clear that the given set of rules depend highly on amount of money can be spent Let us now

suppose that the cost attribute xl which was determined as the highest ranked attribute cannot

be measured The algorithm selected x4 to be the root of the new decision tree After continuing

the algorithm the tree in Figure 3-10 is obtainedThe nodes that are assigned one class indicate

situations in which it is possible to make a specific decision without knowing the value of

attribute xl

x4

Figure 3-10 A decision tree learned without the cost attribute

34 Tailoring Decision Structures to the Decision-making situation

Decision structures are among the simplest structures for organizing a decision-making process

A decision structure specifies explicitly the order in which attributes of an object or situation

need to be evaluated in the process of determining a decision A standard way to generate a

48

decision structure is to learn it from examples of decisions Such a process usually aims at

obtaining a decision structure that has the highest prediction accuracy that is the highest rate of

assigning correct decisions to given situations There can usually be a large number of logically

equivalent decision structures (Michalski 1990) As such they may have the same predictive

accuracy but differ in the way they organize the decision process and thus may differ in the cost

of arriving at a decision To minimize the average decision cost one needs to take into

consideration the distribution of the costs of attribute evaluation and the frequency of different

decisions This report presents an approach to building such Task-oriented decision structures

which advocates that they are built not from examples but rather from decision rules Decision

rules are learned from examples using the AQ15 or AQ17 inductive learning program or are

specified by an expert An efficient algorithm and a new system AQDT-2 that transforms

decision rules to task-oriented decision structures The system is illustrated by applying it to the

problem of learning decision structures in the area of construction engineering (for determining

the best wind bracing for tall buildings) In the experiments AQDT-2 outperformed all other

programs applied to the same data

Decision-making situations can vary in several respects In some situations complete

information about a data item is available (ie values of all attributes are specified) in others the

information may be incomplete To reflect such differences the user specifies a set of parameters

that control the process of creating a decision structure from decision rules AQDT-2 provides

several features for handling different decision-making problems 1) generating a decision

structure that tries to avoid unavailable or costly attributes (tests) 2) generating unknown

leaves in situations where there is insufficient information for generating a complete decision

structure 3) providing the user with the most likely decision when performing a required test is

impossible 4) providing alternative decisions with an estimate of the likelihood of their

correctness when the needed information can not be provided

49

341 Learning Costmiddot Dependent Decision Structures

As described in Sec 2 the LEF criterion enables the system to take into consideration the cost of

measuring tests (attributes) in developing a decision structure In the default setting of LEF the

tests cost is the [lIst criterion and its tolerance is O This means that only the least expensive

attributes pass to the next step of evaluation involving other elementary criteria If an attribute

has high cost or is impossible to measure (has an infinite cost) the LEF chooses another

cheaper attribute ifpossible

342 Assigning Decision Under Insufficient Information

In decision-making situations in which one or more attributes cannot be measured the system

may not be able to assign a definite decision for some cases If no more information can be

obtained but a decision has to be made it is useful to know the probability distribution for

different candidate decisions (Smyth Goodman amp Higgins 1990) The most probable decision is

then chosen The probability distribution can be estimated from the class frequency at the given

node Let us consider a node in the structure connected to the root by the sequence bl b2 bkof

attribute-values Let P(Ci I bl b2 bk) denote the conditional probability of class Ci i=I2 m

at that node given that an example to be classified has attribute-values assigned to branches bl

b2 bull bk in the decision structure Using Bayesian formula we have

P(Ci I bl~ bull bk) = P(Ci) P(bl bk I Ci) P(bl bk) (3-9)

where P(Ci) the apriori probability of class Ci P(bl bk I Ci) is the probability that the

example has attribute-values blbull bk given that it represents decision class Ci To approximate

these probabilities let us suppose that Wi is the number of training examples of Class Ci that

passed the tests leading to this node and twi - the total number of training examples of Class

Ci Let us use these numbers to estimate the probabilities in (9) Assuming that the apriori class

probabilities correspond to the frequency of training examples from different classes we have

50

m P(Ci) = twi IL twj (3-10)

j=l

P(b1 bkl Ci) = Wi I twi (3-11)

m m P(b1 bk) =L Wj IL twj (3-12)

j=l j=1

By substituting (1O) (11) and (12) in (9) we obtain

m P(Ci I b1 bk) = wilL Wj (3-13)

j=l

A related method for handling the problem of unavailability of an attribute is described by

Quinlan (1990) Quinlans method however assigns probabilities to the most probable decision

at the node associated with such an attribute The method does not restructure the decision tree

appropriately to fit the given decision-making situation (in this case to avoid measuring xl)

AQDT -2 assigns probabilities o~ly after it first created a decision structure that suits the given

decision-making si~ation An example is presented in section 42

343 Coping with noise in training data

The proposed methodology can be easily extended to handle the problem of learning from noisy

training data This is done based on ideas of rule truncation described in (Niblett amp Bratko 1986

Bergadanoet al 1992 Cestnik amp Karalic 1991) When noise is expected in the training data

the decision rules with t-weight below a certain threshold (reflecting the expected noise level) are

removed The rule truncation method seems to result in more accurate decision structures than

decision tree pruning method because truncation decisions are solely based on the importance of

the given rule or condition for the decision-making regardless of their evaluation order (unlike

the decision tree pruning which can only prunes attributes within a subtree and thus cannot

freely chose attributes to prune) Examples are presented in section 4

vi

333 An example illustrating the algorithm 42

34 Tailoring Decision Structure to a decision-making situation 47

341 Learning Cost-Dependent Decision Structures 49

342 Assigning Decision Under Insufficient Infonnation 49

343 Coping with noise in training data 50

35 Analysis of the AQDT-2 Attribute Selection Criteria 51

36 Decision Structures vs Decision Trees 53

CHAPTER 4

EMPRICAL ANALYSIS AND COMPARATIVE STUDY OF TIlE MElHOD 58

41 Description of the Experimental Analysis 59

42 Experiments With Average Size Complex and Noise-Free Problems

Wind Brac-ings 60

43 Experiments With Small Size Simple and Noise-Free Problems

e MONK-I 69

44 Experiments With Small Size Complex and Noise-Free Problems

MONK-2 76

45 Experiments With Small Size Simple and Noisy Problems

MONK-3 79

46 Experiments With Large Size Complex and Noise-Free Problems

Diagnosing Breast Cancer 83

47 Experiments With Large Size Complex and Noisy Problems

Musm()()m classifications 84

48 Experiments With Small Size Structured and Noise-Free Problems

East-West Trains 85

49 Experiments With Small Size Simple and Noisy Problems

vii

Congressional Voting Records (1984) 87

410 Analysis of the Results 88

ClIAPfER 5 CONCIUSIONS 95

51 Summary 95

52 Contributions 96

REFERENCES 98

VITA 102

viii

LIST OF TABLES

No TITLE Page

2-1 An example of a decision table 9

2-2 A set of training examples used to illustrate the C45 system 15

2-3 The frequency of different attributes values to different decision classes 17

2-4 The expected values of the frequency of examples in Table 2-3 17

2-5 Attribute selection criteria and their basic evaluation measure 17

2-6 The contingency tables of Mingers example 18

2-7 Mingers results for determining the goodness of split 19

2-8 Mingers results for comaring the total accuracy and size of decision trees

provided by different attribute selection criteria from four problems 19

2-9 Comparing the AQDT approach with EDAG and HOODG approaches 22

3-1 The available tools and the factors that affect the process of testing a software 43

3-2 Calculating the disjointness of each attribute 44

3-3 Evaluation of the AQDT-2 criteria on the Quinlans example in Table 2-2 51

3-4 The data used in Mingers f11st experiments 52

3-5 The performance of AQDT-2 criteria (compare with the other criteria in Table 2-6) 52

3-6 The possible ranking domains and using condition of AQDT-2 criteria 53

3-7 Comparison between Decision Structures and Decision Tree 54

4-1 Evaluation of AQDT-2 attribute selection criteria for the wind bracing problem 62

4-2 The predictive accuracy of AQ15c and AQDT-2 for the wind bracing problem 67

4-3 Evaluation of AQDT-2 attribute selection criteria for the MONK-l problem 71

4-4 The predictive accuracy of AQ15c and AQDT-2 for the MONK-l problem 73

ix

4-5 The predictive accuracy of AQ15c and AQDT-2 for the MONK-2 problem 77

4-6 The predictive accuracy of AQ15c and AQDT-2 for the MONK-3 problem 81

4-7 The set of attributes and their values used in the trains problem 86

4-8 A tabular snmmary of the predictive accuracy of decision trees obtained

by AQDT -2 and C45 for the congressional voting data 88

4-9 Summary of the best parameter settings for the first subfunction of the approach

with different data characteristics 89

4-10 Summary of the performance of AQDT-2 and C45 on different problems 90

x

LIST OF FIGURES

No TITLE Page

2-1 An example to illustrate how attributes break rules 8

2-2 A decision tree learned from the decision table in Table 2-1 10

2-3 A decision tree learned using the gain criterion for selecting attributes 15

2-4 Decision rules and their Exception Directed Acyclic Graph (EDAG) 21

3-1 Architecture of the AQDT approach 24

3-2 A ruleset generated by AQ15 for the concept Voting pattern of

Democratic Representatives 27

3-3 Ven Diagrams ofPossible combinations of attribute values in two decision classes 33

3-4 Decision trees corresponding the Ven diagrams in Figure 3-3 33

3-5 Decision trees showing the maximum number of none leaf nodes 41

3-6 Decision rules for selecting the best tool for testing a software 43

3-7 A decision structure learned for classifying software testing tools 45

3-8 A diagrammatic visualization of the decision rules and the derived decision tree 46

3-9 Decision trees learned ignoring the support metric and the type of the testing tool 47

3-10 A decision tree learned without the cost attribute 47

3-11 Decision structures learned by AQDT-2 using different criteria 55

3-12 The Imams example Example where learning decision structures (trees)

from rules is better than learning them from examples 56

3-13 An example where decision rules are simpler than decision trees 57

4-1 Design of a complete experiment 59

Xl

4-2 Decision rules determined by AQ15c from the wind bracing data 61

4-3 A decision tree learned by C45 for the wind bracing data 63

4-4 A decision structure learned from AQ15c wind bracing rules 64

4-5 A decision structure that does not contain attribute xl 64

4-6 A decision structure without Xl with candidate decisions assigned to leaves 65

4-7 A decision structure determined from rules in Figure 4-4 under the

assumption of 10 classification error in the training data 65

4-8 Diagramatic visualization of decision trees learned for different decision

making situations for the wind bracing data 66

4-9 The accuracy of AQDT-2 and AQ15c with different parameter settings

for the wind bracing PIoblem 68

4-10 Analyzing different parameter setting of AQDT-2 using the wind bracing data 69

4-16 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-20 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-11 A comparison between AQ15c and AQDT-2 against C45 on the wind bracing data 69

4-12 A visualization diagram of the MONK-l problem 70

4-13 Decision rules learned by AQ15c for the MONK-1 problem 71

4-14 The decision tree for the MONK-1 problem generated both by AQDT-l 72

4-15 Compact decision structures generated by AQDT-2 for the MONK-1 problem 72

for the MONK-l Problem 74

4-17 Analyzing different parameter setting of AQDT-2 with the MONK-l data 75

4-18 Comparing AQ15c and AQDT-2 against C45 using the MONK-1 problem 75

4-19 A visualization diagram of the MONK-2 problem 76

for the MONK-2 Problem 78

4-21 Analyzing different parameter setting of AQDT-2 with the MONK-2 data 79

xii

4-22 Comparing AQ15c andAQDT-2 against C45 using the MONK-2 problem 79

4-24 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-35 A visualization diagram of a decision tree learned by AQDT -2 after reducing

4-23 A visualization diagram of the MONK-3 problem 80

for the MONK-3 Problem 82

4-25 Analyzing different parameter setting of AQDT-2 with the MONK-3 data 82

4-26 Comparing AQ15c and AQDT-2 against C45 using the MONK-3 problem 83

4-27 Comparing AQ15c and AQDT-2 against C45 using the breast cancer problem 84

4-28 Comparing AQ15c and AQDT-2 against C45 using the mushroom problem 85

4-29 Decision structures learned by AQDT-2 for different decision-making situations 87

4-30 Comparing decision trees for the Cong Voting-84 data learned by C45 amp AQDT -2 88

4-31 A visualization diagram of decision rules learned by AQ15c for the MONK-2 91

4-32 A visualization diagram shows the testing errors for the rules in Figure 4-31 92

4-33 A visualization diagram of the decision tree learned by AQDT-2 for the MONK-2 93

4-34 A visualization diagram shows testing errors of AQDT-2 decision tree bull 94

the generalization degree to 1 94

DERIVING TASKmiddotORIENTED DECISION STRUCTURES

FROM DECISION RULES

Ibrahim M Fahmi Imam PhD

School of Information Technology and Engineering

Geotge Mason University Fall 1995

Ryszard S Michalski Advisor

ABSTRACT

This dissenation is concerned with research on learning task-oriented decision structures from

decision rules The philosophy behind this research is that it is more appropriate to learn

knowledge and store it in a declarative form and then when a decision making situation occurs

generate from this knowledge the decision structure that is most suitable for the given decision

making situation Learning decision structures from decision rules was first introduced by

Michalski (1978) The flIst implementation of this approach was done by Imam and Michalski

(1993 ab) called AQDT-l

This approach separates the function of generating a knowledge-base from the function of using

the knowledge-base for decision-making The first function focuses on learning accurate

consistent and complete concept description expressed in a declarative form The second function

is performed whenever a new decision-making situation occurrs a task-oriented decision structure

is obtained to suit that situation Thsk-oriented knowledge is defined as knowledge that is adapted

for solving a given decision-making situation (Imam amp Michalski 1994 Michalski amp Imam

1994)

The dissertation introduces the system AQJJT-2 for learning task -oriented decision structures from

decision rules or examples Each decision making situation is defined by a set of parameters that

controls the learning process of the AQDT-2 system The extensive experiments on AQJJT-2 show

that decision structures learned by it usually outperform in terms of accuracy and average size of

the decision structures those learned from examples by other well known systems The results

show also that the system does not work very well with noisy data The system is illustrated and

compared using applications of artificial problems such as the three MONKs problems (Thrun

Mitchell amp Cheng 1991) and the East-West 1iain problem (Michie et al 1994) It was also

applied to real world problems of learning decision structures in the areas of construction

engineering (for determining the best wind bracing design for tall buildings) medical

diagnosis(for learning decision rules for recognizing breast cancer) agriculture diagnosis (for

learning classification rules for distinguishing between poisonous and non-poisonous

mushrooms) and political data (for characterizing democratic and republican voting records)

CHAPTER 1 INTRODUCTION

11 Motivation and Overview

Learning and discovery systems should be able not only to generate and store knowledge but also

to use this knowledge for decision-making The main step in the development of systems for

decision-making is the creation of a knowledge structure that characterizes the decision-making

process The form in which knowledge can be easily obtained may however differ from the form

in which it is most readily used for decision-making It is therefore important to identify the form

of knowledge representation that is most appropIiate for learning (eg due to ease of its

modification) and the form that is most convenient for decision making

A simple and effective tool for describing decision processes is a decision structure which is a

directed acyclic graph that specifies an order of tests to be appliedto an object (or a situation) to

arrive at a decision about that object The nodes of the structure are assigned individual tests

(which may correspond to a single attribute a function of attributes or a relation) the branches are

assigned possible test outcomes or ranges of outcomes and the leaves are assigned a specific

decision a set of candidate decisions with corresponding probabilities or an undetermined

decision A decision structure reduces to a familiar decision tree when each node is assigned a

single attribute and has at most one parent when the branches from each node are assigned single

values of that attribute and when leaves are assigned single definite decisions Thus the problem

of generating a decision structure is a generalization of the problem of generating a decision tree

Decision trees are typically generated from a set of examples of decisions The essential

characteristic of any such method is the attribute selection criterion used for choosing attributes to

be assigned to the nodes of the decision tree being built Such criteria include the entropy

reduction the gain and the gain ratio (Quinlan 1979 83 86) the gini index ofdiversity (Breiman

et al 1984) and others (Cestnik amp Bratko 1991 Cestnik amp Karalic 1991 Mingers 1989a)

3

4

Adecision treedecision structure representations can be an effective tool for describing a decision

process as long as all the required tests can be performed and the decision-making situations it

was designed for remain constant (eg in a doctor-patient example the doctor should detennine

that answers for all symptoms appear in the decision tree) Problems arise when these assumptions

do not hold For example in some situations measuring certain attributes may be difficult or costly

(eg in the doctor-patient example a brain or blood test is needed which is very expensive or the

tools needed are not available) In such situations it is desirable to refonnulate the decision

structure so that the inexpensive attributes are evaluated frrst (assigned to the nodes close to the

root) and the expensive attributes are evaluated only if necessary (by assignment to the nodes far

away from the root) If an attribute cannot be measured at all it is useful to either modify the

structure so that it does not contain that attribute or-when this is impossible-to indicate

alternative candidate decisions and their probabilities A restructuring is also desirable if there is a

significant change in the frequency of occurrence of different decisions (eg in the doctor-patient

example the doctor may request a decision structure expressed in a specific set of symptoms

biased to classify one or more diseases or specify a certain order of testing)

Arestructuring ofa decision structure (or a tree) in order to suit new requirements could be quite

difficult This is because a decision structure is a procedural representation that imposes an

evaluation order on the tests In contrast no evaluation order is imposed by a declarative

representation such as a set of decision rules Tests (conditions) of rules can be evaluated in any

order Thus for a given set of rules one can usually build a huge number of logically equivalent

decision structures (trees) which differ in the test ordering Due to the lack of order constraints

a declarative representation (rules) is much easier to modify to adapt to different situations than a

procedural one (a decision structure or a tree) On the other hand to apply decision rules to make a

decision one needs to decide in which order tests are evaluated and thus needs a decision

structure

An attractive solution to these opposite requirements is to acquire and store knowledge in a

declarative form and transform it to a decision structure when it is needed for decision-making

5

This method allows one to create a decision structure that is most appropriate in a given decisionshy

making situation Because the number of decision rules per decision class is usually small (each

rule is a generalization of a set of examples) generating a decision structure from decision rules

can potentially be perfonned much faster than by generation from training examples Thus this

process could be done on line without any delay noticeable to the user Such virtual decision

structures are easy to tailor to any given decision-making situation

This approach allows one to generate a decision structure that avoids or delays evaluating an

attribute that is difficult to measure in some decision-making situation or that fits well a particular

frequency distribution of decision classes In other situations it may be unnecessary to generate

complete decision structure but it may be sufficient to generate only a part of it which concerns

only with decision classes of interest Thus such an approach has many potential advantages

This dissertation presents a new system called AQflf-2 The AWf-2 system generates a taskshy

oriented decision structure (decision structure that is adapted to the given decision-making

situation) from decision rules The decision rules are learned by either rule leaming system AQ15

(Michalski et al 1986) or system AQ17-DCI which has extensive constructive induction

capabilities (Bloedorn et al 1993)

To associate the decision rules with a given decision-making task AQflf-2 provides a set of

features including 1) enabling the system to include in the decision structure nodes corresponding

to new attributes constructed during the process of learning the decision rules 2) controlling the

degree of generalization needed during the development of the decision structure 3) providing four

new criteria for selecting an attribute to be a node in the decision structure that allow the system to

generate many different but equivalent decision structures from the same set of rules 4) generating

unknown nodes in situations when there is insufficient infonnation for generating a complete

decision structure 5) learning decision structures from discriminant rules as well as

characteristic rules and 6) providing the most likely decision when the decision process stops

due to inability to evaluate an attribute associated with an intennediate node

6

To test the methooology of generating decision structures from decision rules an extensive set of

planned experiments have been designed to test different aspects of the approach The experiments

include testing different combinations of parameters for each sub-function of the approach

analyzing the relatioship between decision rules and the decision structure learned from them and

comparing decision trees learned by the AQUf2 system the well known C45 (Quinlan 1993)

system for learning decision trees from examples Different experiments were designed to examine

the new features of the AQUf approach for learning task-oriented decision structures The

experiments were applied to artificial domains as well as real-world domains including MONK-I

MONK-2 and MONK-3 (Thrun Mitchell amp Cheng 1991) East-West trains (Michie et al

1994) Engineering Design-wind bracings (Arciszewski et al 1992) Mushrooms Breast

Cancer (Mangasarian amp WolbeJg 1990) and Congressional bting Records of 1984 The

MONKs problems are concerned with learning classification rules for robot-like figures MONK-l

requires learning a DNF-type description MONK-2 requires learning a non-DNF-type description

(one that cannot be easily described as a DNF rule using the original attributes) MONK-3 concerns

learning a DNF rule from noisy data The East-West trains dataset is a structural domain that

classifies two sets of trains (Eastbound and Westbound) The Engineering Design-wind

bracing data involves learning conditions for applying different types of wind bracing for ta11

buildings The Mushrooms data is concerned with learning classification rules for distinguishing

between poisonous and non-poisonous mushrooms The Breast Cancer data involves learning

concept descriptions for recognizing breast cancer The congressional voting data includes voting

records on different issues AQUf2 outperformed C45 on the average with respect to both the

predictive accuracy and tree size for most problems Apf-2 did not work very well with noisy

data or with problems that have many rules covering very few examples

12 The Problem Statement

There are many limitations and problems accompanied using decision trees for decision-making

CHAPTER 2 RELATED RESEARCH

21 Learning Decision Trees from Decision Diagrams

The AQDT-2 method proposed here is based on the earlier work by Michalski (1978) which

introduced an algorithm for generating decision trees from decision lists The method proposed

several attribute selection criteria These criteria are of increasing power of the main criterion

order cost estimate (the nth order cost estimates n=l 2 ) Michalski also analyzed two

specific criteria MAL and DMAL for selecting the optimal attribute for a node in the tree

based on properties extracted from the decision diagram In order to better explain the method

it is necessary to define some terms These terms will be used later in the dissertation

Definition 2-1 A ~ of a decision class C is a set of rules whose union includes the set of all

examples of class C and does not include any examples of other decision classes

Definition 2-2 A ~ of a decision class C is disjoint if all rules (rules) are pairwise logically

disjoint In other words for any two rules there exists a condition with the same attribute but

with different values in each rule

Definition 2-3 An minimal cover is a disjoint cover which has the smallest number of rules

among all possible covers

Definition 2-4 A diagram for a given cover is a table constructed graphically by representing

in two dimension space of all possible combination of attributes values and locating on the

diagram all the condition parts of the given rules and marking them with the action specified

with each rule

Michalskis algorithm requires construction of minimalmiddot cover The minimal cover should be

consistent and complete The method is based on the fact that if there are n decision classes

any decision tree that correctly classifies the given rules and has n distinct leaves is a minimal

7

8

decision tree (any consistent decision tree should have at least n leaves) Michalski (1978) has

shown that if only one rule is broken by a selected attribute then instead of having one leaf

(which could potentially represent this rule or the decision class in the tree) there will have to

be at least two leaves representing this rule in the fmal decision tree

The attribute selection criterion MAL introduced in (Michalski 1978) prefers attributes that do

not break any rules or break as few as possible An attribute breaks a rule if the attribute can

divide the rule into two or more sub-rules Figure 2-1 shows two examples of two sets of rules

In the fJIst xl and x3 break at least two rules each (xl breaks the rule [x4=2] amp [x1=lv3] amp

[x3=2v3] and the rule [x4=3] amp [x1=1v2] amp [x3=1] and x3 breaks three rules the rule [x4=2]

amp [xl=2] amp [x3=2v3]the rule [x4=l] amp [xl=3] amp [x3=lv3] and the rule [x4=3] amp [xl=4] amp

[x3=2v3]) In the diagram on the right xl is the only attribute that does not break any rule

Figure 2-1 An example to illustrate how attributes break rules

One of the criteria defmed by Michalski is the first degree cost estimate which assigns to each

attribute an integer equal to the number of rules broken by that attribute This criterion is also

called the static cost estimate of an attribute or the criterion of minimizing added leaves

(MAL)

-

C3

9

The MAL criterion (Minimizing Added Leaves) seeks an attribute that minimizes the

estimated number of additional nodes in the decision tree being generated over a hypothetical

minimal decision tree When there is a tie between two attributes the attribute to be selected is

the one which breaks smaller rules (rules that cover fewer examples or more specialized

rules) AQDT-2 uses an approximate version of this criterion (the attribute dominance)

Another criterion introduced by Michalski was the DMAL criterion The DMAL criterion

(Dynamically Minimizing Added Leaves) is based on a principle similar to that of MAL but

is more complex because once an attribute is selected as a node in the tree some rules andor

parts of the broken rules at each branch are merged into one rule The DMAL ensures that the

value of the total cost estimate of an attribute is decreased by a value equal to the number of

merged rules minus one

Example Learn a decision tree from the following decision table

The minimal cover consists of the following rules

Al lt [x2=O] v [x1=0][x2=2] A2 lt [x2=I] v [xl=2][x2=2] A3 lt [xl=I][x2=2]

The evaluations ofthe MAL criterion for these attributes are 2 for xl 0 for x2 5 for x3 and 5

for x4 The attribute xl divides the two rules [x2=O] and [x2=I] so selecting it will add two

leaves to the optimal number of leaves It is clear that the attribute to be selected as a root of

10

the decision tree is x2 Then three branches are attached to the root node and the decision rules

are divided into subsets each corresponding to one branch For x2 = 0 or 1 a leave node is

generated For x2=2 another attribute is selected to be a node in the tree In this case xl has the

minimum MAL value Figure 2-2 shows the decision tree obtained using the MAL criterion

Al A3 A2

Figure 2-2 A decision tree learned from the decision table in Table 2-1

22 Learning Decision Trees from Examples

Decision tree learning is a field that concerned with generating decision tree that classifies a set

of examples according to the decision classes they belong to The essential aspect of any

inductive decision tree method is the attribute selection criterion The attribute selection

criterion measures how good the attributes are for discriminating among the given set of

decision classes The best attribute according to the selection criterion is chosen to be assigned

to a node in the tree The fIrst algorithm for generating decision trees from examples was

proposed by Hunt Marin and Stone (1966) Hunts algorithm uses a divide and conquer

algorithm for building decision trees This algorithm has been subsequently modified by

Quinlan (1979) and applied by many researchers to a variety of learning problems

Attribute selection criteria can be divided into three categories These categories are logicshy

based information-based and statistics-based The logic-based criteria for selecting attributes

use logical relationships between the attributes and the decision classes to determine the best

attribute to be a node in the decision tree such as the MAL criterion minimizing added leaves

(Michalski 1978) which uses conjunction and disjunction operators The information-based

criteria are based on the information theory These criteria measure the information conveyed

11

by dividing the training examples into subsets Examples of such criteria include the

information measure 1M the entropy reduction measure and the gain criteria (Quinlan 1979

83) the gini index of diversity (Breiman et al 1984) Gain-ratio measure (Quinlan 1986) and

others (Clark amp Niblett 1987 Bratko amp Lavrac 1987 Cestnik amp Karalic 1991) The

statistics-based criteria measure the correlation between the decision classes and the attributes

These criteria use statistical distributions for determining whether or not there is a correlation

The attribute with the highest correlation is selected to be a node in the tree Examples of

statistics-based criteria include Chi-square and G-statistic (Sokal amp Rohlf 1981 Han 1984

Mingers 1989a)

Niblett and Bratko (1986) Quinlan (1987) and Bratko and Kononeko (1987) extended the

method of learning decision trees to also handle data with noise (by pruning) Handling noise

extended the process of learning decision trees to include the creation of an initial complete

decision tree tree pruning which is done by removing subtrees with small statistical Validity

and replacing them by leaf nodes (MiDgers 1989b) More recently pruning has also been used

for simplifying decision trees even for problems without noise (Bohanec amp Bratko 1994)

Pruning decision trees improves their simplicity but reduces their predictive accuracy on the

training examples Quinlan (1990) proposed also a method to handle the unknown attributeshy

value problem by exploring probabilities of an example belonging to different classes

The rest of this section includes a brief description of the attribute selection criterion used by

C45 learning system (Quinlan 1993) C45 uses an information-based criterion for selecting

an attribute to be a node in the tree Also the section includes a brief description of the Chishy

square method for attribute selection (Mingers 1989a) The method is a statistics-based

method for selecting an attribute to be a node in the tree

221 Building Decision Trees Using htformation-based Criteria

12

This section presents a description of the inductive decision tree learning system C4S The

C4S learning system is considered to be one of the most stable accurate and fastest program

for learning decision trees from examples

Learning decision trees from examples requires a collection of examples Each example is

represented by a fixed number of attribute-value pairs C45 (Quinlan 1993) is a learning

program that induces classification decision trees from a set of given examples The C4S

learning system is descended from the learning system ID3 (Quinlan 1979) which is based on

Hunts method for constructing decision tree from a set ofcases (Hunt Marin amp Stone 1966)

The C45 system uses an attribute selection criterion called the Gain Ratio This criterion

calculates the gain in classifying information based on the residual information needed to

classify cases in a set of training examples and the information yielded by the test based on the

relative frequencies of the possible outcomes (decision classes) The gain ratio criterion is

based on an earlier criterion used by ID3 called the Gain Criterion The Gain Criterion uses

the frequency of each decision class in the given set of training examples

Once an attribute is chosen to be a node in the tree the system generates as many links as the

number of its values and classifies the set of examples based on these values If all the

examples at a certain node belong to one decision class the system generates a leaf node and

assigns it to that class Otherwise the system searches for another attribute to be a node in the

tree

The Gain Criterion The gain criterion is based on the information theory That is the

information conveyed by a message depends on its probability and can be measured in bits as

minus the logarithm (base 2) of that probability To explain the gain criterion suppose that for

a given problem Xu bullbull X are given attributes and Cu c are decision classes Suppose S is

any set of cases and T is the initial set of training cases The frequency of class C1 in the set S

is the number of examples in S that belong to class C i bull

13

freq(Cit S) = Number of examples in S belong to CI (2-1)

Suppose that lSI is the total number of examples in S the probability that an example selected

at random from S belongs to class Ci is freq(CIbullS) IISI

The information conveyed by the message that a selected example belongs to a given decision

class Cit is determined by -log1 (freq(Ch S) IISI) bits

The expected information from such a message stating class membership is given by

info(S) = -k

L (freq(C S) I lSI) log1 (freq(C S) IISI) bits (2-2)I

info(S) is known also as the entrQpy of the set S When S is the initial set of training examples

info(T) determines the average amount of information needed to identify the class of an

example in T

Suppose that we selected an attribute X to be the root of the tree and suppose that X has k

possible values The training set T will be divided into k subsets each corresponding to one of

Xs values The expected information of selecting X to partition the training set T infoz(T) can

be found as the sum over all subsets of multiplying the information conveyed by each subset

by its probability

k

infoz(T) =L (ITII 1m) infO(TIraquo (2-3) I

The information gained by partitioning the training examples T into subset using the attribute

X is given by

gain(X) =inlo(T) - inoIT) (2-4)

The attribute to be selected is the attribute with maximum gain value

The Gain Ratio Criterion This criterion indicates the proportion of information generated by

the split that appears helpful for classification Quinlan (1993) pointed out that the gain

criterion has a serious deficiency Basically it is strongly biased toward attributes with many

outcomes (values) For example for any data that contains attributes such as social security

14

number the gain criterion will select that attribute to be the root of the decision tree However

selecting such attributes increases the size of the decision tree Quinlan provided a solution to

this problem by introducing the gain ratio criterion which takes the ratio of the information that

is gained by partitioning the initial set of examples T by the attribute X to the potential

information generated by dividing T into n subsets

Following similar steps to obtain the information conveyed by dividing T into n subsets The

expected information generated by dividing T into n subsets and by analogy to equation 2-2 is

determined by

split info(T) = - Lt

(ITII lTI) 10gl (ITII m ) (2-5) I

The gain ratio is given by

gain ratio(X) = gain(X) I split info(X) (2-6)

and it expresses the proportion of information generated by the split that is useful for

classification

Example Consider the following example presented by Quinlan (1993) Table 2-2 shows the

set of training examples

First determine the amount of information gained by selecting the attribute outlook to be a

root of the decision tree This attribute divides the training examples into three subsets

sunny-with five examples two of which belong to the class Play overcast-with four

examples all of which belong to the class Play and rain-with five examples three of

which belong to the class Play To determine info(11 the average information needed to

identify the class of an example in T there are 14 training examples and two decision classes

Nine of these examples belong to the class Play and five belong to the class Dont Play

Info(11 = - 914 10gl (914) - 514 10gl (514) = 094 bits

When using outlook to divide the training examples the information becomes

15

infO(T) = 514 -25 log2 (25) - 35 10g2 (35)

+ 414 -44 10g2 (44) - 04 log2 (04)

+ 514 -35 logl (35) - 25 logl (25) = 0694 bits

By substituting in equation 2-4 the gain of information results from using the attribute

outlook to split the training examples equal to 0246 The gain information for windy is

0048

Figure 2-3 shows a decision tree learned for this problem using the gain criterion The split

information for outlook is determined as follows

split info(T) = - 514 logl (514) - 414 log2 (414) - 514 log2 (514) = 1577 bits

The gain ratio for outlook = 02461577 =0156

16

Figure 2 -3 A decision tree learned using the gain criterion for selecting attributes

The C45 system handles discrete values as well as continuous values To handle an attribute

with continuous values C45 uses a threshold to transform the continuous domain into two

intervals In other words for each continuous attribute C45 generates two branches one

where the values of that attribute is greater than the determined threshold and the other if the

value is less than or equal to the threshold

Tree pruning in C45 is a process of replacing subtrees with small classification validity by

leaves The C45 system uses Laplace ratio for determining the error rate of different subtrees

This ratio is defined as (e+l) (n+2) where n is the number of the training examples and e is

the number of misclassified examples at a given leaf

222 Building Decision Trees Using Statistics-based Criteria

The Chi-square method for selecting attributes was used by Hart (1984) and Mingers (1989a)

in building decisionmiddot trees The method uses Chi-square statistics to measure the association

between two attributes When building decision trees the method is implemented such that it

determines the association between each attribute and the decision classes The attribute to be

selected is the one with greatest value

To determine the Chi-square value for an attribute consider aij is the number of examples in

class number i where the attribute A takes value numberj In other words aij is the frequency

of the combination of decision class number i and the attribute value number j The Chi-square

value for attribute A is given by

n m

Chi-square (A) =L L [ (aij - Eijl2 Eij ] (2shy

i=1 j=1

where n is the number of decision classes m is the number ofvalues of a given attribute Also

17

Eij = (TCi TVj ) T (2shy

8)

where T Ci and T Vj are the total number of examples belong to the decision class a and the total

number of examples where the attribute A takes value vj respectively T is the total number of

examples

Consider Quinlans example in Table 2-2 Table 2-3 shows the frequency of different

combination of values between the decision class and both the outlook and the Windy

attributes Table 2-4 shows the expected values TCi and T Vj of the frequencies in Table 2-3

of different attributes values for different decision classes

To determine the association value between the decision classes and both the attribute Windy

and the attribute Outlook the observed Chi-square values are

Chi-square (Windy Class) =[(3-39)239] + [(3-21)221] + [(6-51)232] + [(2-29)232]

=021 + 039 + 025 + 025 = 11

Chi-square (Outlook Class) = [(2-32)232] + [(4-26)226] + [(3-32)232] + [(3-18118]

+ [(0-14)214] + [(2-18)2 18] =045 + 075 + 02 + 08 + 14 + 002 = 362

18

Applying the same method to the other attributes the results will favor the attribute Outlook

Once that attribute is selected to be a node in the tree the remaining set of examples are divided

into subsets and the same process is repeated on each subset

Table 2-5 shows a summary of these criteria and their basic evaluation function

Table 2-5 Attribute selection criteria and their basic evaluation measure

Info Measure (IM) Gain Gshystatistics and Gain Ratio Chi-square

II

Entropy(S) = - L (freq(~ 5) 1151) logz (freq(CIt 5) 1151 ) I

G-Statistics =2N 1M (N-No of examples) n m

Chi-square (A B) = L L [(aij - ~j)21Eij ] i=l

223 Analysis of Attribute Selection Criteria

This subsection introduces briefly the analysis of different selection criteria which was done by

Mingers (1989a) Mingers compared six attribute selection criteria that are used in decision

tree programs These criteria are the Information Measure (IM) Chi-square G statistics Gin

index of diversity Marshal correction and Gain RaJio The overall results show that the Gain

Ratio criterion had the strongest xesults

In the flISt experiment Minger tested the six criteria on a problem with ambiguous examples

(ie examples may belong to more than one decision class) to observe how the selected criteria

evaluate the given attributes The problem has two decision classes and two attributes X and

Y It was assumed that attribute X is better for classifying the examples than attribute Y The

training examples were unevenly spread between the two values of X Attribute Y has three

values and the examples were spread randomly among them Table 2-6 (a and b) shows the

contingency tables for both attributes Table 2-7 shows a summary of the goodness of split

provided by the six criteria Mingers noted that the measures that are not based on information

19

theory give radiation (attribute X here) less weighL This may be because the zero in the fIrst

row of radiation has a greater influence in the log calculation In the case of using the Chishy

square criterion the value zero adds the maximum association between any two attributes

because the Chi-square value of a zero cell is the expected value of this cell

b)

Now let us demonstrate results from another experiment done by Mingers In this experiment

Mingers used four different data sets to generate decision trees for eleven different criteria In

the final results he compared the total number of nodes and the total error rate provided by

each criterion over all given problems Table 2-8 shows the final results for five selected

criteria only

20

U lgt ~middot results for comparing the total accuracy and size of decision trees different attribute selection criteria from four VVLU

This experiment was performed on four real world data sets These data are concerned with

profiles of BA Business Studies degree students recurrence of breast cancer classifying types

of Iris and recognizing LCD display digits The data was divided randomly 70 for training

and 30 for testing For more details see Mingers (1989a)

23 Learning Decision Structures

Considering the proposed defInition of decision structures given above two related lines of

research are described in this section Gaines (1994) and Kohavi (1994) proposed two

approaches for generating decision structures that share some of the earlier ideas by Imam and

Michalski (1993b)

In the first approach Brian Gaines introduces a method for transforming decision rules or

decision trees into exceptional decision structures The method builds an Exception Directed

Acyclic Graph (EDAG) from a set of rules or decision trees The method starts by assigning

either a rule say RO or a conclusion say CO or both to the root node If it assigns a conclusion

to the root node it places it on a temporary conclusion list Then it generates a new child node

and add to it a new rule say Rl The method evaluates the new rule Rl if it satisfies the

conclusion CO in its parent node it builds a new child and repeats the processes Otherwise if

the rule Rl does not satisfy the conclusion CO it changes the temporary memory with the new

conclusion Cl which is satisfied by the rule RO The same process is repeated until all rules

that have common conditions with the rule at the root are evaluated The method then creates a

21

new child node from the root and repeat the process until all rules are evaluated In the decision

structure nodes containing rules only represent common conditions to all of its children

The main disadvantages of this approach is that it requires discriminant rules to build such a

decision structure Also such a structure is more complex than the traditional decision trees

that are used for decision-making

Figure 2-4 shows an example of set of rules and their equivalent exceptional decision structure

The decision structure can be read as follows It is Safe except if xl=1 amp x2=1 amp x3=1 and

either x4=3 or x5=I then it is Lost except if x6=1 it is Safe except if x7=1 it is Lost

The second approach introduced by Ronny Kohavi learns decision structures from examples

using a bottom-up algorithm The method uses a greedy Hill-climbing algorithm for inducing

Oblivious read-Once Decision Graphs (HOODO) A read-once decision graph is defined as a

decision graph where each attribute occurs at most once along any computational path In other

words for each path from the root to the leaves of the decision structure attributes may occur

as a node once at most However there may be more than one node with the same attribute in

the decision graph Kohavi also defined a leveled decision graph in which the nodes are

partitioned into a sequence of pairwise disjoint sets such that outgoing edges from each level

terminate at the next level An oblivious decision graph is a decision graph where all nodes at

a given level are labeled by the same attribute

22

Safe lt [xl=2] Safe lt [x2=2] Safe lt [x3=2] Safe lt [x4=1] amp [x5=2] Safe lt [x4=1] amp [x5=3] Safe lt [x6=l] amp [x7=2] Safe lt [x6=1] amp [x7=3] Safe lt [x4=2] amp [x5=2] Safe lt [x4=2] amp [x5=3]

Lost lt [x6=2] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x6=3] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x5=1] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=2] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=3] amp [x3=1] amp [x2=I] amp [xl=l]

The algorithm starts by generating a leaf node for each decision class Then it applies a

nondeterministic method to select an attribute to build new level of the decision structure This

attribute is removed from the data and the data is divided into subsets each corresponding to a

complex combination of that attributes values For each subset the process is repeated until all

the examples of a given subset belong to one decision class For example if the selected

attribute say A has two values (0 and 1) and there are two decision classes (CO and CI) The

data is divided into two subsets the fIrst subset contains examples where A takes value 0 and

belong to class CO or takes value 1 and belong to class Cl The second subset of examples is

the set where A takes value 0 and belong to class CI or takes value 1 and belong to class CO

The number of nodes of the fIrst level (after the leave nodes) is expected to be less than or

equal to ka where k is the number of decision classes and n is the number of values of the

selected attribute and it can increase exponentially before that number is reduced

exponentially to one

It is easy for the reader to fIgure out some major disadvantages of such approach including

The average size of such decision structures is estimated to be very large especially when there

23

is no similarity (ie strong patterns) or logical relationship in the data The time used to learn

such decision structure is relatively very high comparing to systems for learning decision trees

from examples and fmally it could be better to search for attribute which reduces the number

of generated subsets of the data instead of nondeterministically selecting an attribute to build a

new level of the decision structure Kohavi provided a comparison between the C45 system

and his system that can be found in (Kohavi 1994) Later Kohavi and Li (1995) used a

deterministic method for attribute selection which minimizes the width of the penultimate level

of the graph

Table 2-9 shows a comparison between the proposed approach and those two approaches The

EDAG and HOODG systems are unreleased prototype systems

or Decision structures are easy to Decision structures are Decision structures are easy to understand difficult to read understand

CHAPTER 3 DESCRIPTION OF THE APPROACH

31 General Methodology

In the proposed approach the function of learning or discovery is separated from the function of

using the discovered knowledge for decision-making The frrst function is performed by an

inductive learning program that searches for knowledge relevant to a given class of decisions

and stores the learned knowledge in the form of decision rules The second function is performed

when there is a need for assigning a decision to new data points in the database (eg bull a

classification decision) by a program that transforms obtained knowledge into a decision

structure optimized according to the given decision-making situation

The Learning Task GivenA set of training examples describing the concept to be learned Learning goal which specifies the decision classes to be learned from the training examples A background knowledge to control the learning process Determine A concept description in a declarative form of knowledge (decision rules) that satisfies the learning goal

The Decision-making Task GivenA set of decision rules in a conjunctive form A description of the new decision-making situation (eg attributes costs and order preference importance or frequency of decision classes etc) One or more examples that needed to be tested under the given decisionshymaking situation A set of parameters to control the learning process Determine A decision structure that suits the given decision-making situation

The decision rules used here are learned by either the AQ1S (Michalski et al 1986) or AQ17

(Bloedorn et aI 1993) learning systems The reasons for selecting decision rules as a form of

23

24

declarative knowledge are that they do not impose any order on the evaluation of the attributes

and due to the lack of order constraints decision rules can be evaluated in many different

ways which increases the flexibility of adapting them to the different tasks of decision-making

(Bergadano et al 1990) Since the number of rules learned per class is typically much smaller

than the number of examples per class generating a decision structure from decision rules can

potentially be done on line

Such virtual decision structures can be tailored to any given decision-making situation The

needed decision rules have to be generated only once and then they can be used many times for

generating decision structures according to changing requirements of decision-making tasks The

method uses the AQDT-2 system (Imam amp Michalski 1994) for learning decision structures

from decision rules Decision structures represent a procedural form of knowledge which makes

them easy to implement but also harder to change Consequently decision structures can be quite

effective and useful as long as they are used in decision-making situations for which they are

optimized and the attributes specified by the decision structure can be measured without much

cost Figure 3-1 shows an architecture of the proposed methodology

Data

Decision -

- Learning knowledge from database The decision making process

Figure 3 -1 Architecture of the AQDT approach

25

It is assumed that the database is not static but is regularly updated A decision-making problem

arises when there is a case or a set of cases to which the system has to assign a decision based on

the knowledge discovered Each decision-making situation is defmed by a set of attribute-values

Some attribute-values may be missing or unknown A new decision structure is obtained such

that it suits the given decision-making problem The learned decision structure associates the

new set of cases with the proper decisions

32 A Brief Description of the AQIS and AQ17 Rule Learning Programs

The decision rules are generated from examples by an AQ-type inductive learning system

specifically by AQ15 (Michalski et albull 1986) or AQI7-DCI (Bloedorn at al 1993) The rest of

this subsection includes a brief description of the inductive learning systems AQ15 and AQI7

AQ15 learns decision rules for a given set of decision classes from examples of decisions using

the STAR methodology (Michalski 1983) The simplest algorithm based on this methodology

called AQ starts with a seed example of a given decision class and generates a set of the

most general conjunctive descriptions of the seed (alternative decision rules for the seed

example) Such a set is called the star of the seed example The algorithm selects from the star

a description that optimizes a criterion reflecting the needs of the problem domain If the

criterion is not defined the program uses a default criterion that selects the description that

covers the largest number of positive examples (to minimize the total number of rules needed)

and with the second priority that involves the smallest number of attributes (to minimize the

number of attributes needed for arriving at a decision)

If the selected description does not cover all examples of a given decision class a new seed is

selected from uncovered examples and the process continues until a complete class description

is generated The algorithm can work with few examples or with many examples and can

optimize the description according to a variety of easily-modifiable hypothesis quality criteria

26

The learned descriptions are represented in the form of a set of decision rules expressed in an

attributional logic calculus called variable-valued logic 1 or VLl (Michalski 1973) A

distinctive feature of this representation is that it employs in addition to standard logic operators

the internal disjunction operator (a disjunction of values of the same attribute in a condition) and

the range operator (to express conditions involving a range of discrete or continuous values)

These operators help to simplify rules involving multi valued discrete attributes the second

operator is also used for creating logical expressions involving continuous attributes

AQ15 can generate decision rules that represent either characteristic or discriminant concept

descriptions depending on the settings of its parameters (Michalski 1983) A characteristic

description states properties that are true for all objects in the concept The simplest

characteristic concept description is in the form of a single conjunctive rule (in general it can be

a set of such rules) The most desirable is the maximal characteristic description that is a rule

with the longest condition part ie stating as many common properties of objects of the given

class as can be determined A discriminant description states properties that discriminate a given

concept from a fixed set of other concepts The most desirable is the minimal discriminant

descriptions that is a rule with the slwnest condition part For example to distinguish a given

set of tables from a set of chairs one may only need to indicate that tables have large flat top

A characteristic description of the tables would include also properties such as have four legs

have no back have four corners etc Discriminant descriptions are usually much shorter than

characteristic descriptions

Another option provided in AQ15 controls the relationship among the generated descriptions

(rulesets or covers) of different decision classes In the IC (Intersecting Covers) mode

rulesets of different classes may logically intersect over areas of the description space in which

there are no training examples In the DC (Disjoint Covers) mode descriptions of different

classes are logically disjoint The DC mode descriptions are usually more complex both in the

number of rules and the number of conditions There is also a DL mode (a Decision List mode

27

also called VL modeM-for variable-valued logic mode) in which the program generates rule sets

that are linearly ordered To assign a decision to an example using such rulesets the program

evaluates them in order If ruleset is satisfied by the example then the decision is made

otherwise the program proceeds to the evaluation of the ruleset +1 In IC and DC modes

rule sets can be evaluated in any order

Alternatively the system can use rules from the AQ17-DCI program for Data-driven

Constructive Induction AQ17-DCI differs from AQ15 mainly in that it contains a module for

generating additional attributes These attributes are various logical or mathematical

combinations the original attributes The program generates a large number of potential new

attributes and seleets from them those most promising based on an attribute quality criterion

To illustrate the format of rules generated by AQ15 (or AQ17-DCI) an exemplary ruleset is

shown in Figure 3-2 The ruleset (that can be re-represented as a disjunctive normal form

expression) describes a voting record of Democratic Representatives in the US Congress

Each rule is a conjunction of elementary conditions Each condition expresses a simple relational

statement For example the condition [State = northeast v northwest] states that the attribute

State (of the Representative) should take the value northeast or northwest to satisfy the

condition

Rl [Gas_concban = yes] amp [Soc_see_cut = no v not registered] R2 [Draft = Yes v not registered]amp[AlaslaLparks = yes v not registered] amp

[Food_stamp_cap = no] amp [State = northeast v northwest] R3 [Chrysler =yes v not registered] amp [Income =low] R4 [Education =yes] amp [Occupation = yes]

Figure 3-2 A ruleset generated by AQ15 for the concept Voting pattern of Democratic Representatives

The above rules were generated from examples of the voting records For illustration below is

an example of a voting record by a Democratic representative

Draft registration=no Ban of aid to Nicaragua=no Cut expenditure on mx missiles=yes Federal subsidy to nuclear power stations=yes Subsidy to national parks

28

in Alaska=yes Fair housing bill=yes Limit on Pac contributions=yes Limit onood stamp program=no Federal help to education=no StateFrom=north east State Population=large Occupation =unknown Cut in social security spending=no Federal help to Chrysler corp=not registered

By expressing elementary statements in the example as conditions and linking conditions by

conjunction the examples can be re-expressed as decision rules Thus decision rules and

examples formally differ only in the degree of generality

33 Generating Decision Structurestrees from Decision Rules

This section describes the AQDT-1 system for learning decision structures (trees) from decision

rules (Imam amp Michalski 1993ab) Also a description of the AQDT-2 method for learning taskshy

oriented decision structure from decision rules is included and fmally the methodology is

illustrated by two examples

Methods for learning decision trees from examples have been very popular in machine learning

due to their simplicity Decision trees built this way can be quite efficient as long as they are

used in decision-making situations for which they are optimized and these situations remain

relatively stable Problems arise when these situations significantly change and the assumptions

under which the tree was built do not hold anymore For example in some situations it may be

difficult to determine the value of the attribute assigned to some node One would like to avoid

measuring this attribute and still be able to classify the example if this is potentially possible

(Quinlan 1990) If the cost of measuring various attributes changes it is desirable to restructure

the tree so that the inexpensive attributes are evaluated f11st A tree restructuring is also

desirable if there is a significant change in the frequency of occurrence of examples from

different classes A restructuring of a decision tree to suit the above requirements is however

difficult to do The reason for this is that decision trees are a form of decision structure

representation and imposes constraints on the evaluation order of the attributes that are not

logically necessary

29

One problem in developing a method for generating decision structures from decision rules is to

design an attribute selection criterion that is based on the properties of the rules rather than of

the training examples A decision rule normally describes a number of possible examples Only

some of them are examples that have actually been observed ie training examples An attribute

selection criterion is needed to analyze the role of each attribute in the rules It cannot be based

on counting the numbers of training examples covered by each attribute-value and the

frequency of decision classes in the training examples as is done in learning decision trees from

examples because the training examples are assumed to be unavailable

Another problem in learning decision trees from decision rules stems from the fact that decision

rules constitute a more powerful knowledge representation than decision trees They can directly

represent a description in an arbitrary disjunctive normal form while decision trees can represent

directly only descriptions in the disjoint disjunctive normal form In such descriptions all

conjunctions are mutually logically disjoint Therefore when transforming a set of arbitrary

decision rules into a decision tree one faces an additional problem of handling logically

intersecting rules

The solution to both problems (attribute selection and logically intersected rules) in the AQDT-2

system is based on the earlier work by Michalski (1978) which introduced a general method for

generating decision trees from decision rules The method aimed at producing decision trees

with the minimum number of nodes or the minimum cost (where the cost was defined as the

total cost of classifying unknown examples given the cost of measuring individual attributes and

the expected probability distribution of examples of different decision classes) More

explanations are provided in the following section

331 Tbe AQDT-2attribute selection metbod

This section describes the AQDT -2 method for building a decision structure from decision rules

The method for building a single-parent decision structure is similar to that used in standard

30

methods of building a decision tree from examples The major difference is that it assigns tests

(attributes) to the nodes using criteria based on the properties of the decision rules (this includes

statistics about the examples covered by each rule in the case of learning rules from examples)

rather than statistics characterizing the frequency of training examples per decision classes per

attribute-values or per conjunctions of both Other differences are that the branches may be

assigned an internal disjunction of values (not only a single value as in a typical decision tree)

and leaves may be assigned a set of alternative decisions with probabilities Also the tests can be

attributes or names standing for logical or mathematical expressions that involve several

attributes or variables In the following we use the terms test and attribute interchangeably

(to distinguish between an attribute and a name standing for an expression the latter is called a

constructed attribute)

At each step the method chooses the test from an available set of tests that has the highest utility

(see below) for the given set of decision rules This test is assigned to the node The branches

stemming from this node are assigned test values or disjoint groups of values (in the form of

logical disjunction if such occur in the rules subsumed groups of values are removed) Each

branch is associated with a reduced set of rules determined by removing conditions in which the

selected attribute assumes value(s) assigned to this branch If all rules in the reduced ruleset

indicate the same decision class a leaf node is created and assigned this decision class The

process continues until all nodes are leaf nodes If it is not possible to reduce further the rule set

because some attribute is declared as unavailable (infmite cost) then the leaf is assigned a set of

candidate decisions with associated probabilities (see Sec 42)

The test (attribute) utility is a combination of one or more of the following elementary criteria 1)

cost which indicates the cost of using each attribute for making decision 2) disjointness which

captures the effectiveness of the test in discriminating among decision rules for different decision

classes 3) importance which determines the importance of a test in the rules 4) value

31

distribution which characterizes the distribution of the test importance over its of values and 5)

dominance which measures the test presence in the rules These criteria are dermed below

~ The cost of a test expresses the effort or cost needed to measure or apply the test

Djsjointness The disjointness of a test is defined as the sum of the class disjointness-the

disjointness of the test for each decision class Suppose decision classes are Cl C2 bull Cm and

decision rule sets for these classes have been determined Given test A let V 1 V 2 bull V m denote

sets of the values (outcomes) of A that are present in rule sets for classes C 1 C2bullbull Cm

respectively If a ruleset for some class say Ct contains a rule that does not involve test A than

Vt is the set of all possible values of A (the domain of A)

Definition 3 -1 The degree ofclass disjointness D(A Ci ) of test A for the ruleset ofclass Ci is

the sum of the degrees of disjointness D(A Cit Cj) between the ruleset for Ci and rulesets for

Cj j=l 2 m j J L The degree of disjointness between the ruleset for Ci and the ruleset for Cj is

dermed by

if Vi Vj ifVj gtVj (3-1) ifVinVj -tljgt amp -tVi amp -tVj ifVinVj=1jgt

where cjgt denotes the empty set Note that exchanging the second and third conditions of equation

(3-1) may seem to be an improved criteria However it does not clearly distinguish between both

cases (Le for both situations the disjointness will be similar) The current equation is better

because it gives higher scores to attributes that classify different subsets of the two decision

classes than attributes that classify only a subset of one decision class

Definition 3-2 The disjointness of the test A for evaluating a given set of decision rules is the

sum of the degrees of class disjointness of each decision class

32

m m Disjointness(A) = L D(A cn where D(A cn = L D(A Ci Cj) (3-2)

i=l i=l i~j

The disjointness of a test ranges from 0 when the test values in rulesets of different classes are

all the same to 3m(m-l) when every rule set of a given class contains a different set of the test

values If two tests have the same disjointness value the attribute to be selected is the one with

less number of values

Definition The A vera~ Number of Tests (ANT) required to make decision from a decision tree

is dermed as the average number of tests (attributes) to be examined from the root of the tree to

any leaf node in order to reach a decision

Definition A decision structure is a one-node-per-level decision structure if at each level there is

only one node and zero or more leaves

Such decision structure can be generated by combining together all branches that associated with

each one a set of decision rules belongs to more than one decision class

Theorem 1 Consider learning one-node-level decision structure for a database with two

decision classes The disjointness criterion ranks first attributes that add minimum number of

tests to the decision tree

Proof Consider all possible distributions of an attributes values within two decision classes Ci

and Cj There are three cases 1) subset (same as super set) 2) non-empty intersection but not

subset 3) no intersection Figure 3-3 shows all possible distributions (the case of having the

same set of values in both classes is trivial one) Consider that branches leading to one subset

with the same decision class are combined into one branch In the first case there will be two

branches only The first leads to a leaf node and the other leads to an intermediate node where

another attribute to be selected The minimum ANT in this case is 53 In the second case three

branches should be created Two branches leads to a leaf node where all values at each branch

belong to only one and different decision class The third branch leads to an intermediate node

33

where another attribute should be selected furtur classifies the decision classes The minimum

ANT in this case is 64 In the third case only two branches will be generated where each leads

to leaf node with different decision class In this case the minimum ANT is 1

Figure 3-4 shows decision trees equivalent to selecting the corresponding attribute Note that in

case of having more than one attribute-value at branches lead to leaves belonging to one decision

class they will be combined into one branch in the decision structure The symbol 1 means

that an attribute is needed to classify the two decision classes In such cases there will be at least

two additional paths

D(A Ci) = 0 D(A CJ) = 1 D(A Ci) =2 D(A Cj) = 2 D(A Ci) =3 D(A Cj) =3

Figure 3-3 Yen Diagrams of Possible combinations of attribute values in two decision classes

The average number of tests required for making a decision in each possible case is determined

in Figure 3-4 It is clear that in the case of two decision classes the disjointness criterion ranks

highly attributes that reduces the average number of tests required for decision-making The

theorem can be proved in the general case

i ~ i Cj C1middot bull Ci Cj

ANT=32middotANT=53 ANT=l

1 means at least one attribute is needed to complete the decision tree Figure 3-4 Decision trees corresponding the Yen diagrams inFigure 3-3

34

Theorem 2 The attribute with the highest non~zero disjointness is the best attribute to classify

the decision classes

Proof Suppose that the number of decision classes is n Assume also that there are two attributes

A and B where D(A) lt D(B) Since the disjointness criterion considers the mutual relationship

between any two decision classes this means that there are more decision classes where D(A

Ci) lt D(B Ci) than those where D(A Ci) gt D(B Ci) (ignoring classes with equal disjointness

for bCth attributes) Hence there are more decision classes where D(A Ci Cj) lt D(B Ci Cj)

than those where D(A Ci Cj) gt D(B Cit Cj) Let us frrst prove that if D(A Ci Cj) lt D(B Cit

Cj) then B classifies the decision classes better than A

For each pair of decision classes Ci and Cj the possible values of the disjointness of any attribute

are 0 14 or 6 For all positive values ofD(B)= 14 or 6 it is clear that attribute B should have

smaller ANT than attribute A which has lower disjointness This means that if D(A Cit Cj) lt

D(B Cit Cj) then attribute B can classify both decision classes better than attribute A

Similarly B is better for classifying more pairs of decision classes than A This implies that B is

a better classifier than A

Importance The second elementary criterion the importance of a test is based on the

importance score (IS) introduced in (Imam Michalski amp Kerschberg 1993) In the obtained

rules each test is assigned a score that represents the total number of training examples that

are covered by the rules involving this test Decision rules learned by an AQ learning program

are accompanied with information on their strength Rule strength is characterized by t-weight

and u~weight The t-weight (total-weight) of a rule for some class is the number of examples of

that class covered by the rule The importance score of a test is the aggregation of the totalshy

weights of all rules that contain that test in their condition part Given a set of decision rules for

m decision classes Cl Cm and involving n tests AlbullAn and the number of rules associated

with class Ci is denoted by tlrit The importance score is defmed as follow

35

Definition 3 -3 The importance score IS(Aj) of the test A j is detennined by

m IS(Aj)= L IS(Aj Ci) (3-31)

i=l

where ri IS(Aj Ci) = L Rik(Aj) (3-32)

k=l

and Rik the weight of a test Aj in the rule Rk of class Ci is given by

t - weight if A j belongs to rule Rik Rik (Amiddot) = (3-4)J 0 otherwise

where i=I n ik=I ri j=I m

The importance score method has been separately compared as a feature selection method with

a genetic algorithm-based method (Imam amp Vafaie 1994) The importance score method

produced an equal or higher accuracy on three real-world problems than those reported by the

GA method while selecting fewer attributes In addition the IS method was significantly faster

than the GA

yalue distribution The third elementary criterion value distribution concerns the number of

legal values of tests Given two tests with equal importance scores this criterion prefers the test

with the smaller number of legal values Experiments have shown that this criterion is especially

useful when using discriminant decision rules

Definition 34 A value distribution VD(Aj) of a test A j is defined by

VD(Aj) = IS(Aj) I Vj (3-5)

where v is the number of legal values of Aj

Dominance The fourth elementary criterion dominance prefers tests that appear in large

numbers of rules as this indicates their high relevance for discriminating among the rulesets of

given decision classes Since some conditions in the rules have values linked by internal

disjunction counting such rules directly would not reflect properly their relevance Therefore

36

for computing the dominance the rules are counted as if they were converted to rules that do not

have internal disjunction Such a conversion is done by multiplying out the condition parts of

the rules containing internal disjunction For example the condition part [x3=1 v 3]amp[x4=1] is

multiplied out to two rules with condition parts [x3=1]amp[x4=1] and [x3=3]amp[x4=I]

The above criteria are combined into one general test measure using the lexicographic

evaluation functional with tolerances (LEF) (Michalski 1973) LEF is a list of some or all of

the above elementary criteria each associated with a Ittolerance threshold It in percentage The

criteria are applied to tests in the order defined by LEF A test passes to the next criterion only if

it scores on the previous criterion within the range defined by the tolerance (from the top value)

The default LEF is

ltCost d Disjointness t2 Importance t3 Value distr t4 Dominance 0gt (3-6)

where d a t3 t4 and t5 are tolerance thresholds (in percentages) their default values are O

The default value of the cost of each test is 1

The above LEF ranks attributes this way First attributes are evaluated on the basis of their cost

If two or more attributes have the same cost they ranked by their degree of disjointness If two

or more attributes still share the same top score or their scores differ less than the assumed

tolerance threshold t2 the method evaluates these attributes using the second (Importance)

criterion If again two or more attributes share the same top score or their scores differ less than

the tolerance threshold t3 then the third criterion normalized IS is used then similarly the

fourth criterion (dominance) If there is still a tie the method selects the best attribute

randomly

If there is a non-uniform frequency distribution of examples of different classes then the

selection criterion uses a modified definition of the disjointness Namely the previously defmed

37

disjointness for each class is multiplied by the frequency of the class occurrence The class

occurrence is the expected number of future examples that are to be classified to a given class

m

Disjointness(A) = L D(A CO Frq(Ci) (3-7) i=l

where Frq(Cn is the expected frequency of examples of class Cit and it is assumed to be given

by the user The attribute ranking criteria in this case is defined by the LEF

ltCost d Disjointness t2 Importance 13 Normalized-IS 14 Dominance 15gt (3-8)

where the Cost denotes the evaluation cost of an attribute and is to be minimized while other

elementary criteria are treated the same way as in the default version

332 The AQDT-2 algorithm

The AQDT-2 algorithm constructs a decision structure from decision rules by recursively

selecting at each step the best test according to the ranking criteria described above and

assigning it to the new node The process stops when the algorithm creates terminal branches that

are assigned decision classes To facilitate such a process the system creates a special data

structure for each concept description (ruleset) This structure has fields such as the number of

rules the number of decision classes and the number of attributes present in the rules A set of

pointers connect this data structure to set of data structures each represents one decision class

The decision class structure contains fields with information on the number of rules belong to

that class the frequency of the decision class etc It is also connected to a set of data structures

representing the decision rules within each decision class The system creates independently a set

of data structures each is corresponding to one attribute Each attribute description contains the

attributes name domain type the number of legal values a list of the values the number of

rules that contain that attribute and values of that attribute for each rule The attributes are

arranged in an array in a lexicographic order flISt in the descending order of the number of rules

38

that contain that attribute and second in the ascending order of the number of the attributes

legal values

The system can work in two modes In the standard mode the system generates standard

decision trees in which each branch has a specific attribute-value assigned In the compact

mode the system builds a decision structure that may contain

A) or branches Le branches assigned an internal disjunction of attribute-values whenever it

leads to simpler structures For example if a node assigned attribute A has a branch marked by

values 1 v 2 then the control passes along this branch whenever A takes value 1 or 2 The

program creates or branches on the basis of the analysis of the value sets Vi while computing

the degree of attribute disjointness

B) nodes that are assigned derived attributes that is attributes that are certain logical or

mathematical combinations of the original attributes To produce decision structures with

derived attributes the input decision rules are generated by AQ17 (rather than AQI5) The

AQ17 rules may contain conditions involving attributes constructed by the program rather than

those originally given

To generate decision structures from rules the AQDT-2 method prefers either characteristic or

discriminant disjoint rule descriptions (given by an expert or learned by a system) Disjoint rules

are more suitable for building decision structures Assume that the description of each class is in

the form of a ruleset and that this set is the initial ruleset context The AQDT algorithm is

The AQDT2 Algorithm

Step 1 Evaluate each attribute occurring in the ruleset context using the LEF attribute ranking

measure Select the highest ranked attribute Let A represents this highest-ranked

attribute

Step 2 Create a node of the tree (initially the root afterwards a node attached to a branch) and

assign to it the attribute A In standard mode create as many branches from the node as

39

there are legal values of the attribute A and assign these values to the branches In

compact mode (decision structures) create as many branches as there are disjoint value

sets of this attribute in the decision rules and assign these sets to the branches

Step 3 For each branch associate with it a group of rules from the ruleset context that contain a

condition satisfied by the value(s) assigned to this branch For example if a branch is

assigned values i of attribute A then associate with it all rules containing condition [A=

i v ] If a branch is assigned values i v j then associate with it all rules containing

condition [A= i v j v ] Remove from the rules these conditions If there are rules in

the ruleset context that do not contain attribute A add these rules to all rule groups

associated with the branches stemming from the node assigned attribute A (This step is

justified by the consensus law [x=1] == [x=1] amp [y=a] v [x=1] amp [y=b] assuming

that a and b are the only legal values of y) All rules associated with the given branch

constitute a ruleset context for this branch

Step 4 If all the rules in a ruleset context for some branch belong to the same class create a leaf

node and assign to it that class If all branches of the trees have leaf nodes stop

Otherwise repeat steps 1 to 4 for each branch that has no leaf

To select an attribute to be a node in the decision tree (step 1 and 2 of the algorithm) the

algorithm performs two independent iterations In the first iteration it parses through all decision

rules and determine information about each attributes This information includes the importance

score of each attribute the number of rules containing a given attribute the disjoint value sets of

each attribute and the attribute values used in describing each decision class The second

iteration is only performed if the disjointness criterion is ranked first in the LEF function The

second iteration evaluates the attributes disjointness for each decision class against the other

decision classes

40

To determine the complexity of this process suppose that the maximum number of conditions

fonned by a single attribute value in one rule is s (where s ~ n the number of attributes) and r is

the total number of decision rules (in all decision classes)

m

r=LRt (m is the number of decision classes) 1=1

where Ri is the number of rules in decision class Ci bull The complexity of the frrst iteration can be

determined as

Cmpx(Itcl) =O(r s)

In the second iteration the disjointness is calculated between the decision classes and all

attributes The complexity of the second iteration can be given by

Cmpx(Itc2) = O(n m)

Assume that at each node 1 is the maximum of the number of decision classes to be classified at

this node and the number of rules associated with this node

l=max mr (3-9)

Since these two iteration are independent of each other the complexity of the AQDT algorithm

for building one node in the decision tree say Node Complexity NC(AQDT) is given by

NC(AQDT) = 0(1 n)

Usually I equals to the number of rules associated with the given node Thus the AQDT

complexity for building one node is a function of the number of attributes multiplied by the

number of rules associated with this node

At each level of the decision tree these two iterations are repeated for each non-leaf node The

Level Complexity of the AQDT algorithm LC(AQDT) can be given by

LC(AQDT) lt 0(1 n)

Which is less than the complexity of generating the root of the decision tree To explain this

consider the maximum possible number of non-leaf nodes at one level is half the number of the

initial decision rules (integer of rll) Figure 3-5-a shows an example of such situation where at

41

the lowest level each node classifies only two rules each belongs to different decision class This

decision tree could be a multimiddotvalues tree However the number of non-leaf nodes at each level

will be twice or more than the number of nonmiddot leaf nodes at the previous level Considering the

level complexity of the AQDT algorithm equal to (1 s 0) where 0 is the number of non-leaf

nodes at the given level In such cases it is either (1 0 S r) or (1 s lt r) In Figure 3middot5middota the

complexity of the AQDT algorithm at any lower level is given by

LC(AQDT) =0(2 s r(2) lt O(n I) =NC(AQDT)

a) per one level b) per one path

Figure 3 -5 Decision trees showing the maximum number of none leaf nodes

Note also that after selecting an attribute to be the root of the decision structure this attribute

and all conditions containing that attribute are removed from the data structure of the algorithm

Also if a leaf node is generated all rules belonging to the corresponding branch will not be

tested again

Since the disjointness criterion selects the attribute which minimizes the average number of tests

ANT it is true that the AQDT algorithm generates decision trees with the least number of levels

The number of levels per a decision tree is supposed to be less than or equal to the minimum of

both the number of attributes and the number of rules Consider k as the number of levels in a

given decision tree

k S min mr (3-10)

There are two cases represent the most complex situations Figure 3-5-a and 3-5-b In the fIrst

case where the decision rules were divided evenly the number of levels will be a function of the

42

logarithm of the number of rules In such case the complexity of the AQDT algorithm for

generating a decision tree from a set of decision rules is given by

Complexity(AQDT) = 0(1 n log r) (3-11)

The other situation is when the generated decision tree has the maximum number of levels The

maximum possible number of levels per a decision tree equals to one less than the number of

decision rules Figure 3-5-b Using the disjointness criterion it is not likely to get such a decision

tree because it has the maximum average number of test (ANT) that can be determined from the

same set of nodes and leaves However such a decision tree can be generated if the number of

decision classes is one less thanthe number of attributes In such case any disjQint decision rules

should have a maximum length that is less or equal to the floor of the logarithm of the number of

attributes Thus the level complexity of this decision tree is estimated as

LC(AQDT) =0(1 log n)

The maximum number of levels in such decision tree is k-l Thus the complexity of the AQDT

algorithm in such cases is given by

Complexity(AQDT) = 0(1 k log n) (3-12)

Since k is the minimum of n and r then n is the maximum of k and n Also since in (3-12)

r=n+1 then log r is greater than log n From 3-10 3-11 and 3-12 the complexity of the AQDT

algorithm is determined by

Cmplx(AQDT) = OCr k log 1) (3-13)

333 An example illustrating the algorithm

The following simple example illustrates how AQDT is used in selecting the optimal set of

testing resources for testing a software Suppose there are three tools for testing a software 1)

modeling (Tl) 2) checklist (T2) and 3) par_simul (T3) Also assume that there are four

different factors that affect the selection of any tool 1) the cost of using the tool (xl) 2) the

metrics that support the tool (x2) 3) the best phase for applying the tool (x3) and 4) type of tool

43

(automated semi-automated or manual) (x4) Table 3-1 shows these attributes and their possible

values

Suppose that the domain expert provided a set of rules to be used in testing a software Figure 3shy

6 shows a sample of these rules in AQ15c formats

Table 3-1 The available tools and the factors that affect the prclCes

Tl lt= [xl=2] amp [x2=2] v [xl=3] amp [x3=1 v 3] amp [x4=1] T2 lt= [xl=l v 2] amp [x2=3 v 4] v [xl=3] amp [x3=1 v 2] amp [x4=2] T3 lt= [xl=l] amp [x2=1] v [xl=4] amp [x3=2 v 3] amp [x4=3]

Figure 3-6 Decision rules for selecting the best tool for testing software

These rules can be interpreted as

Rule 1 Use the fltst tool for testing if you need average cost and the tool is

supported by the requirement metric

Rule 2 Use the fIrst tool for testing if you can afford high cost for testing either

in the requirement or the analysis phases and you need an automated tool

Rule 3 Use the second tool for testing if the cost limit is low or average and the

tool is supported by the system usage metric or the intractability metric

Rule 4 Use the second tool for testing if you can afford high cost for testing

either in the requirement or the design phases and you need a manual tool

Rule 5 Use the third tool for testing if the you are limited to low cost and the

tool is supported by the error rate metric

Rule 6 Use the third tool for testing if you can afford very-high cost for testing

either in the requirement or the system usage phases and you need a semishy

automated tool

44

Table 3-2 presents information on these rules and the disjointness values for all attributes For

each class the row marked Values lists values occurring in the ruleset for this class For

evaluating the disjointness of an attribute say A each rule in the ruleset above that does not

contain attribute A is characterized as having an additional condition [A= a vb ] where a b

are all legal values of A

The row Class disjointness specifies the class disjointness for each attribute The attribute xl

has the highest disjointness (11) and is assigned to the root of the tree For simplicity assume

the tolerances for each elementary criterion equal O

From the rules in Figure 3-6 we can also determine disjoint groupings of attribute-values used

for the compact mode of the algorithm This done as follows 1) determine for each attribute the

sets of values that the attribute takes in individual decision rules and remove those value sets

that subsume other value sets The remaining value sets are assigned to branches stemming from

the node marked by the given attribute For example x 1 has the following value sets in the

individual decision rules 2 3 I 2 I and 4 (Figure 3-6) Value set I 2 is

removed as it subsumes 2 and I In this case branches are assigned individual values of the

domain of x1 For attribute x2 the value sets are I 2 34 and I 2 3 4 In this case

branches are assigned value sets I 2 and 34

45

Attribute Xl ranks highest (as it is the highest disjointness) and is assigned to the root of the

tree Four branches are created each one corresponding to one of XIs possible values Since all

rules containing [x1=4] belong to class TI the branch marked by 4 is ended by a leafTI Rules

containing other values of X1 belong to more than one class This process is repeated for each

subset of rules until the decision tree is completed Figure 3-7 shows a decision structure learned

by AQDT-2 from the rules in Figure 3-6 The decision structure in Figure 3-7 can be used in

making decisions on which tools can be used for testing a given software

xl

Complexity No of nodes 4 No of leaves 7

Figure 3-7 A decision structure learned for classifying so tware testing tools

Figure 3-8-a shows the diagrammatic visualization of the decision rules and Figure 3-8-b shows

the visualization of the derived decision tree Each diagram in Figure 3-8 consists of cells

representing one combination of attribute-values Attributes and their legal values are shown on

scales surrounding the diagram (eg the horizontal scale for xl shows values 1 2 3 and 4)

Rules are represented by collections of cells in the intersection of the rows and columns

corresponding to the conditions in the rules

The shaded areas correspond to decision rules Rules of the same class have the same type of

shading Empty cells correspond to combination of attribute-values not assigned to any class For

illustration collections of cells corresponding to some of the initial rules are marked R 11 R 21

R31 and R32 Rll denotes the IlfSt rule of Class T1 ie [x1=2] amp [x2=2] R21 denotes the

IlfSt rule of class T2 ie [xl= 1 v 2] amp [x2= 3 v 4] R31 the first rule of class TI ie [x1=1]

46

amp [x2 =1] and denotes R32 denotes the second rule of class T3 Le [xl=4] amp [x3=2 v 3] amp

[x4=3]

a) Decision rules b) Derived decision tree

Comparing diagrams in Figure 3-8-a and 3-8-b AQDT-2 decision tree represents a slightly more

general description of Concepts Tl TI and T3 than the original rules Let us assume that it is

very costly to know which metrics suppon the required tools In other words suppose that we

would like to select the best tools independently of the metric they suppon (this is indicated to

AQDT-l by assigning very high cost to attribute x2) The algorithm again assigns attribute xl to

the root of the tree When xl takes value 1 or 2 it is impossible to assign a specific decision

without measuring attribute x2 For the value 1 of xl the recommended tool can be either TI or

T3 (see the diagram in Figure 3-9-a) and for the value 2 of xl the recommended tool can be

either Tl or T3 However for the value 3 of xl one can make a specific decision after

measuring attribute x4 For the value 1 of x4 tool is Tl and for the value 2 of x4 the

recommended tool is TI Figure 3-9-b shows another decision tree where the type of the tool

was ignored from the data Those decision trees are called indeterminate because some of their

leaves are assigned a disjunction of two or more class names

47

xl

xl

1 12

a) Ignoring the supporting metric b) Ignoring the type of the tooL Figure 3 -9 Decision trees learned ignoring the support metric and the type of the testing tool

It is clear that the given set of rules depend highly on amount of money can be spent Let us now

suppose that the cost attribute xl which was determined as the highest ranked attribute cannot

be measured The algorithm selected x4 to be the root of the new decision tree After continuing

the algorithm the tree in Figure 3-10 is obtainedThe nodes that are assigned one class indicate

situations in which it is possible to make a specific decision without knowing the value of

attribute xl

x4

Figure 3-10 A decision tree learned without the cost attribute

34 Tailoring Decision Structures to the Decision-making situation

Decision structures are among the simplest structures for organizing a decision-making process

A decision structure specifies explicitly the order in which attributes of an object or situation

need to be evaluated in the process of determining a decision A standard way to generate a

48

decision structure is to learn it from examples of decisions Such a process usually aims at

obtaining a decision structure that has the highest prediction accuracy that is the highest rate of

assigning correct decisions to given situations There can usually be a large number of logically

equivalent decision structures (Michalski 1990) As such they may have the same predictive

accuracy but differ in the way they organize the decision process and thus may differ in the cost

of arriving at a decision To minimize the average decision cost one needs to take into

consideration the distribution of the costs of attribute evaluation and the frequency of different

decisions This report presents an approach to building such Task-oriented decision structures

which advocates that they are built not from examples but rather from decision rules Decision

rules are learned from examples using the AQ15 or AQ17 inductive learning program or are

specified by an expert An efficient algorithm and a new system AQDT-2 that transforms

decision rules to task-oriented decision structures The system is illustrated by applying it to the

problem of learning decision structures in the area of construction engineering (for determining

the best wind bracing for tall buildings) In the experiments AQDT-2 outperformed all other

programs applied to the same data

Decision-making situations can vary in several respects In some situations complete

information about a data item is available (ie values of all attributes are specified) in others the

information may be incomplete To reflect such differences the user specifies a set of parameters

that control the process of creating a decision structure from decision rules AQDT-2 provides

several features for handling different decision-making problems 1) generating a decision

structure that tries to avoid unavailable or costly attributes (tests) 2) generating unknown

leaves in situations where there is insufficient information for generating a complete decision

structure 3) providing the user with the most likely decision when performing a required test is

impossible 4) providing alternative decisions with an estimate of the likelihood of their

correctness when the needed information can not be provided

49

341 Learning Costmiddot Dependent Decision Structures

As described in Sec 2 the LEF criterion enables the system to take into consideration the cost of

measuring tests (attributes) in developing a decision structure In the default setting of LEF the

tests cost is the [lIst criterion and its tolerance is O This means that only the least expensive

attributes pass to the next step of evaluation involving other elementary criteria If an attribute

has high cost or is impossible to measure (has an infinite cost) the LEF chooses another

cheaper attribute ifpossible

342 Assigning Decision Under Insufficient Information

In decision-making situations in which one or more attributes cannot be measured the system

may not be able to assign a definite decision for some cases If no more information can be

obtained but a decision has to be made it is useful to know the probability distribution for

different candidate decisions (Smyth Goodman amp Higgins 1990) The most probable decision is

then chosen The probability distribution can be estimated from the class frequency at the given

node Let us consider a node in the structure connected to the root by the sequence bl b2 bkof

attribute-values Let P(Ci I bl b2 bk) denote the conditional probability of class Ci i=I2 m

at that node given that an example to be classified has attribute-values assigned to branches bl

b2 bull bk in the decision structure Using Bayesian formula we have

P(Ci I bl~ bull bk) = P(Ci) P(bl bk I Ci) P(bl bk) (3-9)

where P(Ci) the apriori probability of class Ci P(bl bk I Ci) is the probability that the

example has attribute-values blbull bk given that it represents decision class Ci To approximate

these probabilities let us suppose that Wi is the number of training examples of Class Ci that

passed the tests leading to this node and twi - the total number of training examples of Class

Ci Let us use these numbers to estimate the probabilities in (9) Assuming that the apriori class

probabilities correspond to the frequency of training examples from different classes we have

50

m P(Ci) = twi IL twj (3-10)

j=l

P(b1 bkl Ci) = Wi I twi (3-11)

m m P(b1 bk) =L Wj IL twj (3-12)

j=l j=1

By substituting (1O) (11) and (12) in (9) we obtain

m P(Ci I b1 bk) = wilL Wj (3-13)

j=l

A related method for handling the problem of unavailability of an attribute is described by

Quinlan (1990) Quinlans method however assigns probabilities to the most probable decision

at the node associated with such an attribute The method does not restructure the decision tree

appropriately to fit the given decision-making situation (in this case to avoid measuring xl)

AQDT -2 assigns probabilities o~ly after it first created a decision structure that suits the given

decision-making si~ation An example is presented in section 42

343 Coping with noise in training data

The proposed methodology can be easily extended to handle the problem of learning from noisy

training data This is done based on ideas of rule truncation described in (Niblett amp Bratko 1986

Bergadanoet al 1992 Cestnik amp Karalic 1991) When noise is expected in the training data

the decision rules with t-weight below a certain threshold (reflecting the expected noise level) are

removed The rule truncation method seems to result in more accurate decision structures than

decision tree pruning method because truncation decisions are solely based on the importance of

the given rule or condition for the decision-making regardless of their evaluation order (unlike

the decision tree pruning which can only prunes attributes within a subtree and thus cannot

freely chose attributes to prune) Examples are presented in section 4

vii

Congressional Voting Records (1984) 87

410 Analysis of the Results 88

ClIAPfER 5 CONCIUSIONS 95

51 Summary 95

52 Contributions 96

REFERENCES 98

VITA 102

viii

LIST OF TABLES

No TITLE Page

2-1 An example of a decision table 9

2-2 A set of training examples used to illustrate the C45 system 15

2-3 The frequency of different attributes values to different decision classes 17

2-4 The expected values of the frequency of examples in Table 2-3 17

2-5 Attribute selection criteria and their basic evaluation measure 17

2-6 The contingency tables of Mingers example 18

2-7 Mingers results for determining the goodness of split 19

2-8 Mingers results for comaring the total accuracy and size of decision trees

provided by different attribute selection criteria from four problems 19

2-9 Comparing the AQDT approach with EDAG and HOODG approaches 22

3-1 The available tools and the factors that affect the process of testing a software 43

3-2 Calculating the disjointness of each attribute 44

3-3 Evaluation of the AQDT-2 criteria on the Quinlans example in Table 2-2 51

3-4 The data used in Mingers f11st experiments 52

3-5 The performance of AQDT-2 criteria (compare with the other criteria in Table 2-6) 52

3-6 The possible ranking domains and using condition of AQDT-2 criteria 53

3-7 Comparison between Decision Structures and Decision Tree 54

4-1 Evaluation of AQDT-2 attribute selection criteria for the wind bracing problem 62

4-2 The predictive accuracy of AQ15c and AQDT-2 for the wind bracing problem 67

4-3 Evaluation of AQDT-2 attribute selection criteria for the MONK-l problem 71

4-4 The predictive accuracy of AQ15c and AQDT-2 for the MONK-l problem 73

ix

4-5 The predictive accuracy of AQ15c and AQDT-2 for the MONK-2 problem 77

4-6 The predictive accuracy of AQ15c and AQDT-2 for the MONK-3 problem 81

4-7 The set of attributes and their values used in the trains problem 86

4-8 A tabular snmmary of the predictive accuracy of decision trees obtained

by AQDT -2 and C45 for the congressional voting data 88

4-9 Summary of the best parameter settings for the first subfunction of the approach

with different data characteristics 89

4-10 Summary of the performance of AQDT-2 and C45 on different problems 90

x

LIST OF FIGURES

No TITLE Page

2-1 An example to illustrate how attributes break rules 8

2-2 A decision tree learned from the decision table in Table 2-1 10

2-3 A decision tree learned using the gain criterion for selecting attributes 15

2-4 Decision rules and their Exception Directed Acyclic Graph (EDAG) 21

3-1 Architecture of the AQDT approach 24

3-2 A ruleset generated by AQ15 for the concept Voting pattern of

Democratic Representatives 27

3-3 Ven Diagrams ofPossible combinations of attribute values in two decision classes 33

3-4 Decision trees corresponding the Ven diagrams in Figure 3-3 33

3-5 Decision trees showing the maximum number of none leaf nodes 41

3-6 Decision rules for selecting the best tool for testing a software 43

3-7 A decision structure learned for classifying software testing tools 45

3-8 A diagrammatic visualization of the decision rules and the derived decision tree 46

3-9 Decision trees learned ignoring the support metric and the type of the testing tool 47

3-10 A decision tree learned without the cost attribute 47

3-11 Decision structures learned by AQDT-2 using different criteria 55

3-12 The Imams example Example where learning decision structures (trees)

from rules is better than learning them from examples 56

3-13 An example where decision rules are simpler than decision trees 57

4-1 Design of a complete experiment 59

Xl

4-2 Decision rules determined by AQ15c from the wind bracing data 61

4-3 A decision tree learned by C45 for the wind bracing data 63

4-4 A decision structure learned from AQ15c wind bracing rules 64

4-5 A decision structure that does not contain attribute xl 64

4-6 A decision structure without Xl with candidate decisions assigned to leaves 65

4-7 A decision structure determined from rules in Figure 4-4 under the

assumption of 10 classification error in the training data 65

4-8 Diagramatic visualization of decision trees learned for different decision

making situations for the wind bracing data 66

4-9 The accuracy of AQDT-2 and AQ15c with different parameter settings

for the wind bracing PIoblem 68

4-10 Analyzing different parameter setting of AQDT-2 using the wind bracing data 69

4-16 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-20 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-11 A comparison between AQ15c and AQDT-2 against C45 on the wind bracing data 69

4-12 A visualization diagram of the MONK-l problem 70

4-13 Decision rules learned by AQ15c for the MONK-1 problem 71

4-14 The decision tree for the MONK-1 problem generated both by AQDT-l 72

4-15 Compact decision structures generated by AQDT-2 for the MONK-1 problem 72

for the MONK-l Problem 74

4-17 Analyzing different parameter setting of AQDT-2 with the MONK-l data 75

4-18 Comparing AQ15c and AQDT-2 against C45 using the MONK-1 problem 75

4-19 A visualization diagram of the MONK-2 problem 76

for the MONK-2 Problem 78

4-21 Analyzing different parameter setting of AQDT-2 with the MONK-2 data 79

xii

4-22 Comparing AQ15c andAQDT-2 against C45 using the MONK-2 problem 79

4-24 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-35 A visualization diagram of a decision tree learned by AQDT -2 after reducing

4-23 A visualization diagram of the MONK-3 problem 80

for the MONK-3 Problem 82

4-25 Analyzing different parameter setting of AQDT-2 with the MONK-3 data 82

4-26 Comparing AQ15c and AQDT-2 against C45 using the MONK-3 problem 83

4-27 Comparing AQ15c and AQDT-2 against C45 using the breast cancer problem 84

4-28 Comparing AQ15c and AQDT-2 against C45 using the mushroom problem 85

4-29 Decision structures learned by AQDT-2 for different decision-making situations 87

4-30 Comparing decision trees for the Cong Voting-84 data learned by C45 amp AQDT -2 88

4-31 A visualization diagram of decision rules learned by AQ15c for the MONK-2 91

4-32 A visualization diagram shows the testing errors for the rules in Figure 4-31 92

4-33 A visualization diagram of the decision tree learned by AQDT-2 for the MONK-2 93

4-34 A visualization diagram shows testing errors of AQDT-2 decision tree bull 94

the generalization degree to 1 94

DERIVING TASKmiddotORIENTED DECISION STRUCTURES

FROM DECISION RULES

Ibrahim M Fahmi Imam PhD

School of Information Technology and Engineering

Geotge Mason University Fall 1995

Ryszard S Michalski Advisor

ABSTRACT

This dissenation is concerned with research on learning task-oriented decision structures from

decision rules The philosophy behind this research is that it is more appropriate to learn

knowledge and store it in a declarative form and then when a decision making situation occurs

generate from this knowledge the decision structure that is most suitable for the given decision

making situation Learning decision structures from decision rules was first introduced by

Michalski (1978) The flIst implementation of this approach was done by Imam and Michalski

(1993 ab) called AQDT-l

This approach separates the function of generating a knowledge-base from the function of using

the knowledge-base for decision-making The first function focuses on learning accurate

consistent and complete concept description expressed in a declarative form The second function

is performed whenever a new decision-making situation occurrs a task-oriented decision structure

is obtained to suit that situation Thsk-oriented knowledge is defined as knowledge that is adapted

for solving a given decision-making situation (Imam amp Michalski 1994 Michalski amp Imam

1994)

The dissertation introduces the system AQJJT-2 for learning task -oriented decision structures from

decision rules or examples Each decision making situation is defined by a set of parameters that

controls the learning process of the AQDT-2 system The extensive experiments on AQJJT-2 show

that decision structures learned by it usually outperform in terms of accuracy and average size of

the decision structures those learned from examples by other well known systems The results

show also that the system does not work very well with noisy data The system is illustrated and

compared using applications of artificial problems such as the three MONKs problems (Thrun

Mitchell amp Cheng 1991) and the East-West 1iain problem (Michie et al 1994) It was also

applied to real world problems of learning decision structures in the areas of construction

engineering (for determining the best wind bracing design for tall buildings) medical

diagnosis(for learning decision rules for recognizing breast cancer) agriculture diagnosis (for

learning classification rules for distinguishing between poisonous and non-poisonous

mushrooms) and political data (for characterizing democratic and republican voting records)

CHAPTER 1 INTRODUCTION

11 Motivation and Overview

Learning and discovery systems should be able not only to generate and store knowledge but also

to use this knowledge for decision-making The main step in the development of systems for

decision-making is the creation of a knowledge structure that characterizes the decision-making

process The form in which knowledge can be easily obtained may however differ from the form

in which it is most readily used for decision-making It is therefore important to identify the form

of knowledge representation that is most appropIiate for learning (eg due to ease of its

modification) and the form that is most convenient for decision making

A simple and effective tool for describing decision processes is a decision structure which is a

directed acyclic graph that specifies an order of tests to be appliedto an object (or a situation) to

arrive at a decision about that object The nodes of the structure are assigned individual tests

(which may correspond to a single attribute a function of attributes or a relation) the branches are

assigned possible test outcomes or ranges of outcomes and the leaves are assigned a specific

decision a set of candidate decisions with corresponding probabilities or an undetermined

decision A decision structure reduces to a familiar decision tree when each node is assigned a

single attribute and has at most one parent when the branches from each node are assigned single

values of that attribute and when leaves are assigned single definite decisions Thus the problem

of generating a decision structure is a generalization of the problem of generating a decision tree

Decision trees are typically generated from a set of examples of decisions The essential

characteristic of any such method is the attribute selection criterion used for choosing attributes to

be assigned to the nodes of the decision tree being built Such criteria include the entropy

reduction the gain and the gain ratio (Quinlan 1979 83 86) the gini index ofdiversity (Breiman

et al 1984) and others (Cestnik amp Bratko 1991 Cestnik amp Karalic 1991 Mingers 1989a)

3

4

Adecision treedecision structure representations can be an effective tool for describing a decision

process as long as all the required tests can be performed and the decision-making situations it

was designed for remain constant (eg in a doctor-patient example the doctor should detennine

that answers for all symptoms appear in the decision tree) Problems arise when these assumptions

do not hold For example in some situations measuring certain attributes may be difficult or costly

(eg in the doctor-patient example a brain or blood test is needed which is very expensive or the

tools needed are not available) In such situations it is desirable to refonnulate the decision

structure so that the inexpensive attributes are evaluated frrst (assigned to the nodes close to the

root) and the expensive attributes are evaluated only if necessary (by assignment to the nodes far

away from the root) If an attribute cannot be measured at all it is useful to either modify the

structure so that it does not contain that attribute or-when this is impossible-to indicate

alternative candidate decisions and their probabilities A restructuring is also desirable if there is a

significant change in the frequency of occurrence of different decisions (eg in the doctor-patient

example the doctor may request a decision structure expressed in a specific set of symptoms

biased to classify one or more diseases or specify a certain order of testing)

Arestructuring ofa decision structure (or a tree) in order to suit new requirements could be quite

difficult This is because a decision structure is a procedural representation that imposes an

evaluation order on the tests In contrast no evaluation order is imposed by a declarative

representation such as a set of decision rules Tests (conditions) of rules can be evaluated in any

order Thus for a given set of rules one can usually build a huge number of logically equivalent

decision structures (trees) which differ in the test ordering Due to the lack of order constraints

a declarative representation (rules) is much easier to modify to adapt to different situations than a

procedural one (a decision structure or a tree) On the other hand to apply decision rules to make a

decision one needs to decide in which order tests are evaluated and thus needs a decision

structure

An attractive solution to these opposite requirements is to acquire and store knowledge in a

declarative form and transform it to a decision structure when it is needed for decision-making

5

This method allows one to create a decision structure that is most appropriate in a given decisionshy

making situation Because the number of decision rules per decision class is usually small (each

rule is a generalization of a set of examples) generating a decision structure from decision rules

can potentially be perfonned much faster than by generation from training examples Thus this

process could be done on line without any delay noticeable to the user Such virtual decision

structures are easy to tailor to any given decision-making situation

This approach allows one to generate a decision structure that avoids or delays evaluating an

attribute that is difficult to measure in some decision-making situation or that fits well a particular

frequency distribution of decision classes In other situations it may be unnecessary to generate

complete decision structure but it may be sufficient to generate only a part of it which concerns

only with decision classes of interest Thus such an approach has many potential advantages

This dissertation presents a new system called AQflf-2 The AWf-2 system generates a taskshy

oriented decision structure (decision structure that is adapted to the given decision-making

situation) from decision rules The decision rules are learned by either rule leaming system AQ15

(Michalski et al 1986) or system AQ17-DCI which has extensive constructive induction

capabilities (Bloedorn et al 1993)

To associate the decision rules with a given decision-making task AQflf-2 provides a set of

features including 1) enabling the system to include in the decision structure nodes corresponding

to new attributes constructed during the process of learning the decision rules 2) controlling the

degree of generalization needed during the development of the decision structure 3) providing four

new criteria for selecting an attribute to be a node in the decision structure that allow the system to

generate many different but equivalent decision structures from the same set of rules 4) generating

unknown nodes in situations when there is insufficient infonnation for generating a complete

decision structure 5) learning decision structures from discriminant rules as well as

characteristic rules and 6) providing the most likely decision when the decision process stops

due to inability to evaluate an attribute associated with an intennediate node

6

To test the methooology of generating decision structures from decision rules an extensive set of

planned experiments have been designed to test different aspects of the approach The experiments

include testing different combinations of parameters for each sub-function of the approach

analyzing the relatioship between decision rules and the decision structure learned from them and

comparing decision trees learned by the AQUf2 system the well known C45 (Quinlan 1993)

system for learning decision trees from examples Different experiments were designed to examine

the new features of the AQUf approach for learning task-oriented decision structures The

experiments were applied to artificial domains as well as real-world domains including MONK-I

MONK-2 and MONK-3 (Thrun Mitchell amp Cheng 1991) East-West trains (Michie et al

1994) Engineering Design-wind bracings (Arciszewski et al 1992) Mushrooms Breast

Cancer (Mangasarian amp WolbeJg 1990) and Congressional bting Records of 1984 The

MONKs problems are concerned with learning classification rules for robot-like figures MONK-l

requires learning a DNF-type description MONK-2 requires learning a non-DNF-type description

(one that cannot be easily described as a DNF rule using the original attributes) MONK-3 concerns

learning a DNF rule from noisy data The East-West trains dataset is a structural domain that

classifies two sets of trains (Eastbound and Westbound) The Engineering Design-wind

bracing data involves learning conditions for applying different types of wind bracing for ta11

buildings The Mushrooms data is concerned with learning classification rules for distinguishing

between poisonous and non-poisonous mushrooms The Breast Cancer data involves learning

concept descriptions for recognizing breast cancer The congressional voting data includes voting

records on different issues AQUf2 outperformed C45 on the average with respect to both the

predictive accuracy and tree size for most problems Apf-2 did not work very well with noisy

data or with problems that have many rules covering very few examples

12 The Problem Statement

There are many limitations and problems accompanied using decision trees for decision-making

CHAPTER 2 RELATED RESEARCH

21 Learning Decision Trees from Decision Diagrams

The AQDT-2 method proposed here is based on the earlier work by Michalski (1978) which

introduced an algorithm for generating decision trees from decision lists The method proposed

several attribute selection criteria These criteria are of increasing power of the main criterion

order cost estimate (the nth order cost estimates n=l 2 ) Michalski also analyzed two

specific criteria MAL and DMAL for selecting the optimal attribute for a node in the tree

based on properties extracted from the decision diagram In order to better explain the method

it is necessary to define some terms These terms will be used later in the dissertation

Definition 2-1 A ~ of a decision class C is a set of rules whose union includes the set of all

examples of class C and does not include any examples of other decision classes

Definition 2-2 A ~ of a decision class C is disjoint if all rules (rules) are pairwise logically

disjoint In other words for any two rules there exists a condition with the same attribute but

with different values in each rule

Definition 2-3 An minimal cover is a disjoint cover which has the smallest number of rules

among all possible covers

Definition 2-4 A diagram for a given cover is a table constructed graphically by representing

in two dimension space of all possible combination of attributes values and locating on the

diagram all the condition parts of the given rules and marking them with the action specified

with each rule

Michalskis algorithm requires construction of minimalmiddot cover The minimal cover should be

consistent and complete The method is based on the fact that if there are n decision classes

any decision tree that correctly classifies the given rules and has n distinct leaves is a minimal

7

8

decision tree (any consistent decision tree should have at least n leaves) Michalski (1978) has

shown that if only one rule is broken by a selected attribute then instead of having one leaf

(which could potentially represent this rule or the decision class in the tree) there will have to

be at least two leaves representing this rule in the fmal decision tree

The attribute selection criterion MAL introduced in (Michalski 1978) prefers attributes that do

not break any rules or break as few as possible An attribute breaks a rule if the attribute can

divide the rule into two or more sub-rules Figure 2-1 shows two examples of two sets of rules

In the fJIst xl and x3 break at least two rules each (xl breaks the rule [x4=2] amp [x1=lv3] amp

[x3=2v3] and the rule [x4=3] amp [x1=1v2] amp [x3=1] and x3 breaks three rules the rule [x4=2]

amp [xl=2] amp [x3=2v3]the rule [x4=l] amp [xl=3] amp [x3=lv3] and the rule [x4=3] amp [xl=4] amp

[x3=2v3]) In the diagram on the right xl is the only attribute that does not break any rule

Figure 2-1 An example to illustrate how attributes break rules

One of the criteria defmed by Michalski is the first degree cost estimate which assigns to each

attribute an integer equal to the number of rules broken by that attribute This criterion is also

called the static cost estimate of an attribute or the criterion of minimizing added leaves

(MAL)

-

C3

9

The MAL criterion (Minimizing Added Leaves) seeks an attribute that minimizes the

estimated number of additional nodes in the decision tree being generated over a hypothetical

minimal decision tree When there is a tie between two attributes the attribute to be selected is

the one which breaks smaller rules (rules that cover fewer examples or more specialized

rules) AQDT-2 uses an approximate version of this criterion (the attribute dominance)

Another criterion introduced by Michalski was the DMAL criterion The DMAL criterion

(Dynamically Minimizing Added Leaves) is based on a principle similar to that of MAL but

is more complex because once an attribute is selected as a node in the tree some rules andor

parts of the broken rules at each branch are merged into one rule The DMAL ensures that the

value of the total cost estimate of an attribute is decreased by a value equal to the number of

merged rules minus one

Example Learn a decision tree from the following decision table

The minimal cover consists of the following rules

Al lt [x2=O] v [x1=0][x2=2] A2 lt [x2=I] v [xl=2][x2=2] A3 lt [xl=I][x2=2]

The evaluations ofthe MAL criterion for these attributes are 2 for xl 0 for x2 5 for x3 and 5

for x4 The attribute xl divides the two rules [x2=O] and [x2=I] so selecting it will add two

leaves to the optimal number of leaves It is clear that the attribute to be selected as a root of

10

the decision tree is x2 Then three branches are attached to the root node and the decision rules

are divided into subsets each corresponding to one branch For x2 = 0 or 1 a leave node is

generated For x2=2 another attribute is selected to be a node in the tree In this case xl has the

minimum MAL value Figure 2-2 shows the decision tree obtained using the MAL criterion

Al A3 A2

Figure 2-2 A decision tree learned from the decision table in Table 2-1

22 Learning Decision Trees from Examples

Decision tree learning is a field that concerned with generating decision tree that classifies a set

of examples according to the decision classes they belong to The essential aspect of any

inductive decision tree method is the attribute selection criterion The attribute selection

criterion measures how good the attributes are for discriminating among the given set of

decision classes The best attribute according to the selection criterion is chosen to be assigned

to a node in the tree The fIrst algorithm for generating decision trees from examples was

proposed by Hunt Marin and Stone (1966) Hunts algorithm uses a divide and conquer

algorithm for building decision trees This algorithm has been subsequently modified by

Quinlan (1979) and applied by many researchers to a variety of learning problems

Attribute selection criteria can be divided into three categories These categories are logicshy

based information-based and statistics-based The logic-based criteria for selecting attributes

use logical relationships between the attributes and the decision classes to determine the best

attribute to be a node in the decision tree such as the MAL criterion minimizing added leaves

(Michalski 1978) which uses conjunction and disjunction operators The information-based

criteria are based on the information theory These criteria measure the information conveyed

11

by dividing the training examples into subsets Examples of such criteria include the

information measure 1M the entropy reduction measure and the gain criteria (Quinlan 1979

83) the gini index of diversity (Breiman et al 1984) Gain-ratio measure (Quinlan 1986) and

others (Clark amp Niblett 1987 Bratko amp Lavrac 1987 Cestnik amp Karalic 1991) The

statistics-based criteria measure the correlation between the decision classes and the attributes

These criteria use statistical distributions for determining whether or not there is a correlation

The attribute with the highest correlation is selected to be a node in the tree Examples of

statistics-based criteria include Chi-square and G-statistic (Sokal amp Rohlf 1981 Han 1984

Mingers 1989a)

Niblett and Bratko (1986) Quinlan (1987) and Bratko and Kononeko (1987) extended the

method of learning decision trees to also handle data with noise (by pruning) Handling noise

extended the process of learning decision trees to include the creation of an initial complete

decision tree tree pruning which is done by removing subtrees with small statistical Validity

and replacing them by leaf nodes (MiDgers 1989b) More recently pruning has also been used

for simplifying decision trees even for problems without noise (Bohanec amp Bratko 1994)

Pruning decision trees improves their simplicity but reduces their predictive accuracy on the

training examples Quinlan (1990) proposed also a method to handle the unknown attributeshy

value problem by exploring probabilities of an example belonging to different classes

The rest of this section includes a brief description of the attribute selection criterion used by

C45 learning system (Quinlan 1993) C45 uses an information-based criterion for selecting

an attribute to be a node in the tree Also the section includes a brief description of the Chishy

square method for attribute selection (Mingers 1989a) The method is a statistics-based

method for selecting an attribute to be a node in the tree

221 Building Decision Trees Using htformation-based Criteria

12

This section presents a description of the inductive decision tree learning system C4S The

C4S learning system is considered to be one of the most stable accurate and fastest program

for learning decision trees from examples

Learning decision trees from examples requires a collection of examples Each example is

represented by a fixed number of attribute-value pairs C45 (Quinlan 1993) is a learning

program that induces classification decision trees from a set of given examples The C4S

learning system is descended from the learning system ID3 (Quinlan 1979) which is based on

Hunts method for constructing decision tree from a set ofcases (Hunt Marin amp Stone 1966)

The C45 system uses an attribute selection criterion called the Gain Ratio This criterion

calculates the gain in classifying information based on the residual information needed to

classify cases in a set of training examples and the information yielded by the test based on the

relative frequencies of the possible outcomes (decision classes) The gain ratio criterion is

based on an earlier criterion used by ID3 called the Gain Criterion The Gain Criterion uses

the frequency of each decision class in the given set of training examples

Once an attribute is chosen to be a node in the tree the system generates as many links as the

number of its values and classifies the set of examples based on these values If all the

examples at a certain node belong to one decision class the system generates a leaf node and

assigns it to that class Otherwise the system searches for another attribute to be a node in the

tree

The Gain Criterion The gain criterion is based on the information theory That is the

information conveyed by a message depends on its probability and can be measured in bits as

minus the logarithm (base 2) of that probability To explain the gain criterion suppose that for

a given problem Xu bullbull X are given attributes and Cu c are decision classes Suppose S is

any set of cases and T is the initial set of training cases The frequency of class C1 in the set S

is the number of examples in S that belong to class C i bull

13

freq(Cit S) = Number of examples in S belong to CI (2-1)

Suppose that lSI is the total number of examples in S the probability that an example selected

at random from S belongs to class Ci is freq(CIbullS) IISI

The information conveyed by the message that a selected example belongs to a given decision

class Cit is determined by -log1 (freq(Ch S) IISI) bits

The expected information from such a message stating class membership is given by

info(S) = -k

L (freq(C S) I lSI) log1 (freq(C S) IISI) bits (2-2)I

info(S) is known also as the entrQpy of the set S When S is the initial set of training examples

info(T) determines the average amount of information needed to identify the class of an

example in T

Suppose that we selected an attribute X to be the root of the tree and suppose that X has k

possible values The training set T will be divided into k subsets each corresponding to one of

Xs values The expected information of selecting X to partition the training set T infoz(T) can

be found as the sum over all subsets of multiplying the information conveyed by each subset

by its probability

k

infoz(T) =L (ITII 1m) infO(TIraquo (2-3) I

The information gained by partitioning the training examples T into subset using the attribute

X is given by

gain(X) =inlo(T) - inoIT) (2-4)

The attribute to be selected is the attribute with maximum gain value

The Gain Ratio Criterion This criterion indicates the proportion of information generated by

the split that appears helpful for classification Quinlan (1993) pointed out that the gain

criterion has a serious deficiency Basically it is strongly biased toward attributes with many

outcomes (values) For example for any data that contains attributes such as social security

14

number the gain criterion will select that attribute to be the root of the decision tree However

selecting such attributes increases the size of the decision tree Quinlan provided a solution to

this problem by introducing the gain ratio criterion which takes the ratio of the information that

is gained by partitioning the initial set of examples T by the attribute X to the potential

information generated by dividing T into n subsets

Following similar steps to obtain the information conveyed by dividing T into n subsets The

expected information generated by dividing T into n subsets and by analogy to equation 2-2 is

determined by

split info(T) = - Lt

(ITII lTI) 10gl (ITII m ) (2-5) I

The gain ratio is given by

gain ratio(X) = gain(X) I split info(X) (2-6)

and it expresses the proportion of information generated by the split that is useful for

classification

Example Consider the following example presented by Quinlan (1993) Table 2-2 shows the

set of training examples

First determine the amount of information gained by selecting the attribute outlook to be a

root of the decision tree This attribute divides the training examples into three subsets

sunny-with five examples two of which belong to the class Play overcast-with four

examples all of which belong to the class Play and rain-with five examples three of

which belong to the class Play To determine info(11 the average information needed to

identify the class of an example in T there are 14 training examples and two decision classes

Nine of these examples belong to the class Play and five belong to the class Dont Play

Info(11 = - 914 10gl (914) - 514 10gl (514) = 094 bits

When using outlook to divide the training examples the information becomes

15

infO(T) = 514 -25 log2 (25) - 35 10g2 (35)

+ 414 -44 10g2 (44) - 04 log2 (04)

+ 514 -35 logl (35) - 25 logl (25) = 0694 bits

By substituting in equation 2-4 the gain of information results from using the attribute

outlook to split the training examples equal to 0246 The gain information for windy is

0048

Figure 2-3 shows a decision tree learned for this problem using the gain criterion The split

information for outlook is determined as follows

split info(T) = - 514 logl (514) - 414 log2 (414) - 514 log2 (514) = 1577 bits

The gain ratio for outlook = 02461577 =0156

16

Figure 2 -3 A decision tree learned using the gain criterion for selecting attributes

The C45 system handles discrete values as well as continuous values To handle an attribute

with continuous values C45 uses a threshold to transform the continuous domain into two

intervals In other words for each continuous attribute C45 generates two branches one

where the values of that attribute is greater than the determined threshold and the other if the

value is less than or equal to the threshold

Tree pruning in C45 is a process of replacing subtrees with small classification validity by

leaves The C45 system uses Laplace ratio for determining the error rate of different subtrees

This ratio is defined as (e+l) (n+2) where n is the number of the training examples and e is

the number of misclassified examples at a given leaf

222 Building Decision Trees Using Statistics-based Criteria

The Chi-square method for selecting attributes was used by Hart (1984) and Mingers (1989a)

in building decisionmiddot trees The method uses Chi-square statistics to measure the association

between two attributes When building decision trees the method is implemented such that it

determines the association between each attribute and the decision classes The attribute to be

selected is the one with greatest value

To determine the Chi-square value for an attribute consider aij is the number of examples in

class number i where the attribute A takes value numberj In other words aij is the frequency

of the combination of decision class number i and the attribute value number j The Chi-square

value for attribute A is given by

n m

Chi-square (A) =L L [ (aij - Eijl2 Eij ] (2shy

i=1 j=1

where n is the number of decision classes m is the number ofvalues of a given attribute Also

17

Eij = (TCi TVj ) T (2shy

8)

where T Ci and T Vj are the total number of examples belong to the decision class a and the total

number of examples where the attribute A takes value vj respectively T is the total number of

examples

Consider Quinlans example in Table 2-2 Table 2-3 shows the frequency of different

combination of values between the decision class and both the outlook and the Windy

attributes Table 2-4 shows the expected values TCi and T Vj of the frequencies in Table 2-3

of different attributes values for different decision classes

To determine the association value between the decision classes and both the attribute Windy

and the attribute Outlook the observed Chi-square values are

Chi-square (Windy Class) =[(3-39)239] + [(3-21)221] + [(6-51)232] + [(2-29)232]

=021 + 039 + 025 + 025 = 11

Chi-square (Outlook Class) = [(2-32)232] + [(4-26)226] + [(3-32)232] + [(3-18118]

+ [(0-14)214] + [(2-18)2 18] =045 + 075 + 02 + 08 + 14 + 002 = 362

18

Applying the same method to the other attributes the results will favor the attribute Outlook

Once that attribute is selected to be a node in the tree the remaining set of examples are divided

into subsets and the same process is repeated on each subset

Table 2-5 shows a summary of these criteria and their basic evaluation function

Table 2-5 Attribute selection criteria and their basic evaluation measure

Info Measure (IM) Gain Gshystatistics and Gain Ratio Chi-square

II

Entropy(S) = - L (freq(~ 5) 1151) logz (freq(CIt 5) 1151 ) I

G-Statistics =2N 1M (N-No of examples) n m

Chi-square (A B) = L L [(aij - ~j)21Eij ] i=l

223 Analysis of Attribute Selection Criteria

This subsection introduces briefly the analysis of different selection criteria which was done by

Mingers (1989a) Mingers compared six attribute selection criteria that are used in decision

tree programs These criteria are the Information Measure (IM) Chi-square G statistics Gin

index of diversity Marshal correction and Gain RaJio The overall results show that the Gain

Ratio criterion had the strongest xesults

In the flISt experiment Minger tested the six criteria on a problem with ambiguous examples

(ie examples may belong to more than one decision class) to observe how the selected criteria

evaluate the given attributes The problem has two decision classes and two attributes X and

Y It was assumed that attribute X is better for classifying the examples than attribute Y The

training examples were unevenly spread between the two values of X Attribute Y has three

values and the examples were spread randomly among them Table 2-6 (a and b) shows the

contingency tables for both attributes Table 2-7 shows a summary of the goodness of split

provided by the six criteria Mingers noted that the measures that are not based on information

19

theory give radiation (attribute X here) less weighL This may be because the zero in the fIrst

row of radiation has a greater influence in the log calculation In the case of using the Chishy

square criterion the value zero adds the maximum association between any two attributes

because the Chi-square value of a zero cell is the expected value of this cell

b)

Now let us demonstrate results from another experiment done by Mingers In this experiment

Mingers used four different data sets to generate decision trees for eleven different criteria In

the final results he compared the total number of nodes and the total error rate provided by

each criterion over all given problems Table 2-8 shows the final results for five selected

criteria only

20

U lgt ~middot results for comparing the total accuracy and size of decision trees different attribute selection criteria from four VVLU

This experiment was performed on four real world data sets These data are concerned with

profiles of BA Business Studies degree students recurrence of breast cancer classifying types

of Iris and recognizing LCD display digits The data was divided randomly 70 for training

and 30 for testing For more details see Mingers (1989a)

23 Learning Decision Structures

Considering the proposed defInition of decision structures given above two related lines of

research are described in this section Gaines (1994) and Kohavi (1994) proposed two

approaches for generating decision structures that share some of the earlier ideas by Imam and

Michalski (1993b)

In the first approach Brian Gaines introduces a method for transforming decision rules or

decision trees into exceptional decision structures The method builds an Exception Directed

Acyclic Graph (EDAG) from a set of rules or decision trees The method starts by assigning

either a rule say RO or a conclusion say CO or both to the root node If it assigns a conclusion

to the root node it places it on a temporary conclusion list Then it generates a new child node

and add to it a new rule say Rl The method evaluates the new rule Rl if it satisfies the

conclusion CO in its parent node it builds a new child and repeats the processes Otherwise if

the rule Rl does not satisfy the conclusion CO it changes the temporary memory with the new

conclusion Cl which is satisfied by the rule RO The same process is repeated until all rules

that have common conditions with the rule at the root are evaluated The method then creates a

21

new child node from the root and repeat the process until all rules are evaluated In the decision

structure nodes containing rules only represent common conditions to all of its children

The main disadvantages of this approach is that it requires discriminant rules to build such a

decision structure Also such a structure is more complex than the traditional decision trees

that are used for decision-making

Figure 2-4 shows an example of set of rules and their equivalent exceptional decision structure

The decision structure can be read as follows It is Safe except if xl=1 amp x2=1 amp x3=1 and

either x4=3 or x5=I then it is Lost except if x6=1 it is Safe except if x7=1 it is Lost

The second approach introduced by Ronny Kohavi learns decision structures from examples

using a bottom-up algorithm The method uses a greedy Hill-climbing algorithm for inducing

Oblivious read-Once Decision Graphs (HOODO) A read-once decision graph is defined as a

decision graph where each attribute occurs at most once along any computational path In other

words for each path from the root to the leaves of the decision structure attributes may occur

as a node once at most However there may be more than one node with the same attribute in

the decision graph Kohavi also defined a leveled decision graph in which the nodes are

partitioned into a sequence of pairwise disjoint sets such that outgoing edges from each level

terminate at the next level An oblivious decision graph is a decision graph where all nodes at

a given level are labeled by the same attribute

22

Safe lt [xl=2] Safe lt [x2=2] Safe lt [x3=2] Safe lt [x4=1] amp [x5=2] Safe lt [x4=1] amp [x5=3] Safe lt [x6=l] amp [x7=2] Safe lt [x6=1] amp [x7=3] Safe lt [x4=2] amp [x5=2] Safe lt [x4=2] amp [x5=3]

Lost lt [x6=2] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x6=3] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x5=1] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=2] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=3] amp [x3=1] amp [x2=I] amp [xl=l]

The algorithm starts by generating a leaf node for each decision class Then it applies a

nondeterministic method to select an attribute to build new level of the decision structure This

attribute is removed from the data and the data is divided into subsets each corresponding to a

complex combination of that attributes values For each subset the process is repeated until all

the examples of a given subset belong to one decision class For example if the selected

attribute say A has two values (0 and 1) and there are two decision classes (CO and CI) The

data is divided into two subsets the fIrst subset contains examples where A takes value 0 and

belong to class CO or takes value 1 and belong to class Cl The second subset of examples is

the set where A takes value 0 and belong to class CI or takes value 1 and belong to class CO

The number of nodes of the fIrst level (after the leave nodes) is expected to be less than or

equal to ka where k is the number of decision classes and n is the number of values of the

selected attribute and it can increase exponentially before that number is reduced

exponentially to one

It is easy for the reader to fIgure out some major disadvantages of such approach including

The average size of such decision structures is estimated to be very large especially when there

23

is no similarity (ie strong patterns) or logical relationship in the data The time used to learn

such decision structure is relatively very high comparing to systems for learning decision trees

from examples and fmally it could be better to search for attribute which reduces the number

of generated subsets of the data instead of nondeterministically selecting an attribute to build a

new level of the decision structure Kohavi provided a comparison between the C45 system

and his system that can be found in (Kohavi 1994) Later Kohavi and Li (1995) used a

deterministic method for attribute selection which minimizes the width of the penultimate level

of the graph

Table 2-9 shows a comparison between the proposed approach and those two approaches The

EDAG and HOODG systems are unreleased prototype systems

or Decision structures are easy to Decision structures are Decision structures are easy to understand difficult to read understand

CHAPTER 3 DESCRIPTION OF THE APPROACH

31 General Methodology

In the proposed approach the function of learning or discovery is separated from the function of

using the discovered knowledge for decision-making The frrst function is performed by an

inductive learning program that searches for knowledge relevant to a given class of decisions

and stores the learned knowledge in the form of decision rules The second function is performed

when there is a need for assigning a decision to new data points in the database (eg bull a

classification decision) by a program that transforms obtained knowledge into a decision

structure optimized according to the given decision-making situation

The Learning Task GivenA set of training examples describing the concept to be learned Learning goal which specifies the decision classes to be learned from the training examples A background knowledge to control the learning process Determine A concept description in a declarative form of knowledge (decision rules) that satisfies the learning goal

The Decision-making Task GivenA set of decision rules in a conjunctive form A description of the new decision-making situation (eg attributes costs and order preference importance or frequency of decision classes etc) One or more examples that needed to be tested under the given decisionshymaking situation A set of parameters to control the learning process Determine A decision structure that suits the given decision-making situation

The decision rules used here are learned by either the AQ1S (Michalski et al 1986) or AQ17

(Bloedorn et aI 1993) learning systems The reasons for selecting decision rules as a form of

23

24

declarative knowledge are that they do not impose any order on the evaluation of the attributes

and due to the lack of order constraints decision rules can be evaluated in many different

ways which increases the flexibility of adapting them to the different tasks of decision-making

(Bergadano et al 1990) Since the number of rules learned per class is typically much smaller

than the number of examples per class generating a decision structure from decision rules can

potentially be done on line

Such virtual decision structures can be tailored to any given decision-making situation The

needed decision rules have to be generated only once and then they can be used many times for

generating decision structures according to changing requirements of decision-making tasks The

method uses the AQDT-2 system (Imam amp Michalski 1994) for learning decision structures

from decision rules Decision structures represent a procedural form of knowledge which makes

them easy to implement but also harder to change Consequently decision structures can be quite

effective and useful as long as they are used in decision-making situations for which they are

optimized and the attributes specified by the decision structure can be measured without much

cost Figure 3-1 shows an architecture of the proposed methodology

Data

Decision -

- Learning knowledge from database The decision making process

Figure 3 -1 Architecture of the AQDT approach

25

It is assumed that the database is not static but is regularly updated A decision-making problem

arises when there is a case or a set of cases to which the system has to assign a decision based on

the knowledge discovered Each decision-making situation is defmed by a set of attribute-values

Some attribute-values may be missing or unknown A new decision structure is obtained such

that it suits the given decision-making problem The learned decision structure associates the

new set of cases with the proper decisions

32 A Brief Description of the AQIS and AQ17 Rule Learning Programs

The decision rules are generated from examples by an AQ-type inductive learning system

specifically by AQ15 (Michalski et albull 1986) or AQI7-DCI (Bloedorn at al 1993) The rest of

this subsection includes a brief description of the inductive learning systems AQ15 and AQI7

AQ15 learns decision rules for a given set of decision classes from examples of decisions using

the STAR methodology (Michalski 1983) The simplest algorithm based on this methodology

called AQ starts with a seed example of a given decision class and generates a set of the

most general conjunctive descriptions of the seed (alternative decision rules for the seed

example) Such a set is called the star of the seed example The algorithm selects from the star

a description that optimizes a criterion reflecting the needs of the problem domain If the

criterion is not defined the program uses a default criterion that selects the description that

covers the largest number of positive examples (to minimize the total number of rules needed)

and with the second priority that involves the smallest number of attributes (to minimize the

number of attributes needed for arriving at a decision)

If the selected description does not cover all examples of a given decision class a new seed is

selected from uncovered examples and the process continues until a complete class description

is generated The algorithm can work with few examples or with many examples and can

optimize the description according to a variety of easily-modifiable hypothesis quality criteria

26

The learned descriptions are represented in the form of a set of decision rules expressed in an

attributional logic calculus called variable-valued logic 1 or VLl (Michalski 1973) A

distinctive feature of this representation is that it employs in addition to standard logic operators

the internal disjunction operator (a disjunction of values of the same attribute in a condition) and

the range operator (to express conditions involving a range of discrete or continuous values)

These operators help to simplify rules involving multi valued discrete attributes the second

operator is also used for creating logical expressions involving continuous attributes

AQ15 can generate decision rules that represent either characteristic or discriminant concept

descriptions depending on the settings of its parameters (Michalski 1983) A characteristic

description states properties that are true for all objects in the concept The simplest

characteristic concept description is in the form of a single conjunctive rule (in general it can be

a set of such rules) The most desirable is the maximal characteristic description that is a rule

with the longest condition part ie stating as many common properties of objects of the given

class as can be determined A discriminant description states properties that discriminate a given

concept from a fixed set of other concepts The most desirable is the minimal discriminant

descriptions that is a rule with the slwnest condition part For example to distinguish a given

set of tables from a set of chairs one may only need to indicate that tables have large flat top

A characteristic description of the tables would include also properties such as have four legs

have no back have four corners etc Discriminant descriptions are usually much shorter than

characteristic descriptions

Another option provided in AQ15 controls the relationship among the generated descriptions

(rulesets or covers) of different decision classes In the IC (Intersecting Covers) mode

rulesets of different classes may logically intersect over areas of the description space in which

there are no training examples In the DC (Disjoint Covers) mode descriptions of different

classes are logically disjoint The DC mode descriptions are usually more complex both in the

number of rules and the number of conditions There is also a DL mode (a Decision List mode

27

also called VL modeM-for variable-valued logic mode) in which the program generates rule sets

that are linearly ordered To assign a decision to an example using such rulesets the program

evaluates them in order If ruleset is satisfied by the example then the decision is made

otherwise the program proceeds to the evaluation of the ruleset +1 In IC and DC modes

rule sets can be evaluated in any order

Alternatively the system can use rules from the AQ17-DCI program for Data-driven

Constructive Induction AQ17-DCI differs from AQ15 mainly in that it contains a module for

generating additional attributes These attributes are various logical or mathematical

combinations the original attributes The program generates a large number of potential new

attributes and seleets from them those most promising based on an attribute quality criterion

To illustrate the format of rules generated by AQ15 (or AQ17-DCI) an exemplary ruleset is

shown in Figure 3-2 The ruleset (that can be re-represented as a disjunctive normal form

expression) describes a voting record of Democratic Representatives in the US Congress

Each rule is a conjunction of elementary conditions Each condition expresses a simple relational

statement For example the condition [State = northeast v northwest] states that the attribute

State (of the Representative) should take the value northeast or northwest to satisfy the

condition

Rl [Gas_concban = yes] amp [Soc_see_cut = no v not registered] R2 [Draft = Yes v not registered]amp[AlaslaLparks = yes v not registered] amp

[Food_stamp_cap = no] amp [State = northeast v northwest] R3 [Chrysler =yes v not registered] amp [Income =low] R4 [Education =yes] amp [Occupation = yes]

Figure 3-2 A ruleset generated by AQ15 for the concept Voting pattern of Democratic Representatives

The above rules were generated from examples of the voting records For illustration below is

an example of a voting record by a Democratic representative

Draft registration=no Ban of aid to Nicaragua=no Cut expenditure on mx missiles=yes Federal subsidy to nuclear power stations=yes Subsidy to national parks

28

in Alaska=yes Fair housing bill=yes Limit on Pac contributions=yes Limit onood stamp program=no Federal help to education=no StateFrom=north east State Population=large Occupation =unknown Cut in social security spending=no Federal help to Chrysler corp=not registered

By expressing elementary statements in the example as conditions and linking conditions by

conjunction the examples can be re-expressed as decision rules Thus decision rules and

examples formally differ only in the degree of generality

33 Generating Decision Structurestrees from Decision Rules

This section describes the AQDT-1 system for learning decision structures (trees) from decision

rules (Imam amp Michalski 1993ab) Also a description of the AQDT-2 method for learning taskshy

oriented decision structure from decision rules is included and fmally the methodology is

illustrated by two examples

Methods for learning decision trees from examples have been very popular in machine learning

due to their simplicity Decision trees built this way can be quite efficient as long as they are

used in decision-making situations for which they are optimized and these situations remain

relatively stable Problems arise when these situations significantly change and the assumptions

under which the tree was built do not hold anymore For example in some situations it may be

difficult to determine the value of the attribute assigned to some node One would like to avoid

measuring this attribute and still be able to classify the example if this is potentially possible

(Quinlan 1990) If the cost of measuring various attributes changes it is desirable to restructure

the tree so that the inexpensive attributes are evaluated f11st A tree restructuring is also

desirable if there is a significant change in the frequency of occurrence of examples from

different classes A restructuring of a decision tree to suit the above requirements is however

difficult to do The reason for this is that decision trees are a form of decision structure

representation and imposes constraints on the evaluation order of the attributes that are not

logically necessary

29

One problem in developing a method for generating decision structures from decision rules is to

design an attribute selection criterion that is based on the properties of the rules rather than of

the training examples A decision rule normally describes a number of possible examples Only

some of them are examples that have actually been observed ie training examples An attribute

selection criterion is needed to analyze the role of each attribute in the rules It cannot be based

on counting the numbers of training examples covered by each attribute-value and the

frequency of decision classes in the training examples as is done in learning decision trees from

examples because the training examples are assumed to be unavailable

Another problem in learning decision trees from decision rules stems from the fact that decision

rules constitute a more powerful knowledge representation than decision trees They can directly

represent a description in an arbitrary disjunctive normal form while decision trees can represent

directly only descriptions in the disjoint disjunctive normal form In such descriptions all

conjunctions are mutually logically disjoint Therefore when transforming a set of arbitrary

decision rules into a decision tree one faces an additional problem of handling logically

intersecting rules

The solution to both problems (attribute selection and logically intersected rules) in the AQDT-2

system is based on the earlier work by Michalski (1978) which introduced a general method for

generating decision trees from decision rules The method aimed at producing decision trees

with the minimum number of nodes or the minimum cost (where the cost was defined as the

total cost of classifying unknown examples given the cost of measuring individual attributes and

the expected probability distribution of examples of different decision classes) More

explanations are provided in the following section

331 Tbe AQDT-2attribute selection metbod

This section describes the AQDT -2 method for building a decision structure from decision rules

The method for building a single-parent decision structure is similar to that used in standard

30

methods of building a decision tree from examples The major difference is that it assigns tests

(attributes) to the nodes using criteria based on the properties of the decision rules (this includes

statistics about the examples covered by each rule in the case of learning rules from examples)

rather than statistics characterizing the frequency of training examples per decision classes per

attribute-values or per conjunctions of both Other differences are that the branches may be

assigned an internal disjunction of values (not only a single value as in a typical decision tree)

and leaves may be assigned a set of alternative decisions with probabilities Also the tests can be

attributes or names standing for logical or mathematical expressions that involve several

attributes or variables In the following we use the terms test and attribute interchangeably

(to distinguish between an attribute and a name standing for an expression the latter is called a

constructed attribute)

At each step the method chooses the test from an available set of tests that has the highest utility

(see below) for the given set of decision rules This test is assigned to the node The branches

stemming from this node are assigned test values or disjoint groups of values (in the form of

logical disjunction if such occur in the rules subsumed groups of values are removed) Each

branch is associated with a reduced set of rules determined by removing conditions in which the

selected attribute assumes value(s) assigned to this branch If all rules in the reduced ruleset

indicate the same decision class a leaf node is created and assigned this decision class The

process continues until all nodes are leaf nodes If it is not possible to reduce further the rule set

because some attribute is declared as unavailable (infmite cost) then the leaf is assigned a set of

candidate decisions with associated probabilities (see Sec 42)

The test (attribute) utility is a combination of one or more of the following elementary criteria 1)

cost which indicates the cost of using each attribute for making decision 2) disjointness which

captures the effectiveness of the test in discriminating among decision rules for different decision

classes 3) importance which determines the importance of a test in the rules 4) value

31

distribution which characterizes the distribution of the test importance over its of values and 5)

dominance which measures the test presence in the rules These criteria are dermed below

~ The cost of a test expresses the effort or cost needed to measure or apply the test

Djsjointness The disjointness of a test is defined as the sum of the class disjointness-the

disjointness of the test for each decision class Suppose decision classes are Cl C2 bull Cm and

decision rule sets for these classes have been determined Given test A let V 1 V 2 bull V m denote

sets of the values (outcomes) of A that are present in rule sets for classes C 1 C2bullbull Cm

respectively If a ruleset for some class say Ct contains a rule that does not involve test A than

Vt is the set of all possible values of A (the domain of A)

Definition 3 -1 The degree ofclass disjointness D(A Ci ) of test A for the ruleset ofclass Ci is

the sum of the degrees of disjointness D(A Cit Cj) between the ruleset for Ci and rulesets for

Cj j=l 2 m j J L The degree of disjointness between the ruleset for Ci and the ruleset for Cj is

dermed by

if Vi Vj ifVj gtVj (3-1) ifVinVj -tljgt amp -tVi amp -tVj ifVinVj=1jgt

where cjgt denotes the empty set Note that exchanging the second and third conditions of equation

(3-1) may seem to be an improved criteria However it does not clearly distinguish between both

cases (Le for both situations the disjointness will be similar) The current equation is better

because it gives higher scores to attributes that classify different subsets of the two decision

classes than attributes that classify only a subset of one decision class

Definition 3-2 The disjointness of the test A for evaluating a given set of decision rules is the

sum of the degrees of class disjointness of each decision class

32

m m Disjointness(A) = L D(A cn where D(A cn = L D(A Ci Cj) (3-2)

i=l i=l i~j

The disjointness of a test ranges from 0 when the test values in rulesets of different classes are

all the same to 3m(m-l) when every rule set of a given class contains a different set of the test

values If two tests have the same disjointness value the attribute to be selected is the one with

less number of values

Definition The A vera~ Number of Tests (ANT) required to make decision from a decision tree

is dermed as the average number of tests (attributes) to be examined from the root of the tree to

any leaf node in order to reach a decision

Definition A decision structure is a one-node-per-level decision structure if at each level there is

only one node and zero or more leaves

Such decision structure can be generated by combining together all branches that associated with

each one a set of decision rules belongs to more than one decision class

Theorem 1 Consider learning one-node-level decision structure for a database with two

decision classes The disjointness criterion ranks first attributes that add minimum number of

tests to the decision tree

Proof Consider all possible distributions of an attributes values within two decision classes Ci

and Cj There are three cases 1) subset (same as super set) 2) non-empty intersection but not

subset 3) no intersection Figure 3-3 shows all possible distributions (the case of having the

same set of values in both classes is trivial one) Consider that branches leading to one subset

with the same decision class are combined into one branch In the first case there will be two

branches only The first leads to a leaf node and the other leads to an intermediate node where

another attribute to be selected The minimum ANT in this case is 53 In the second case three

branches should be created Two branches leads to a leaf node where all values at each branch

belong to only one and different decision class The third branch leads to an intermediate node

33

where another attribute should be selected furtur classifies the decision classes The minimum

ANT in this case is 64 In the third case only two branches will be generated where each leads

to leaf node with different decision class In this case the minimum ANT is 1

Figure 3-4 shows decision trees equivalent to selecting the corresponding attribute Note that in

case of having more than one attribute-value at branches lead to leaves belonging to one decision

class they will be combined into one branch in the decision structure The symbol 1 means

that an attribute is needed to classify the two decision classes In such cases there will be at least

two additional paths

D(A Ci) = 0 D(A CJ) = 1 D(A Ci) =2 D(A Cj) = 2 D(A Ci) =3 D(A Cj) =3

Figure 3-3 Yen Diagrams of Possible combinations of attribute values in two decision classes

The average number of tests required for making a decision in each possible case is determined

in Figure 3-4 It is clear that in the case of two decision classes the disjointness criterion ranks

highly attributes that reduces the average number of tests required for decision-making The

theorem can be proved in the general case

i ~ i Cj C1middot bull Ci Cj

ANT=32middotANT=53 ANT=l

1 means at least one attribute is needed to complete the decision tree Figure 3-4 Decision trees corresponding the Yen diagrams inFigure 3-3

34

Theorem 2 The attribute with the highest non~zero disjointness is the best attribute to classify

the decision classes

Proof Suppose that the number of decision classes is n Assume also that there are two attributes

A and B where D(A) lt D(B) Since the disjointness criterion considers the mutual relationship

between any two decision classes this means that there are more decision classes where D(A

Ci) lt D(B Ci) than those where D(A Ci) gt D(B Ci) (ignoring classes with equal disjointness

for bCth attributes) Hence there are more decision classes where D(A Ci Cj) lt D(B Ci Cj)

than those where D(A Ci Cj) gt D(B Cit Cj) Let us frrst prove that if D(A Ci Cj) lt D(B Cit

Cj) then B classifies the decision classes better than A

For each pair of decision classes Ci and Cj the possible values of the disjointness of any attribute

are 0 14 or 6 For all positive values ofD(B)= 14 or 6 it is clear that attribute B should have

smaller ANT than attribute A which has lower disjointness This means that if D(A Cit Cj) lt

D(B Cit Cj) then attribute B can classify both decision classes better than attribute A

Similarly B is better for classifying more pairs of decision classes than A This implies that B is

a better classifier than A

Importance The second elementary criterion the importance of a test is based on the

importance score (IS) introduced in (Imam Michalski amp Kerschberg 1993) In the obtained

rules each test is assigned a score that represents the total number of training examples that

are covered by the rules involving this test Decision rules learned by an AQ learning program

are accompanied with information on their strength Rule strength is characterized by t-weight

and u~weight The t-weight (total-weight) of a rule for some class is the number of examples of

that class covered by the rule The importance score of a test is the aggregation of the totalshy

weights of all rules that contain that test in their condition part Given a set of decision rules for

m decision classes Cl Cm and involving n tests AlbullAn and the number of rules associated

with class Ci is denoted by tlrit The importance score is defmed as follow

35

Definition 3 -3 The importance score IS(Aj) of the test A j is detennined by

m IS(Aj)= L IS(Aj Ci) (3-31)

i=l

where ri IS(Aj Ci) = L Rik(Aj) (3-32)

k=l

and Rik the weight of a test Aj in the rule Rk of class Ci is given by

t - weight if A j belongs to rule Rik Rik (Amiddot) = (3-4)J 0 otherwise

where i=I n ik=I ri j=I m

The importance score method has been separately compared as a feature selection method with

a genetic algorithm-based method (Imam amp Vafaie 1994) The importance score method

produced an equal or higher accuracy on three real-world problems than those reported by the

GA method while selecting fewer attributes In addition the IS method was significantly faster

than the GA

yalue distribution The third elementary criterion value distribution concerns the number of

legal values of tests Given two tests with equal importance scores this criterion prefers the test

with the smaller number of legal values Experiments have shown that this criterion is especially

useful when using discriminant decision rules

Definition 34 A value distribution VD(Aj) of a test A j is defined by

VD(Aj) = IS(Aj) I Vj (3-5)

where v is the number of legal values of Aj

Dominance The fourth elementary criterion dominance prefers tests that appear in large

numbers of rules as this indicates their high relevance for discriminating among the rulesets of

given decision classes Since some conditions in the rules have values linked by internal

disjunction counting such rules directly would not reflect properly their relevance Therefore

36

for computing the dominance the rules are counted as if they were converted to rules that do not

have internal disjunction Such a conversion is done by multiplying out the condition parts of

the rules containing internal disjunction For example the condition part [x3=1 v 3]amp[x4=1] is

multiplied out to two rules with condition parts [x3=1]amp[x4=1] and [x3=3]amp[x4=I]

The above criteria are combined into one general test measure using the lexicographic

evaluation functional with tolerances (LEF) (Michalski 1973) LEF is a list of some or all of

the above elementary criteria each associated with a Ittolerance threshold It in percentage The

criteria are applied to tests in the order defined by LEF A test passes to the next criterion only if

it scores on the previous criterion within the range defined by the tolerance (from the top value)

The default LEF is

ltCost d Disjointness t2 Importance t3 Value distr t4 Dominance 0gt (3-6)

where d a t3 t4 and t5 are tolerance thresholds (in percentages) their default values are O

The default value of the cost of each test is 1

The above LEF ranks attributes this way First attributes are evaluated on the basis of their cost

If two or more attributes have the same cost they ranked by their degree of disjointness If two

or more attributes still share the same top score or their scores differ less than the assumed

tolerance threshold t2 the method evaluates these attributes using the second (Importance)

criterion If again two or more attributes share the same top score or their scores differ less than

the tolerance threshold t3 then the third criterion normalized IS is used then similarly the

fourth criterion (dominance) If there is still a tie the method selects the best attribute

randomly

If there is a non-uniform frequency distribution of examples of different classes then the

selection criterion uses a modified definition of the disjointness Namely the previously defmed

37

disjointness for each class is multiplied by the frequency of the class occurrence The class

occurrence is the expected number of future examples that are to be classified to a given class

m

Disjointness(A) = L D(A CO Frq(Ci) (3-7) i=l

where Frq(Cn is the expected frequency of examples of class Cit and it is assumed to be given

by the user The attribute ranking criteria in this case is defined by the LEF

ltCost d Disjointness t2 Importance 13 Normalized-IS 14 Dominance 15gt (3-8)

where the Cost denotes the evaluation cost of an attribute and is to be minimized while other

elementary criteria are treated the same way as in the default version

332 The AQDT-2 algorithm

The AQDT-2 algorithm constructs a decision structure from decision rules by recursively

selecting at each step the best test according to the ranking criteria described above and

assigning it to the new node The process stops when the algorithm creates terminal branches that

are assigned decision classes To facilitate such a process the system creates a special data

structure for each concept description (ruleset) This structure has fields such as the number of

rules the number of decision classes and the number of attributes present in the rules A set of

pointers connect this data structure to set of data structures each represents one decision class

The decision class structure contains fields with information on the number of rules belong to

that class the frequency of the decision class etc It is also connected to a set of data structures

representing the decision rules within each decision class The system creates independently a set

of data structures each is corresponding to one attribute Each attribute description contains the

attributes name domain type the number of legal values a list of the values the number of

rules that contain that attribute and values of that attribute for each rule The attributes are

arranged in an array in a lexicographic order flISt in the descending order of the number of rules

38

that contain that attribute and second in the ascending order of the number of the attributes

legal values

The system can work in two modes In the standard mode the system generates standard

decision trees in which each branch has a specific attribute-value assigned In the compact

mode the system builds a decision structure that may contain

A) or branches Le branches assigned an internal disjunction of attribute-values whenever it

leads to simpler structures For example if a node assigned attribute A has a branch marked by

values 1 v 2 then the control passes along this branch whenever A takes value 1 or 2 The

program creates or branches on the basis of the analysis of the value sets Vi while computing

the degree of attribute disjointness

B) nodes that are assigned derived attributes that is attributes that are certain logical or

mathematical combinations of the original attributes To produce decision structures with

derived attributes the input decision rules are generated by AQ17 (rather than AQI5) The

AQ17 rules may contain conditions involving attributes constructed by the program rather than

those originally given

To generate decision structures from rules the AQDT-2 method prefers either characteristic or

discriminant disjoint rule descriptions (given by an expert or learned by a system) Disjoint rules

are more suitable for building decision structures Assume that the description of each class is in

the form of a ruleset and that this set is the initial ruleset context The AQDT algorithm is

The AQDT2 Algorithm

Step 1 Evaluate each attribute occurring in the ruleset context using the LEF attribute ranking

measure Select the highest ranked attribute Let A represents this highest-ranked

attribute

Step 2 Create a node of the tree (initially the root afterwards a node attached to a branch) and

assign to it the attribute A In standard mode create as many branches from the node as

39

there are legal values of the attribute A and assign these values to the branches In

compact mode (decision structures) create as many branches as there are disjoint value

sets of this attribute in the decision rules and assign these sets to the branches

Step 3 For each branch associate with it a group of rules from the ruleset context that contain a

condition satisfied by the value(s) assigned to this branch For example if a branch is

assigned values i of attribute A then associate with it all rules containing condition [A=

i v ] If a branch is assigned values i v j then associate with it all rules containing

condition [A= i v j v ] Remove from the rules these conditions If there are rules in

the ruleset context that do not contain attribute A add these rules to all rule groups

associated with the branches stemming from the node assigned attribute A (This step is

justified by the consensus law [x=1] == [x=1] amp [y=a] v [x=1] amp [y=b] assuming

that a and b are the only legal values of y) All rules associated with the given branch

constitute a ruleset context for this branch

Step 4 If all the rules in a ruleset context for some branch belong to the same class create a leaf

node and assign to it that class If all branches of the trees have leaf nodes stop

Otherwise repeat steps 1 to 4 for each branch that has no leaf

To select an attribute to be a node in the decision tree (step 1 and 2 of the algorithm) the

algorithm performs two independent iterations In the first iteration it parses through all decision

rules and determine information about each attributes This information includes the importance

score of each attribute the number of rules containing a given attribute the disjoint value sets of

each attribute and the attribute values used in describing each decision class The second

iteration is only performed if the disjointness criterion is ranked first in the LEF function The

second iteration evaluates the attributes disjointness for each decision class against the other

decision classes

40

To determine the complexity of this process suppose that the maximum number of conditions

fonned by a single attribute value in one rule is s (where s ~ n the number of attributes) and r is

the total number of decision rules (in all decision classes)

m

r=LRt (m is the number of decision classes) 1=1

where Ri is the number of rules in decision class Ci bull The complexity of the frrst iteration can be

determined as

Cmpx(Itcl) =O(r s)

In the second iteration the disjointness is calculated between the decision classes and all

attributes The complexity of the second iteration can be given by

Cmpx(Itc2) = O(n m)

Assume that at each node 1 is the maximum of the number of decision classes to be classified at

this node and the number of rules associated with this node

l=max mr (3-9)

Since these two iteration are independent of each other the complexity of the AQDT algorithm

for building one node in the decision tree say Node Complexity NC(AQDT) is given by

NC(AQDT) = 0(1 n)

Usually I equals to the number of rules associated with the given node Thus the AQDT

complexity for building one node is a function of the number of attributes multiplied by the

number of rules associated with this node

At each level of the decision tree these two iterations are repeated for each non-leaf node The

Level Complexity of the AQDT algorithm LC(AQDT) can be given by

LC(AQDT) lt 0(1 n)

Which is less than the complexity of generating the root of the decision tree To explain this

consider the maximum possible number of non-leaf nodes at one level is half the number of the

initial decision rules (integer of rll) Figure 3-5-a shows an example of such situation where at

41

the lowest level each node classifies only two rules each belongs to different decision class This

decision tree could be a multimiddotvalues tree However the number of non-leaf nodes at each level

will be twice or more than the number of nonmiddot leaf nodes at the previous level Considering the

level complexity of the AQDT algorithm equal to (1 s 0) where 0 is the number of non-leaf

nodes at the given level In such cases it is either (1 0 S r) or (1 s lt r) In Figure 3middot5middota the

complexity of the AQDT algorithm at any lower level is given by

LC(AQDT) =0(2 s r(2) lt O(n I) =NC(AQDT)

a) per one level b) per one path

Figure 3 -5 Decision trees showing the maximum number of none leaf nodes

Note also that after selecting an attribute to be the root of the decision structure this attribute

and all conditions containing that attribute are removed from the data structure of the algorithm

Also if a leaf node is generated all rules belonging to the corresponding branch will not be

tested again

Since the disjointness criterion selects the attribute which minimizes the average number of tests

ANT it is true that the AQDT algorithm generates decision trees with the least number of levels

The number of levels per a decision tree is supposed to be less than or equal to the minimum of

both the number of attributes and the number of rules Consider k as the number of levels in a

given decision tree

k S min mr (3-10)

There are two cases represent the most complex situations Figure 3-5-a and 3-5-b In the fIrst

case where the decision rules were divided evenly the number of levels will be a function of the

42

logarithm of the number of rules In such case the complexity of the AQDT algorithm for

generating a decision tree from a set of decision rules is given by

Complexity(AQDT) = 0(1 n log r) (3-11)

The other situation is when the generated decision tree has the maximum number of levels The

maximum possible number of levels per a decision tree equals to one less than the number of

decision rules Figure 3-5-b Using the disjointness criterion it is not likely to get such a decision

tree because it has the maximum average number of test (ANT) that can be determined from the

same set of nodes and leaves However such a decision tree can be generated if the number of

decision classes is one less thanthe number of attributes In such case any disjQint decision rules

should have a maximum length that is less or equal to the floor of the logarithm of the number of

attributes Thus the level complexity of this decision tree is estimated as

LC(AQDT) =0(1 log n)

The maximum number of levels in such decision tree is k-l Thus the complexity of the AQDT

algorithm in such cases is given by

Complexity(AQDT) = 0(1 k log n) (3-12)

Since k is the minimum of n and r then n is the maximum of k and n Also since in (3-12)

r=n+1 then log r is greater than log n From 3-10 3-11 and 3-12 the complexity of the AQDT

algorithm is determined by

Cmplx(AQDT) = OCr k log 1) (3-13)

333 An example illustrating the algorithm

The following simple example illustrates how AQDT is used in selecting the optimal set of

testing resources for testing a software Suppose there are three tools for testing a software 1)

modeling (Tl) 2) checklist (T2) and 3) par_simul (T3) Also assume that there are four

different factors that affect the selection of any tool 1) the cost of using the tool (xl) 2) the

metrics that support the tool (x2) 3) the best phase for applying the tool (x3) and 4) type of tool

43

(automated semi-automated or manual) (x4) Table 3-1 shows these attributes and their possible

values

Suppose that the domain expert provided a set of rules to be used in testing a software Figure 3shy

6 shows a sample of these rules in AQ15c formats

Table 3-1 The available tools and the factors that affect the prclCes

Tl lt= [xl=2] amp [x2=2] v [xl=3] amp [x3=1 v 3] amp [x4=1] T2 lt= [xl=l v 2] amp [x2=3 v 4] v [xl=3] amp [x3=1 v 2] amp [x4=2] T3 lt= [xl=l] amp [x2=1] v [xl=4] amp [x3=2 v 3] amp [x4=3]

Figure 3-6 Decision rules for selecting the best tool for testing software

These rules can be interpreted as

Rule 1 Use the fltst tool for testing if you need average cost and the tool is

supported by the requirement metric

Rule 2 Use the fIrst tool for testing if you can afford high cost for testing either

in the requirement or the analysis phases and you need an automated tool

Rule 3 Use the second tool for testing if the cost limit is low or average and the

tool is supported by the system usage metric or the intractability metric

Rule 4 Use the second tool for testing if you can afford high cost for testing

either in the requirement or the design phases and you need a manual tool

Rule 5 Use the third tool for testing if the you are limited to low cost and the

tool is supported by the error rate metric

Rule 6 Use the third tool for testing if you can afford very-high cost for testing

either in the requirement or the system usage phases and you need a semishy

automated tool

44

Table 3-2 presents information on these rules and the disjointness values for all attributes For

each class the row marked Values lists values occurring in the ruleset for this class For

evaluating the disjointness of an attribute say A each rule in the ruleset above that does not

contain attribute A is characterized as having an additional condition [A= a vb ] where a b

are all legal values of A

The row Class disjointness specifies the class disjointness for each attribute The attribute xl

has the highest disjointness (11) and is assigned to the root of the tree For simplicity assume

the tolerances for each elementary criterion equal O

From the rules in Figure 3-6 we can also determine disjoint groupings of attribute-values used

for the compact mode of the algorithm This done as follows 1) determine for each attribute the

sets of values that the attribute takes in individual decision rules and remove those value sets

that subsume other value sets The remaining value sets are assigned to branches stemming from

the node marked by the given attribute For example x 1 has the following value sets in the

individual decision rules 2 3 I 2 I and 4 (Figure 3-6) Value set I 2 is

removed as it subsumes 2 and I In this case branches are assigned individual values of the

domain of x1 For attribute x2 the value sets are I 2 34 and I 2 3 4 In this case

branches are assigned value sets I 2 and 34

45

Attribute Xl ranks highest (as it is the highest disjointness) and is assigned to the root of the

tree Four branches are created each one corresponding to one of XIs possible values Since all

rules containing [x1=4] belong to class TI the branch marked by 4 is ended by a leafTI Rules

containing other values of X1 belong to more than one class This process is repeated for each

subset of rules until the decision tree is completed Figure 3-7 shows a decision structure learned

by AQDT-2 from the rules in Figure 3-6 The decision structure in Figure 3-7 can be used in

making decisions on which tools can be used for testing a given software

xl

Complexity No of nodes 4 No of leaves 7

Figure 3-7 A decision structure learned for classifying so tware testing tools

Figure 3-8-a shows the diagrammatic visualization of the decision rules and Figure 3-8-b shows

the visualization of the derived decision tree Each diagram in Figure 3-8 consists of cells

representing one combination of attribute-values Attributes and their legal values are shown on

scales surrounding the diagram (eg the horizontal scale for xl shows values 1 2 3 and 4)

Rules are represented by collections of cells in the intersection of the rows and columns

corresponding to the conditions in the rules

The shaded areas correspond to decision rules Rules of the same class have the same type of

shading Empty cells correspond to combination of attribute-values not assigned to any class For

illustration collections of cells corresponding to some of the initial rules are marked R 11 R 21

R31 and R32 Rll denotes the IlfSt rule of Class T1 ie [x1=2] amp [x2=2] R21 denotes the

IlfSt rule of class T2 ie [xl= 1 v 2] amp [x2= 3 v 4] R31 the first rule of class TI ie [x1=1]

46

amp [x2 =1] and denotes R32 denotes the second rule of class T3 Le [xl=4] amp [x3=2 v 3] amp

[x4=3]

a) Decision rules b) Derived decision tree

Comparing diagrams in Figure 3-8-a and 3-8-b AQDT-2 decision tree represents a slightly more

general description of Concepts Tl TI and T3 than the original rules Let us assume that it is

very costly to know which metrics suppon the required tools In other words suppose that we

would like to select the best tools independently of the metric they suppon (this is indicated to

AQDT-l by assigning very high cost to attribute x2) The algorithm again assigns attribute xl to

the root of the tree When xl takes value 1 or 2 it is impossible to assign a specific decision

without measuring attribute x2 For the value 1 of xl the recommended tool can be either TI or

T3 (see the diagram in Figure 3-9-a) and for the value 2 of xl the recommended tool can be

either Tl or T3 However for the value 3 of xl one can make a specific decision after

measuring attribute x4 For the value 1 of x4 tool is Tl and for the value 2 of x4 the

recommended tool is TI Figure 3-9-b shows another decision tree where the type of the tool

was ignored from the data Those decision trees are called indeterminate because some of their

leaves are assigned a disjunction of two or more class names

47

xl

xl

1 12

a) Ignoring the supporting metric b) Ignoring the type of the tooL Figure 3 -9 Decision trees learned ignoring the support metric and the type of the testing tool

It is clear that the given set of rules depend highly on amount of money can be spent Let us now

suppose that the cost attribute xl which was determined as the highest ranked attribute cannot

be measured The algorithm selected x4 to be the root of the new decision tree After continuing

the algorithm the tree in Figure 3-10 is obtainedThe nodes that are assigned one class indicate

situations in which it is possible to make a specific decision without knowing the value of

attribute xl

x4

Figure 3-10 A decision tree learned without the cost attribute

34 Tailoring Decision Structures to the Decision-making situation

Decision structures are among the simplest structures for organizing a decision-making process

A decision structure specifies explicitly the order in which attributes of an object or situation

need to be evaluated in the process of determining a decision A standard way to generate a

48

decision structure is to learn it from examples of decisions Such a process usually aims at

obtaining a decision structure that has the highest prediction accuracy that is the highest rate of

assigning correct decisions to given situations There can usually be a large number of logically

equivalent decision structures (Michalski 1990) As such they may have the same predictive

accuracy but differ in the way they organize the decision process and thus may differ in the cost

of arriving at a decision To minimize the average decision cost one needs to take into

consideration the distribution of the costs of attribute evaluation and the frequency of different

decisions This report presents an approach to building such Task-oriented decision structures

which advocates that they are built not from examples but rather from decision rules Decision

rules are learned from examples using the AQ15 or AQ17 inductive learning program or are

specified by an expert An efficient algorithm and a new system AQDT-2 that transforms

decision rules to task-oriented decision structures The system is illustrated by applying it to the

problem of learning decision structures in the area of construction engineering (for determining

the best wind bracing for tall buildings) In the experiments AQDT-2 outperformed all other

programs applied to the same data

Decision-making situations can vary in several respects In some situations complete

information about a data item is available (ie values of all attributes are specified) in others the

information may be incomplete To reflect such differences the user specifies a set of parameters

that control the process of creating a decision structure from decision rules AQDT-2 provides

several features for handling different decision-making problems 1) generating a decision

structure that tries to avoid unavailable or costly attributes (tests) 2) generating unknown

leaves in situations where there is insufficient information for generating a complete decision

structure 3) providing the user with the most likely decision when performing a required test is

impossible 4) providing alternative decisions with an estimate of the likelihood of their

correctness when the needed information can not be provided

49

341 Learning Costmiddot Dependent Decision Structures

As described in Sec 2 the LEF criterion enables the system to take into consideration the cost of

measuring tests (attributes) in developing a decision structure In the default setting of LEF the

tests cost is the [lIst criterion and its tolerance is O This means that only the least expensive

attributes pass to the next step of evaluation involving other elementary criteria If an attribute

has high cost or is impossible to measure (has an infinite cost) the LEF chooses another

cheaper attribute ifpossible

342 Assigning Decision Under Insufficient Information

In decision-making situations in which one or more attributes cannot be measured the system

may not be able to assign a definite decision for some cases If no more information can be

obtained but a decision has to be made it is useful to know the probability distribution for

different candidate decisions (Smyth Goodman amp Higgins 1990) The most probable decision is

then chosen The probability distribution can be estimated from the class frequency at the given

node Let us consider a node in the structure connected to the root by the sequence bl b2 bkof

attribute-values Let P(Ci I bl b2 bk) denote the conditional probability of class Ci i=I2 m

at that node given that an example to be classified has attribute-values assigned to branches bl

b2 bull bk in the decision structure Using Bayesian formula we have

P(Ci I bl~ bull bk) = P(Ci) P(bl bk I Ci) P(bl bk) (3-9)

where P(Ci) the apriori probability of class Ci P(bl bk I Ci) is the probability that the

example has attribute-values blbull bk given that it represents decision class Ci To approximate

these probabilities let us suppose that Wi is the number of training examples of Class Ci that

passed the tests leading to this node and twi - the total number of training examples of Class

Ci Let us use these numbers to estimate the probabilities in (9) Assuming that the apriori class

probabilities correspond to the frequency of training examples from different classes we have

50

m P(Ci) = twi IL twj (3-10)

j=l

P(b1 bkl Ci) = Wi I twi (3-11)

m m P(b1 bk) =L Wj IL twj (3-12)

j=l j=1

By substituting (1O) (11) and (12) in (9) we obtain

m P(Ci I b1 bk) = wilL Wj (3-13)

j=l

A related method for handling the problem of unavailability of an attribute is described by

Quinlan (1990) Quinlans method however assigns probabilities to the most probable decision

at the node associated with such an attribute The method does not restructure the decision tree

appropriately to fit the given decision-making situation (in this case to avoid measuring xl)

AQDT -2 assigns probabilities o~ly after it first created a decision structure that suits the given

decision-making si~ation An example is presented in section 42

343 Coping with noise in training data

The proposed methodology can be easily extended to handle the problem of learning from noisy

training data This is done based on ideas of rule truncation described in (Niblett amp Bratko 1986

Bergadanoet al 1992 Cestnik amp Karalic 1991) When noise is expected in the training data

the decision rules with t-weight below a certain threshold (reflecting the expected noise level) are

removed The rule truncation method seems to result in more accurate decision structures than

decision tree pruning method because truncation decisions are solely based on the importance of

the given rule or condition for the decision-making regardless of their evaluation order (unlike

the decision tree pruning which can only prunes attributes within a subtree and thus cannot

freely chose attributes to prune) Examples are presented in section 4

viii

LIST OF TABLES

No TITLE Page

2-1 An example of a decision table 9

2-2 A set of training examples used to illustrate the C45 system 15

2-3 The frequency of different attributes values to different decision classes 17

2-4 The expected values of the frequency of examples in Table 2-3 17

2-5 Attribute selection criteria and their basic evaluation measure 17

2-6 The contingency tables of Mingers example 18

2-7 Mingers results for determining the goodness of split 19

2-8 Mingers results for comaring the total accuracy and size of decision trees

provided by different attribute selection criteria from four problems 19

2-9 Comparing the AQDT approach with EDAG and HOODG approaches 22

3-1 The available tools and the factors that affect the process of testing a software 43

3-2 Calculating the disjointness of each attribute 44

3-3 Evaluation of the AQDT-2 criteria on the Quinlans example in Table 2-2 51

3-4 The data used in Mingers f11st experiments 52

3-5 The performance of AQDT-2 criteria (compare with the other criteria in Table 2-6) 52

3-6 The possible ranking domains and using condition of AQDT-2 criteria 53

3-7 Comparison between Decision Structures and Decision Tree 54

4-1 Evaluation of AQDT-2 attribute selection criteria for the wind bracing problem 62

4-2 The predictive accuracy of AQ15c and AQDT-2 for the wind bracing problem 67

4-3 Evaluation of AQDT-2 attribute selection criteria for the MONK-l problem 71

4-4 The predictive accuracy of AQ15c and AQDT-2 for the MONK-l problem 73

ix

4-5 The predictive accuracy of AQ15c and AQDT-2 for the MONK-2 problem 77

4-6 The predictive accuracy of AQ15c and AQDT-2 for the MONK-3 problem 81

4-7 The set of attributes and their values used in the trains problem 86

4-8 A tabular snmmary of the predictive accuracy of decision trees obtained

by AQDT -2 and C45 for the congressional voting data 88

4-9 Summary of the best parameter settings for the first subfunction of the approach

with different data characteristics 89

4-10 Summary of the performance of AQDT-2 and C45 on different problems 90

x

LIST OF FIGURES

No TITLE Page

2-1 An example to illustrate how attributes break rules 8

2-2 A decision tree learned from the decision table in Table 2-1 10

2-3 A decision tree learned using the gain criterion for selecting attributes 15

2-4 Decision rules and their Exception Directed Acyclic Graph (EDAG) 21

3-1 Architecture of the AQDT approach 24

3-2 A ruleset generated by AQ15 for the concept Voting pattern of

Democratic Representatives 27

3-3 Ven Diagrams ofPossible combinations of attribute values in two decision classes 33

3-4 Decision trees corresponding the Ven diagrams in Figure 3-3 33

3-5 Decision trees showing the maximum number of none leaf nodes 41

3-6 Decision rules for selecting the best tool for testing a software 43

3-7 A decision structure learned for classifying software testing tools 45

3-8 A diagrammatic visualization of the decision rules and the derived decision tree 46

3-9 Decision trees learned ignoring the support metric and the type of the testing tool 47

3-10 A decision tree learned without the cost attribute 47

3-11 Decision structures learned by AQDT-2 using different criteria 55

3-12 The Imams example Example where learning decision structures (trees)

from rules is better than learning them from examples 56

3-13 An example where decision rules are simpler than decision trees 57

4-1 Design of a complete experiment 59

Xl

4-2 Decision rules determined by AQ15c from the wind bracing data 61

4-3 A decision tree learned by C45 for the wind bracing data 63

4-4 A decision structure learned from AQ15c wind bracing rules 64

4-5 A decision structure that does not contain attribute xl 64

4-6 A decision structure without Xl with candidate decisions assigned to leaves 65

4-7 A decision structure determined from rules in Figure 4-4 under the

assumption of 10 classification error in the training data 65

4-8 Diagramatic visualization of decision trees learned for different decision

making situations for the wind bracing data 66

4-9 The accuracy of AQDT-2 and AQ15c with different parameter settings

for the wind bracing PIoblem 68

4-10 Analyzing different parameter setting of AQDT-2 using the wind bracing data 69

4-16 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-20 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-11 A comparison between AQ15c and AQDT-2 against C45 on the wind bracing data 69

4-12 A visualization diagram of the MONK-l problem 70

4-13 Decision rules learned by AQ15c for the MONK-1 problem 71

4-14 The decision tree for the MONK-1 problem generated both by AQDT-l 72

4-15 Compact decision structures generated by AQDT-2 for the MONK-1 problem 72

for the MONK-l Problem 74

4-17 Analyzing different parameter setting of AQDT-2 with the MONK-l data 75

4-18 Comparing AQ15c and AQDT-2 against C45 using the MONK-1 problem 75

4-19 A visualization diagram of the MONK-2 problem 76

for the MONK-2 Problem 78

4-21 Analyzing different parameter setting of AQDT-2 with the MONK-2 data 79

xii

4-22 Comparing AQ15c andAQDT-2 against C45 using the MONK-2 problem 79

4-24 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-35 A visualization diagram of a decision tree learned by AQDT -2 after reducing

4-23 A visualization diagram of the MONK-3 problem 80

for the MONK-3 Problem 82

4-25 Analyzing different parameter setting of AQDT-2 with the MONK-3 data 82

4-26 Comparing AQ15c and AQDT-2 against C45 using the MONK-3 problem 83

4-27 Comparing AQ15c and AQDT-2 against C45 using the breast cancer problem 84

4-28 Comparing AQ15c and AQDT-2 against C45 using the mushroom problem 85

4-29 Decision structures learned by AQDT-2 for different decision-making situations 87

4-30 Comparing decision trees for the Cong Voting-84 data learned by C45 amp AQDT -2 88

4-31 A visualization diagram of decision rules learned by AQ15c for the MONK-2 91

4-32 A visualization diagram shows the testing errors for the rules in Figure 4-31 92

4-33 A visualization diagram of the decision tree learned by AQDT-2 for the MONK-2 93

4-34 A visualization diagram shows testing errors of AQDT-2 decision tree bull 94

the generalization degree to 1 94

DERIVING TASKmiddotORIENTED DECISION STRUCTURES

FROM DECISION RULES

Ibrahim M Fahmi Imam PhD

School of Information Technology and Engineering

Geotge Mason University Fall 1995

Ryszard S Michalski Advisor

ABSTRACT

This dissenation is concerned with research on learning task-oriented decision structures from

decision rules The philosophy behind this research is that it is more appropriate to learn

knowledge and store it in a declarative form and then when a decision making situation occurs

generate from this knowledge the decision structure that is most suitable for the given decision

making situation Learning decision structures from decision rules was first introduced by

Michalski (1978) The flIst implementation of this approach was done by Imam and Michalski

(1993 ab) called AQDT-l

This approach separates the function of generating a knowledge-base from the function of using

the knowledge-base for decision-making The first function focuses on learning accurate

consistent and complete concept description expressed in a declarative form The second function

is performed whenever a new decision-making situation occurrs a task-oriented decision structure

is obtained to suit that situation Thsk-oriented knowledge is defined as knowledge that is adapted

for solving a given decision-making situation (Imam amp Michalski 1994 Michalski amp Imam

1994)

The dissertation introduces the system AQJJT-2 for learning task -oriented decision structures from

decision rules or examples Each decision making situation is defined by a set of parameters that

controls the learning process of the AQDT-2 system The extensive experiments on AQJJT-2 show

that decision structures learned by it usually outperform in terms of accuracy and average size of

the decision structures those learned from examples by other well known systems The results

show also that the system does not work very well with noisy data The system is illustrated and

compared using applications of artificial problems such as the three MONKs problems (Thrun

Mitchell amp Cheng 1991) and the East-West 1iain problem (Michie et al 1994) It was also

applied to real world problems of learning decision structures in the areas of construction

engineering (for determining the best wind bracing design for tall buildings) medical

diagnosis(for learning decision rules for recognizing breast cancer) agriculture diagnosis (for

learning classification rules for distinguishing between poisonous and non-poisonous

mushrooms) and political data (for characterizing democratic and republican voting records)

CHAPTER 1 INTRODUCTION

11 Motivation and Overview

Learning and discovery systems should be able not only to generate and store knowledge but also

to use this knowledge for decision-making The main step in the development of systems for

decision-making is the creation of a knowledge structure that characterizes the decision-making

process The form in which knowledge can be easily obtained may however differ from the form

in which it is most readily used for decision-making It is therefore important to identify the form

of knowledge representation that is most appropIiate for learning (eg due to ease of its

modification) and the form that is most convenient for decision making

A simple and effective tool for describing decision processes is a decision structure which is a

directed acyclic graph that specifies an order of tests to be appliedto an object (or a situation) to

arrive at a decision about that object The nodes of the structure are assigned individual tests

(which may correspond to a single attribute a function of attributes or a relation) the branches are

assigned possible test outcomes or ranges of outcomes and the leaves are assigned a specific

decision a set of candidate decisions with corresponding probabilities or an undetermined

decision A decision structure reduces to a familiar decision tree when each node is assigned a

single attribute and has at most one parent when the branches from each node are assigned single

values of that attribute and when leaves are assigned single definite decisions Thus the problem

of generating a decision structure is a generalization of the problem of generating a decision tree

Decision trees are typically generated from a set of examples of decisions The essential

characteristic of any such method is the attribute selection criterion used for choosing attributes to

be assigned to the nodes of the decision tree being built Such criteria include the entropy

reduction the gain and the gain ratio (Quinlan 1979 83 86) the gini index ofdiversity (Breiman

et al 1984) and others (Cestnik amp Bratko 1991 Cestnik amp Karalic 1991 Mingers 1989a)

3

4

Adecision treedecision structure representations can be an effective tool for describing a decision

process as long as all the required tests can be performed and the decision-making situations it

was designed for remain constant (eg in a doctor-patient example the doctor should detennine

that answers for all symptoms appear in the decision tree) Problems arise when these assumptions

do not hold For example in some situations measuring certain attributes may be difficult or costly

(eg in the doctor-patient example a brain or blood test is needed which is very expensive or the

tools needed are not available) In such situations it is desirable to refonnulate the decision

structure so that the inexpensive attributes are evaluated frrst (assigned to the nodes close to the

root) and the expensive attributes are evaluated only if necessary (by assignment to the nodes far

away from the root) If an attribute cannot be measured at all it is useful to either modify the

structure so that it does not contain that attribute or-when this is impossible-to indicate

alternative candidate decisions and their probabilities A restructuring is also desirable if there is a

significant change in the frequency of occurrence of different decisions (eg in the doctor-patient

example the doctor may request a decision structure expressed in a specific set of symptoms

biased to classify one or more diseases or specify a certain order of testing)

Arestructuring ofa decision structure (or a tree) in order to suit new requirements could be quite

difficult This is because a decision structure is a procedural representation that imposes an

evaluation order on the tests In contrast no evaluation order is imposed by a declarative

representation such as a set of decision rules Tests (conditions) of rules can be evaluated in any

order Thus for a given set of rules one can usually build a huge number of logically equivalent

decision structures (trees) which differ in the test ordering Due to the lack of order constraints

a declarative representation (rules) is much easier to modify to adapt to different situations than a

procedural one (a decision structure or a tree) On the other hand to apply decision rules to make a

decision one needs to decide in which order tests are evaluated and thus needs a decision

structure

An attractive solution to these opposite requirements is to acquire and store knowledge in a

declarative form and transform it to a decision structure when it is needed for decision-making

5

This method allows one to create a decision structure that is most appropriate in a given decisionshy

making situation Because the number of decision rules per decision class is usually small (each

rule is a generalization of a set of examples) generating a decision structure from decision rules

can potentially be perfonned much faster than by generation from training examples Thus this

process could be done on line without any delay noticeable to the user Such virtual decision

structures are easy to tailor to any given decision-making situation

This approach allows one to generate a decision structure that avoids or delays evaluating an

attribute that is difficult to measure in some decision-making situation or that fits well a particular

frequency distribution of decision classes In other situations it may be unnecessary to generate

complete decision structure but it may be sufficient to generate only a part of it which concerns

only with decision classes of interest Thus such an approach has many potential advantages

This dissertation presents a new system called AQflf-2 The AWf-2 system generates a taskshy

oriented decision structure (decision structure that is adapted to the given decision-making

situation) from decision rules The decision rules are learned by either rule leaming system AQ15

(Michalski et al 1986) or system AQ17-DCI which has extensive constructive induction

capabilities (Bloedorn et al 1993)

To associate the decision rules with a given decision-making task AQflf-2 provides a set of

features including 1) enabling the system to include in the decision structure nodes corresponding

to new attributes constructed during the process of learning the decision rules 2) controlling the

degree of generalization needed during the development of the decision structure 3) providing four

new criteria for selecting an attribute to be a node in the decision structure that allow the system to

generate many different but equivalent decision structures from the same set of rules 4) generating

unknown nodes in situations when there is insufficient infonnation for generating a complete

decision structure 5) learning decision structures from discriminant rules as well as

characteristic rules and 6) providing the most likely decision when the decision process stops

due to inability to evaluate an attribute associated with an intennediate node

6

To test the methooology of generating decision structures from decision rules an extensive set of

planned experiments have been designed to test different aspects of the approach The experiments

include testing different combinations of parameters for each sub-function of the approach

analyzing the relatioship between decision rules and the decision structure learned from them and

comparing decision trees learned by the AQUf2 system the well known C45 (Quinlan 1993)

system for learning decision trees from examples Different experiments were designed to examine

the new features of the AQUf approach for learning task-oriented decision structures The

experiments were applied to artificial domains as well as real-world domains including MONK-I

MONK-2 and MONK-3 (Thrun Mitchell amp Cheng 1991) East-West trains (Michie et al

1994) Engineering Design-wind bracings (Arciszewski et al 1992) Mushrooms Breast

Cancer (Mangasarian amp WolbeJg 1990) and Congressional bting Records of 1984 The

MONKs problems are concerned with learning classification rules for robot-like figures MONK-l

requires learning a DNF-type description MONK-2 requires learning a non-DNF-type description

(one that cannot be easily described as a DNF rule using the original attributes) MONK-3 concerns

learning a DNF rule from noisy data The East-West trains dataset is a structural domain that

classifies two sets of trains (Eastbound and Westbound) The Engineering Design-wind

bracing data involves learning conditions for applying different types of wind bracing for ta11

buildings The Mushrooms data is concerned with learning classification rules for distinguishing

between poisonous and non-poisonous mushrooms The Breast Cancer data involves learning

concept descriptions for recognizing breast cancer The congressional voting data includes voting

records on different issues AQUf2 outperformed C45 on the average with respect to both the

predictive accuracy and tree size for most problems Apf-2 did not work very well with noisy

data or with problems that have many rules covering very few examples

12 The Problem Statement

There are many limitations and problems accompanied using decision trees for decision-making

CHAPTER 2 RELATED RESEARCH

21 Learning Decision Trees from Decision Diagrams

The AQDT-2 method proposed here is based on the earlier work by Michalski (1978) which

introduced an algorithm for generating decision trees from decision lists The method proposed

several attribute selection criteria These criteria are of increasing power of the main criterion

order cost estimate (the nth order cost estimates n=l 2 ) Michalski also analyzed two

specific criteria MAL and DMAL for selecting the optimal attribute for a node in the tree

based on properties extracted from the decision diagram In order to better explain the method

it is necessary to define some terms These terms will be used later in the dissertation

Definition 2-1 A ~ of a decision class C is a set of rules whose union includes the set of all

examples of class C and does not include any examples of other decision classes

Definition 2-2 A ~ of a decision class C is disjoint if all rules (rules) are pairwise logically

disjoint In other words for any two rules there exists a condition with the same attribute but

with different values in each rule

Definition 2-3 An minimal cover is a disjoint cover which has the smallest number of rules

among all possible covers

Definition 2-4 A diagram for a given cover is a table constructed graphically by representing

in two dimension space of all possible combination of attributes values and locating on the

diagram all the condition parts of the given rules and marking them with the action specified

with each rule

Michalskis algorithm requires construction of minimalmiddot cover The minimal cover should be

consistent and complete The method is based on the fact that if there are n decision classes

any decision tree that correctly classifies the given rules and has n distinct leaves is a minimal

7

8

decision tree (any consistent decision tree should have at least n leaves) Michalski (1978) has

shown that if only one rule is broken by a selected attribute then instead of having one leaf

(which could potentially represent this rule or the decision class in the tree) there will have to

be at least two leaves representing this rule in the fmal decision tree

The attribute selection criterion MAL introduced in (Michalski 1978) prefers attributes that do

not break any rules or break as few as possible An attribute breaks a rule if the attribute can

divide the rule into two or more sub-rules Figure 2-1 shows two examples of two sets of rules

In the fJIst xl and x3 break at least two rules each (xl breaks the rule [x4=2] amp [x1=lv3] amp

[x3=2v3] and the rule [x4=3] amp [x1=1v2] amp [x3=1] and x3 breaks three rules the rule [x4=2]

amp [xl=2] amp [x3=2v3]the rule [x4=l] amp [xl=3] amp [x3=lv3] and the rule [x4=3] amp [xl=4] amp

[x3=2v3]) In the diagram on the right xl is the only attribute that does not break any rule

Figure 2-1 An example to illustrate how attributes break rules

One of the criteria defmed by Michalski is the first degree cost estimate which assigns to each

attribute an integer equal to the number of rules broken by that attribute This criterion is also

called the static cost estimate of an attribute or the criterion of minimizing added leaves

(MAL)

-

C3

9

The MAL criterion (Minimizing Added Leaves) seeks an attribute that minimizes the

estimated number of additional nodes in the decision tree being generated over a hypothetical

minimal decision tree When there is a tie between two attributes the attribute to be selected is

the one which breaks smaller rules (rules that cover fewer examples or more specialized

rules) AQDT-2 uses an approximate version of this criterion (the attribute dominance)

Another criterion introduced by Michalski was the DMAL criterion The DMAL criterion

(Dynamically Minimizing Added Leaves) is based on a principle similar to that of MAL but

is more complex because once an attribute is selected as a node in the tree some rules andor

parts of the broken rules at each branch are merged into one rule The DMAL ensures that the

value of the total cost estimate of an attribute is decreased by a value equal to the number of

merged rules minus one

Example Learn a decision tree from the following decision table

The minimal cover consists of the following rules

Al lt [x2=O] v [x1=0][x2=2] A2 lt [x2=I] v [xl=2][x2=2] A3 lt [xl=I][x2=2]

The evaluations ofthe MAL criterion for these attributes are 2 for xl 0 for x2 5 for x3 and 5

for x4 The attribute xl divides the two rules [x2=O] and [x2=I] so selecting it will add two

leaves to the optimal number of leaves It is clear that the attribute to be selected as a root of

10

the decision tree is x2 Then three branches are attached to the root node and the decision rules

are divided into subsets each corresponding to one branch For x2 = 0 or 1 a leave node is

generated For x2=2 another attribute is selected to be a node in the tree In this case xl has the

minimum MAL value Figure 2-2 shows the decision tree obtained using the MAL criterion

Al A3 A2

Figure 2-2 A decision tree learned from the decision table in Table 2-1

22 Learning Decision Trees from Examples

Decision tree learning is a field that concerned with generating decision tree that classifies a set

of examples according to the decision classes they belong to The essential aspect of any

inductive decision tree method is the attribute selection criterion The attribute selection

criterion measures how good the attributes are for discriminating among the given set of

decision classes The best attribute according to the selection criterion is chosen to be assigned

to a node in the tree The fIrst algorithm for generating decision trees from examples was

proposed by Hunt Marin and Stone (1966) Hunts algorithm uses a divide and conquer

algorithm for building decision trees This algorithm has been subsequently modified by

Quinlan (1979) and applied by many researchers to a variety of learning problems

Attribute selection criteria can be divided into three categories These categories are logicshy

based information-based and statistics-based The logic-based criteria for selecting attributes

use logical relationships between the attributes and the decision classes to determine the best

attribute to be a node in the decision tree such as the MAL criterion minimizing added leaves

(Michalski 1978) which uses conjunction and disjunction operators The information-based

criteria are based on the information theory These criteria measure the information conveyed

11

by dividing the training examples into subsets Examples of such criteria include the

information measure 1M the entropy reduction measure and the gain criteria (Quinlan 1979

83) the gini index of diversity (Breiman et al 1984) Gain-ratio measure (Quinlan 1986) and

others (Clark amp Niblett 1987 Bratko amp Lavrac 1987 Cestnik amp Karalic 1991) The

statistics-based criteria measure the correlation between the decision classes and the attributes

These criteria use statistical distributions for determining whether or not there is a correlation

The attribute with the highest correlation is selected to be a node in the tree Examples of

statistics-based criteria include Chi-square and G-statistic (Sokal amp Rohlf 1981 Han 1984

Mingers 1989a)

Niblett and Bratko (1986) Quinlan (1987) and Bratko and Kononeko (1987) extended the

method of learning decision trees to also handle data with noise (by pruning) Handling noise

extended the process of learning decision trees to include the creation of an initial complete

decision tree tree pruning which is done by removing subtrees with small statistical Validity

and replacing them by leaf nodes (MiDgers 1989b) More recently pruning has also been used

for simplifying decision trees even for problems without noise (Bohanec amp Bratko 1994)

Pruning decision trees improves their simplicity but reduces their predictive accuracy on the

training examples Quinlan (1990) proposed also a method to handle the unknown attributeshy

value problem by exploring probabilities of an example belonging to different classes

The rest of this section includes a brief description of the attribute selection criterion used by

C45 learning system (Quinlan 1993) C45 uses an information-based criterion for selecting

an attribute to be a node in the tree Also the section includes a brief description of the Chishy

square method for attribute selection (Mingers 1989a) The method is a statistics-based

method for selecting an attribute to be a node in the tree

221 Building Decision Trees Using htformation-based Criteria

12

This section presents a description of the inductive decision tree learning system C4S The

C4S learning system is considered to be one of the most stable accurate and fastest program

for learning decision trees from examples

Learning decision trees from examples requires a collection of examples Each example is

represented by a fixed number of attribute-value pairs C45 (Quinlan 1993) is a learning

program that induces classification decision trees from a set of given examples The C4S

learning system is descended from the learning system ID3 (Quinlan 1979) which is based on

Hunts method for constructing decision tree from a set ofcases (Hunt Marin amp Stone 1966)

The C45 system uses an attribute selection criterion called the Gain Ratio This criterion

calculates the gain in classifying information based on the residual information needed to

classify cases in a set of training examples and the information yielded by the test based on the

relative frequencies of the possible outcomes (decision classes) The gain ratio criterion is

based on an earlier criterion used by ID3 called the Gain Criterion The Gain Criterion uses

the frequency of each decision class in the given set of training examples

Once an attribute is chosen to be a node in the tree the system generates as many links as the

number of its values and classifies the set of examples based on these values If all the

examples at a certain node belong to one decision class the system generates a leaf node and

assigns it to that class Otherwise the system searches for another attribute to be a node in the

tree

The Gain Criterion The gain criterion is based on the information theory That is the

information conveyed by a message depends on its probability and can be measured in bits as

minus the logarithm (base 2) of that probability To explain the gain criterion suppose that for

a given problem Xu bullbull X are given attributes and Cu c are decision classes Suppose S is

any set of cases and T is the initial set of training cases The frequency of class C1 in the set S

is the number of examples in S that belong to class C i bull

13

freq(Cit S) = Number of examples in S belong to CI (2-1)

Suppose that lSI is the total number of examples in S the probability that an example selected

at random from S belongs to class Ci is freq(CIbullS) IISI

The information conveyed by the message that a selected example belongs to a given decision

class Cit is determined by -log1 (freq(Ch S) IISI) bits

The expected information from such a message stating class membership is given by

info(S) = -k

L (freq(C S) I lSI) log1 (freq(C S) IISI) bits (2-2)I

info(S) is known also as the entrQpy of the set S When S is the initial set of training examples

info(T) determines the average amount of information needed to identify the class of an

example in T

Suppose that we selected an attribute X to be the root of the tree and suppose that X has k

possible values The training set T will be divided into k subsets each corresponding to one of

Xs values The expected information of selecting X to partition the training set T infoz(T) can

be found as the sum over all subsets of multiplying the information conveyed by each subset

by its probability

k

infoz(T) =L (ITII 1m) infO(TIraquo (2-3) I

The information gained by partitioning the training examples T into subset using the attribute

X is given by

gain(X) =inlo(T) - inoIT) (2-4)

The attribute to be selected is the attribute with maximum gain value

The Gain Ratio Criterion This criterion indicates the proportion of information generated by

the split that appears helpful for classification Quinlan (1993) pointed out that the gain

criterion has a serious deficiency Basically it is strongly biased toward attributes with many

outcomes (values) For example for any data that contains attributes such as social security

14

number the gain criterion will select that attribute to be the root of the decision tree However

selecting such attributes increases the size of the decision tree Quinlan provided a solution to

this problem by introducing the gain ratio criterion which takes the ratio of the information that

is gained by partitioning the initial set of examples T by the attribute X to the potential

information generated by dividing T into n subsets

Following similar steps to obtain the information conveyed by dividing T into n subsets The

expected information generated by dividing T into n subsets and by analogy to equation 2-2 is

determined by

split info(T) = - Lt

(ITII lTI) 10gl (ITII m ) (2-5) I

The gain ratio is given by

gain ratio(X) = gain(X) I split info(X) (2-6)

and it expresses the proportion of information generated by the split that is useful for

classification

Example Consider the following example presented by Quinlan (1993) Table 2-2 shows the

set of training examples

First determine the amount of information gained by selecting the attribute outlook to be a

root of the decision tree This attribute divides the training examples into three subsets

sunny-with five examples two of which belong to the class Play overcast-with four

examples all of which belong to the class Play and rain-with five examples three of

which belong to the class Play To determine info(11 the average information needed to

identify the class of an example in T there are 14 training examples and two decision classes

Nine of these examples belong to the class Play and five belong to the class Dont Play

Info(11 = - 914 10gl (914) - 514 10gl (514) = 094 bits

When using outlook to divide the training examples the information becomes

15

infO(T) = 514 -25 log2 (25) - 35 10g2 (35)

+ 414 -44 10g2 (44) - 04 log2 (04)

+ 514 -35 logl (35) - 25 logl (25) = 0694 bits

By substituting in equation 2-4 the gain of information results from using the attribute

outlook to split the training examples equal to 0246 The gain information for windy is

0048

Figure 2-3 shows a decision tree learned for this problem using the gain criterion The split

information for outlook is determined as follows

split info(T) = - 514 logl (514) - 414 log2 (414) - 514 log2 (514) = 1577 bits

The gain ratio for outlook = 02461577 =0156

16

Figure 2 -3 A decision tree learned using the gain criterion for selecting attributes

The C45 system handles discrete values as well as continuous values To handle an attribute

with continuous values C45 uses a threshold to transform the continuous domain into two

intervals In other words for each continuous attribute C45 generates two branches one

where the values of that attribute is greater than the determined threshold and the other if the

value is less than or equal to the threshold

Tree pruning in C45 is a process of replacing subtrees with small classification validity by

leaves The C45 system uses Laplace ratio for determining the error rate of different subtrees

This ratio is defined as (e+l) (n+2) where n is the number of the training examples and e is

the number of misclassified examples at a given leaf

222 Building Decision Trees Using Statistics-based Criteria

The Chi-square method for selecting attributes was used by Hart (1984) and Mingers (1989a)

in building decisionmiddot trees The method uses Chi-square statistics to measure the association

between two attributes When building decision trees the method is implemented such that it

determines the association between each attribute and the decision classes The attribute to be

selected is the one with greatest value

To determine the Chi-square value for an attribute consider aij is the number of examples in

class number i where the attribute A takes value numberj In other words aij is the frequency

of the combination of decision class number i and the attribute value number j The Chi-square

value for attribute A is given by

n m

Chi-square (A) =L L [ (aij - Eijl2 Eij ] (2shy

i=1 j=1

where n is the number of decision classes m is the number ofvalues of a given attribute Also

17

Eij = (TCi TVj ) T (2shy

8)

where T Ci and T Vj are the total number of examples belong to the decision class a and the total

number of examples where the attribute A takes value vj respectively T is the total number of

examples

Consider Quinlans example in Table 2-2 Table 2-3 shows the frequency of different

combination of values between the decision class and both the outlook and the Windy

attributes Table 2-4 shows the expected values TCi and T Vj of the frequencies in Table 2-3

of different attributes values for different decision classes

To determine the association value between the decision classes and both the attribute Windy

and the attribute Outlook the observed Chi-square values are

Chi-square (Windy Class) =[(3-39)239] + [(3-21)221] + [(6-51)232] + [(2-29)232]

=021 + 039 + 025 + 025 = 11

Chi-square (Outlook Class) = [(2-32)232] + [(4-26)226] + [(3-32)232] + [(3-18118]

+ [(0-14)214] + [(2-18)2 18] =045 + 075 + 02 + 08 + 14 + 002 = 362

18

Applying the same method to the other attributes the results will favor the attribute Outlook

Once that attribute is selected to be a node in the tree the remaining set of examples are divided

into subsets and the same process is repeated on each subset

Table 2-5 shows a summary of these criteria and their basic evaluation function

Table 2-5 Attribute selection criteria and their basic evaluation measure

Info Measure (IM) Gain Gshystatistics and Gain Ratio Chi-square

II

Entropy(S) = - L (freq(~ 5) 1151) logz (freq(CIt 5) 1151 ) I

G-Statistics =2N 1M (N-No of examples) n m

Chi-square (A B) = L L [(aij - ~j)21Eij ] i=l

223 Analysis of Attribute Selection Criteria

This subsection introduces briefly the analysis of different selection criteria which was done by

Mingers (1989a) Mingers compared six attribute selection criteria that are used in decision

tree programs These criteria are the Information Measure (IM) Chi-square G statistics Gin

index of diversity Marshal correction and Gain RaJio The overall results show that the Gain

Ratio criterion had the strongest xesults

In the flISt experiment Minger tested the six criteria on a problem with ambiguous examples

(ie examples may belong to more than one decision class) to observe how the selected criteria

evaluate the given attributes The problem has two decision classes and two attributes X and

Y It was assumed that attribute X is better for classifying the examples than attribute Y The

training examples were unevenly spread between the two values of X Attribute Y has three

values and the examples were spread randomly among them Table 2-6 (a and b) shows the

contingency tables for both attributes Table 2-7 shows a summary of the goodness of split

provided by the six criteria Mingers noted that the measures that are not based on information

19

theory give radiation (attribute X here) less weighL This may be because the zero in the fIrst

row of radiation has a greater influence in the log calculation In the case of using the Chishy

square criterion the value zero adds the maximum association between any two attributes

because the Chi-square value of a zero cell is the expected value of this cell

b)

Now let us demonstrate results from another experiment done by Mingers In this experiment

Mingers used four different data sets to generate decision trees for eleven different criteria In

the final results he compared the total number of nodes and the total error rate provided by

each criterion over all given problems Table 2-8 shows the final results for five selected

criteria only

20

U lgt ~middot results for comparing the total accuracy and size of decision trees different attribute selection criteria from four VVLU

This experiment was performed on four real world data sets These data are concerned with

profiles of BA Business Studies degree students recurrence of breast cancer classifying types

of Iris and recognizing LCD display digits The data was divided randomly 70 for training

and 30 for testing For more details see Mingers (1989a)

23 Learning Decision Structures

Considering the proposed defInition of decision structures given above two related lines of

research are described in this section Gaines (1994) and Kohavi (1994) proposed two

approaches for generating decision structures that share some of the earlier ideas by Imam and

Michalski (1993b)

In the first approach Brian Gaines introduces a method for transforming decision rules or

decision trees into exceptional decision structures The method builds an Exception Directed

Acyclic Graph (EDAG) from a set of rules or decision trees The method starts by assigning

either a rule say RO or a conclusion say CO or both to the root node If it assigns a conclusion

to the root node it places it on a temporary conclusion list Then it generates a new child node

and add to it a new rule say Rl The method evaluates the new rule Rl if it satisfies the

conclusion CO in its parent node it builds a new child and repeats the processes Otherwise if

the rule Rl does not satisfy the conclusion CO it changes the temporary memory with the new

conclusion Cl which is satisfied by the rule RO The same process is repeated until all rules

that have common conditions with the rule at the root are evaluated The method then creates a

21

new child node from the root and repeat the process until all rules are evaluated In the decision

structure nodes containing rules only represent common conditions to all of its children

The main disadvantages of this approach is that it requires discriminant rules to build such a

decision structure Also such a structure is more complex than the traditional decision trees

that are used for decision-making

Figure 2-4 shows an example of set of rules and their equivalent exceptional decision structure

The decision structure can be read as follows It is Safe except if xl=1 amp x2=1 amp x3=1 and

either x4=3 or x5=I then it is Lost except if x6=1 it is Safe except if x7=1 it is Lost

The second approach introduced by Ronny Kohavi learns decision structures from examples

using a bottom-up algorithm The method uses a greedy Hill-climbing algorithm for inducing

Oblivious read-Once Decision Graphs (HOODO) A read-once decision graph is defined as a

decision graph where each attribute occurs at most once along any computational path In other

words for each path from the root to the leaves of the decision structure attributes may occur

as a node once at most However there may be more than one node with the same attribute in

the decision graph Kohavi also defined a leveled decision graph in which the nodes are

partitioned into a sequence of pairwise disjoint sets such that outgoing edges from each level

terminate at the next level An oblivious decision graph is a decision graph where all nodes at

a given level are labeled by the same attribute

22

Safe lt [xl=2] Safe lt [x2=2] Safe lt [x3=2] Safe lt [x4=1] amp [x5=2] Safe lt [x4=1] amp [x5=3] Safe lt [x6=l] amp [x7=2] Safe lt [x6=1] amp [x7=3] Safe lt [x4=2] amp [x5=2] Safe lt [x4=2] amp [x5=3]

Lost lt [x6=2] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x6=3] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x5=1] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=2] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=3] amp [x3=1] amp [x2=I] amp [xl=l]

The algorithm starts by generating a leaf node for each decision class Then it applies a

nondeterministic method to select an attribute to build new level of the decision structure This

attribute is removed from the data and the data is divided into subsets each corresponding to a

complex combination of that attributes values For each subset the process is repeated until all

the examples of a given subset belong to one decision class For example if the selected

attribute say A has two values (0 and 1) and there are two decision classes (CO and CI) The

data is divided into two subsets the fIrst subset contains examples where A takes value 0 and

belong to class CO or takes value 1 and belong to class Cl The second subset of examples is

the set where A takes value 0 and belong to class CI or takes value 1 and belong to class CO

The number of nodes of the fIrst level (after the leave nodes) is expected to be less than or

equal to ka where k is the number of decision classes and n is the number of values of the

selected attribute and it can increase exponentially before that number is reduced

exponentially to one

It is easy for the reader to fIgure out some major disadvantages of such approach including

The average size of such decision structures is estimated to be very large especially when there

23

is no similarity (ie strong patterns) or logical relationship in the data The time used to learn

such decision structure is relatively very high comparing to systems for learning decision trees

from examples and fmally it could be better to search for attribute which reduces the number

of generated subsets of the data instead of nondeterministically selecting an attribute to build a

new level of the decision structure Kohavi provided a comparison between the C45 system

and his system that can be found in (Kohavi 1994) Later Kohavi and Li (1995) used a

deterministic method for attribute selection which minimizes the width of the penultimate level

of the graph

Table 2-9 shows a comparison between the proposed approach and those two approaches The

EDAG and HOODG systems are unreleased prototype systems

or Decision structures are easy to Decision structures are Decision structures are easy to understand difficult to read understand

CHAPTER 3 DESCRIPTION OF THE APPROACH

31 General Methodology

In the proposed approach the function of learning or discovery is separated from the function of

using the discovered knowledge for decision-making The frrst function is performed by an

inductive learning program that searches for knowledge relevant to a given class of decisions

and stores the learned knowledge in the form of decision rules The second function is performed

when there is a need for assigning a decision to new data points in the database (eg bull a

classification decision) by a program that transforms obtained knowledge into a decision

structure optimized according to the given decision-making situation

The Learning Task GivenA set of training examples describing the concept to be learned Learning goal which specifies the decision classes to be learned from the training examples A background knowledge to control the learning process Determine A concept description in a declarative form of knowledge (decision rules) that satisfies the learning goal

The Decision-making Task GivenA set of decision rules in a conjunctive form A description of the new decision-making situation (eg attributes costs and order preference importance or frequency of decision classes etc) One or more examples that needed to be tested under the given decisionshymaking situation A set of parameters to control the learning process Determine A decision structure that suits the given decision-making situation

The decision rules used here are learned by either the AQ1S (Michalski et al 1986) or AQ17

(Bloedorn et aI 1993) learning systems The reasons for selecting decision rules as a form of

23

24

declarative knowledge are that they do not impose any order on the evaluation of the attributes

and due to the lack of order constraints decision rules can be evaluated in many different

ways which increases the flexibility of adapting them to the different tasks of decision-making

(Bergadano et al 1990) Since the number of rules learned per class is typically much smaller

than the number of examples per class generating a decision structure from decision rules can

potentially be done on line

Such virtual decision structures can be tailored to any given decision-making situation The

needed decision rules have to be generated only once and then they can be used many times for

generating decision structures according to changing requirements of decision-making tasks The

method uses the AQDT-2 system (Imam amp Michalski 1994) for learning decision structures

from decision rules Decision structures represent a procedural form of knowledge which makes

them easy to implement but also harder to change Consequently decision structures can be quite

effective and useful as long as they are used in decision-making situations for which they are

optimized and the attributes specified by the decision structure can be measured without much

cost Figure 3-1 shows an architecture of the proposed methodology

Data

Decision -

- Learning knowledge from database The decision making process

Figure 3 -1 Architecture of the AQDT approach

25

It is assumed that the database is not static but is regularly updated A decision-making problem

arises when there is a case or a set of cases to which the system has to assign a decision based on

the knowledge discovered Each decision-making situation is defmed by a set of attribute-values

Some attribute-values may be missing or unknown A new decision structure is obtained such

that it suits the given decision-making problem The learned decision structure associates the

new set of cases with the proper decisions

32 A Brief Description of the AQIS and AQ17 Rule Learning Programs

The decision rules are generated from examples by an AQ-type inductive learning system

specifically by AQ15 (Michalski et albull 1986) or AQI7-DCI (Bloedorn at al 1993) The rest of

this subsection includes a brief description of the inductive learning systems AQ15 and AQI7

AQ15 learns decision rules for a given set of decision classes from examples of decisions using

the STAR methodology (Michalski 1983) The simplest algorithm based on this methodology

called AQ starts with a seed example of a given decision class and generates a set of the

most general conjunctive descriptions of the seed (alternative decision rules for the seed

example) Such a set is called the star of the seed example The algorithm selects from the star

a description that optimizes a criterion reflecting the needs of the problem domain If the

criterion is not defined the program uses a default criterion that selects the description that

covers the largest number of positive examples (to minimize the total number of rules needed)

and with the second priority that involves the smallest number of attributes (to minimize the

number of attributes needed for arriving at a decision)

If the selected description does not cover all examples of a given decision class a new seed is

selected from uncovered examples and the process continues until a complete class description

is generated The algorithm can work with few examples or with many examples and can

optimize the description according to a variety of easily-modifiable hypothesis quality criteria

26

The learned descriptions are represented in the form of a set of decision rules expressed in an

attributional logic calculus called variable-valued logic 1 or VLl (Michalski 1973) A

distinctive feature of this representation is that it employs in addition to standard logic operators

the internal disjunction operator (a disjunction of values of the same attribute in a condition) and

the range operator (to express conditions involving a range of discrete or continuous values)

These operators help to simplify rules involving multi valued discrete attributes the second

operator is also used for creating logical expressions involving continuous attributes

AQ15 can generate decision rules that represent either characteristic or discriminant concept

descriptions depending on the settings of its parameters (Michalski 1983) A characteristic

description states properties that are true for all objects in the concept The simplest

characteristic concept description is in the form of a single conjunctive rule (in general it can be

a set of such rules) The most desirable is the maximal characteristic description that is a rule

with the longest condition part ie stating as many common properties of objects of the given

class as can be determined A discriminant description states properties that discriminate a given

concept from a fixed set of other concepts The most desirable is the minimal discriminant

descriptions that is a rule with the slwnest condition part For example to distinguish a given

set of tables from a set of chairs one may only need to indicate that tables have large flat top

A characteristic description of the tables would include also properties such as have four legs

have no back have four corners etc Discriminant descriptions are usually much shorter than

characteristic descriptions

Another option provided in AQ15 controls the relationship among the generated descriptions

(rulesets or covers) of different decision classes In the IC (Intersecting Covers) mode

rulesets of different classes may logically intersect over areas of the description space in which

there are no training examples In the DC (Disjoint Covers) mode descriptions of different

classes are logically disjoint The DC mode descriptions are usually more complex both in the

number of rules and the number of conditions There is also a DL mode (a Decision List mode

27

also called VL modeM-for variable-valued logic mode) in which the program generates rule sets

that are linearly ordered To assign a decision to an example using such rulesets the program

evaluates them in order If ruleset is satisfied by the example then the decision is made

otherwise the program proceeds to the evaluation of the ruleset +1 In IC and DC modes

rule sets can be evaluated in any order

Alternatively the system can use rules from the AQ17-DCI program for Data-driven

Constructive Induction AQ17-DCI differs from AQ15 mainly in that it contains a module for

generating additional attributes These attributes are various logical or mathematical

combinations the original attributes The program generates a large number of potential new

attributes and seleets from them those most promising based on an attribute quality criterion

To illustrate the format of rules generated by AQ15 (or AQ17-DCI) an exemplary ruleset is

shown in Figure 3-2 The ruleset (that can be re-represented as a disjunctive normal form

expression) describes a voting record of Democratic Representatives in the US Congress

Each rule is a conjunction of elementary conditions Each condition expresses a simple relational

statement For example the condition [State = northeast v northwest] states that the attribute

State (of the Representative) should take the value northeast or northwest to satisfy the

condition

Rl [Gas_concban = yes] amp [Soc_see_cut = no v not registered] R2 [Draft = Yes v not registered]amp[AlaslaLparks = yes v not registered] amp

[Food_stamp_cap = no] amp [State = northeast v northwest] R3 [Chrysler =yes v not registered] amp [Income =low] R4 [Education =yes] amp [Occupation = yes]

Figure 3-2 A ruleset generated by AQ15 for the concept Voting pattern of Democratic Representatives

The above rules were generated from examples of the voting records For illustration below is

an example of a voting record by a Democratic representative

Draft registration=no Ban of aid to Nicaragua=no Cut expenditure on mx missiles=yes Federal subsidy to nuclear power stations=yes Subsidy to national parks

28

in Alaska=yes Fair housing bill=yes Limit on Pac contributions=yes Limit onood stamp program=no Federal help to education=no StateFrom=north east State Population=large Occupation =unknown Cut in social security spending=no Federal help to Chrysler corp=not registered

By expressing elementary statements in the example as conditions and linking conditions by

conjunction the examples can be re-expressed as decision rules Thus decision rules and

examples formally differ only in the degree of generality

33 Generating Decision Structurestrees from Decision Rules

This section describes the AQDT-1 system for learning decision structures (trees) from decision

rules (Imam amp Michalski 1993ab) Also a description of the AQDT-2 method for learning taskshy

oriented decision structure from decision rules is included and fmally the methodology is

illustrated by two examples

Methods for learning decision trees from examples have been very popular in machine learning

due to their simplicity Decision trees built this way can be quite efficient as long as they are

used in decision-making situations for which they are optimized and these situations remain

relatively stable Problems arise when these situations significantly change and the assumptions

under which the tree was built do not hold anymore For example in some situations it may be

difficult to determine the value of the attribute assigned to some node One would like to avoid

measuring this attribute and still be able to classify the example if this is potentially possible

(Quinlan 1990) If the cost of measuring various attributes changes it is desirable to restructure

the tree so that the inexpensive attributes are evaluated f11st A tree restructuring is also

desirable if there is a significant change in the frequency of occurrence of examples from

different classes A restructuring of a decision tree to suit the above requirements is however

difficult to do The reason for this is that decision trees are a form of decision structure

representation and imposes constraints on the evaluation order of the attributes that are not

logically necessary

29

One problem in developing a method for generating decision structures from decision rules is to

design an attribute selection criterion that is based on the properties of the rules rather than of

the training examples A decision rule normally describes a number of possible examples Only

some of them are examples that have actually been observed ie training examples An attribute

selection criterion is needed to analyze the role of each attribute in the rules It cannot be based

on counting the numbers of training examples covered by each attribute-value and the

frequency of decision classes in the training examples as is done in learning decision trees from

examples because the training examples are assumed to be unavailable

Another problem in learning decision trees from decision rules stems from the fact that decision

rules constitute a more powerful knowledge representation than decision trees They can directly

represent a description in an arbitrary disjunctive normal form while decision trees can represent

directly only descriptions in the disjoint disjunctive normal form In such descriptions all

conjunctions are mutually logically disjoint Therefore when transforming a set of arbitrary

decision rules into a decision tree one faces an additional problem of handling logically

intersecting rules

The solution to both problems (attribute selection and logically intersected rules) in the AQDT-2

system is based on the earlier work by Michalski (1978) which introduced a general method for

generating decision trees from decision rules The method aimed at producing decision trees

with the minimum number of nodes or the minimum cost (where the cost was defined as the

total cost of classifying unknown examples given the cost of measuring individual attributes and

the expected probability distribution of examples of different decision classes) More

explanations are provided in the following section

331 Tbe AQDT-2attribute selection metbod

This section describes the AQDT -2 method for building a decision structure from decision rules

The method for building a single-parent decision structure is similar to that used in standard

30

methods of building a decision tree from examples The major difference is that it assigns tests

(attributes) to the nodes using criteria based on the properties of the decision rules (this includes

statistics about the examples covered by each rule in the case of learning rules from examples)

rather than statistics characterizing the frequency of training examples per decision classes per

attribute-values or per conjunctions of both Other differences are that the branches may be

assigned an internal disjunction of values (not only a single value as in a typical decision tree)

and leaves may be assigned a set of alternative decisions with probabilities Also the tests can be

attributes or names standing for logical or mathematical expressions that involve several

attributes or variables In the following we use the terms test and attribute interchangeably

(to distinguish between an attribute and a name standing for an expression the latter is called a

constructed attribute)

At each step the method chooses the test from an available set of tests that has the highest utility

(see below) for the given set of decision rules This test is assigned to the node The branches

stemming from this node are assigned test values or disjoint groups of values (in the form of

logical disjunction if such occur in the rules subsumed groups of values are removed) Each

branch is associated with a reduced set of rules determined by removing conditions in which the

selected attribute assumes value(s) assigned to this branch If all rules in the reduced ruleset

indicate the same decision class a leaf node is created and assigned this decision class The

process continues until all nodes are leaf nodes If it is not possible to reduce further the rule set

because some attribute is declared as unavailable (infmite cost) then the leaf is assigned a set of

candidate decisions with associated probabilities (see Sec 42)

The test (attribute) utility is a combination of one or more of the following elementary criteria 1)

cost which indicates the cost of using each attribute for making decision 2) disjointness which

captures the effectiveness of the test in discriminating among decision rules for different decision

classes 3) importance which determines the importance of a test in the rules 4) value

31

distribution which characterizes the distribution of the test importance over its of values and 5)

dominance which measures the test presence in the rules These criteria are dermed below

~ The cost of a test expresses the effort or cost needed to measure or apply the test

Djsjointness The disjointness of a test is defined as the sum of the class disjointness-the

disjointness of the test for each decision class Suppose decision classes are Cl C2 bull Cm and

decision rule sets for these classes have been determined Given test A let V 1 V 2 bull V m denote

sets of the values (outcomes) of A that are present in rule sets for classes C 1 C2bullbull Cm

respectively If a ruleset for some class say Ct contains a rule that does not involve test A than

Vt is the set of all possible values of A (the domain of A)

Definition 3 -1 The degree ofclass disjointness D(A Ci ) of test A for the ruleset ofclass Ci is

the sum of the degrees of disjointness D(A Cit Cj) between the ruleset for Ci and rulesets for

Cj j=l 2 m j J L The degree of disjointness between the ruleset for Ci and the ruleset for Cj is

dermed by

if Vi Vj ifVj gtVj (3-1) ifVinVj -tljgt amp -tVi amp -tVj ifVinVj=1jgt

where cjgt denotes the empty set Note that exchanging the second and third conditions of equation

(3-1) may seem to be an improved criteria However it does not clearly distinguish between both

cases (Le for both situations the disjointness will be similar) The current equation is better

because it gives higher scores to attributes that classify different subsets of the two decision

classes than attributes that classify only a subset of one decision class

Definition 3-2 The disjointness of the test A for evaluating a given set of decision rules is the

sum of the degrees of class disjointness of each decision class

32

m m Disjointness(A) = L D(A cn where D(A cn = L D(A Ci Cj) (3-2)

i=l i=l i~j

The disjointness of a test ranges from 0 when the test values in rulesets of different classes are

all the same to 3m(m-l) when every rule set of a given class contains a different set of the test

values If two tests have the same disjointness value the attribute to be selected is the one with

less number of values

Definition The A vera~ Number of Tests (ANT) required to make decision from a decision tree

is dermed as the average number of tests (attributes) to be examined from the root of the tree to

any leaf node in order to reach a decision

Definition A decision structure is a one-node-per-level decision structure if at each level there is

only one node and zero or more leaves

Such decision structure can be generated by combining together all branches that associated with

each one a set of decision rules belongs to more than one decision class

Theorem 1 Consider learning one-node-level decision structure for a database with two

decision classes The disjointness criterion ranks first attributes that add minimum number of

tests to the decision tree

Proof Consider all possible distributions of an attributes values within two decision classes Ci

and Cj There are three cases 1) subset (same as super set) 2) non-empty intersection but not

subset 3) no intersection Figure 3-3 shows all possible distributions (the case of having the

same set of values in both classes is trivial one) Consider that branches leading to one subset

with the same decision class are combined into one branch In the first case there will be two

branches only The first leads to a leaf node and the other leads to an intermediate node where

another attribute to be selected The minimum ANT in this case is 53 In the second case three

branches should be created Two branches leads to a leaf node where all values at each branch

belong to only one and different decision class The third branch leads to an intermediate node

33

where another attribute should be selected furtur classifies the decision classes The minimum

ANT in this case is 64 In the third case only two branches will be generated where each leads

to leaf node with different decision class In this case the minimum ANT is 1

Figure 3-4 shows decision trees equivalent to selecting the corresponding attribute Note that in

case of having more than one attribute-value at branches lead to leaves belonging to one decision

class they will be combined into one branch in the decision structure The symbol 1 means

that an attribute is needed to classify the two decision classes In such cases there will be at least

two additional paths

D(A Ci) = 0 D(A CJ) = 1 D(A Ci) =2 D(A Cj) = 2 D(A Ci) =3 D(A Cj) =3

Figure 3-3 Yen Diagrams of Possible combinations of attribute values in two decision classes

The average number of tests required for making a decision in each possible case is determined

in Figure 3-4 It is clear that in the case of two decision classes the disjointness criterion ranks

highly attributes that reduces the average number of tests required for decision-making The

theorem can be proved in the general case

i ~ i Cj C1middot bull Ci Cj

ANT=32middotANT=53 ANT=l

1 means at least one attribute is needed to complete the decision tree Figure 3-4 Decision trees corresponding the Yen diagrams inFigure 3-3

34

Theorem 2 The attribute with the highest non~zero disjointness is the best attribute to classify

the decision classes

Proof Suppose that the number of decision classes is n Assume also that there are two attributes

A and B where D(A) lt D(B) Since the disjointness criterion considers the mutual relationship

between any two decision classes this means that there are more decision classes where D(A

Ci) lt D(B Ci) than those where D(A Ci) gt D(B Ci) (ignoring classes with equal disjointness

for bCth attributes) Hence there are more decision classes where D(A Ci Cj) lt D(B Ci Cj)

than those where D(A Ci Cj) gt D(B Cit Cj) Let us frrst prove that if D(A Ci Cj) lt D(B Cit

Cj) then B classifies the decision classes better than A

For each pair of decision classes Ci and Cj the possible values of the disjointness of any attribute

are 0 14 or 6 For all positive values ofD(B)= 14 or 6 it is clear that attribute B should have

smaller ANT than attribute A which has lower disjointness This means that if D(A Cit Cj) lt

D(B Cit Cj) then attribute B can classify both decision classes better than attribute A

Similarly B is better for classifying more pairs of decision classes than A This implies that B is

a better classifier than A

Importance The second elementary criterion the importance of a test is based on the

importance score (IS) introduced in (Imam Michalski amp Kerschberg 1993) In the obtained

rules each test is assigned a score that represents the total number of training examples that

are covered by the rules involving this test Decision rules learned by an AQ learning program

are accompanied with information on their strength Rule strength is characterized by t-weight

and u~weight The t-weight (total-weight) of a rule for some class is the number of examples of

that class covered by the rule The importance score of a test is the aggregation of the totalshy

weights of all rules that contain that test in their condition part Given a set of decision rules for

m decision classes Cl Cm and involving n tests AlbullAn and the number of rules associated

with class Ci is denoted by tlrit The importance score is defmed as follow

35

Definition 3 -3 The importance score IS(Aj) of the test A j is detennined by

m IS(Aj)= L IS(Aj Ci) (3-31)

i=l

where ri IS(Aj Ci) = L Rik(Aj) (3-32)

k=l

and Rik the weight of a test Aj in the rule Rk of class Ci is given by

t - weight if A j belongs to rule Rik Rik (Amiddot) = (3-4)J 0 otherwise

where i=I n ik=I ri j=I m

The importance score method has been separately compared as a feature selection method with

a genetic algorithm-based method (Imam amp Vafaie 1994) The importance score method

produced an equal or higher accuracy on three real-world problems than those reported by the

GA method while selecting fewer attributes In addition the IS method was significantly faster

than the GA

yalue distribution The third elementary criterion value distribution concerns the number of

legal values of tests Given two tests with equal importance scores this criterion prefers the test

with the smaller number of legal values Experiments have shown that this criterion is especially

useful when using discriminant decision rules

Definition 34 A value distribution VD(Aj) of a test A j is defined by

VD(Aj) = IS(Aj) I Vj (3-5)

where v is the number of legal values of Aj

Dominance The fourth elementary criterion dominance prefers tests that appear in large

numbers of rules as this indicates their high relevance for discriminating among the rulesets of

given decision classes Since some conditions in the rules have values linked by internal

disjunction counting such rules directly would not reflect properly their relevance Therefore

36

for computing the dominance the rules are counted as if they were converted to rules that do not

have internal disjunction Such a conversion is done by multiplying out the condition parts of

the rules containing internal disjunction For example the condition part [x3=1 v 3]amp[x4=1] is

multiplied out to two rules with condition parts [x3=1]amp[x4=1] and [x3=3]amp[x4=I]

The above criteria are combined into one general test measure using the lexicographic

evaluation functional with tolerances (LEF) (Michalski 1973) LEF is a list of some or all of

the above elementary criteria each associated with a Ittolerance threshold It in percentage The

criteria are applied to tests in the order defined by LEF A test passes to the next criterion only if

it scores on the previous criterion within the range defined by the tolerance (from the top value)

The default LEF is

ltCost d Disjointness t2 Importance t3 Value distr t4 Dominance 0gt (3-6)

where d a t3 t4 and t5 are tolerance thresholds (in percentages) their default values are O

The default value of the cost of each test is 1

The above LEF ranks attributes this way First attributes are evaluated on the basis of their cost

If two or more attributes have the same cost they ranked by their degree of disjointness If two

or more attributes still share the same top score or their scores differ less than the assumed

tolerance threshold t2 the method evaluates these attributes using the second (Importance)

criterion If again two or more attributes share the same top score or their scores differ less than

the tolerance threshold t3 then the third criterion normalized IS is used then similarly the

fourth criterion (dominance) If there is still a tie the method selects the best attribute

randomly

If there is a non-uniform frequency distribution of examples of different classes then the

selection criterion uses a modified definition of the disjointness Namely the previously defmed

37

disjointness for each class is multiplied by the frequency of the class occurrence The class

occurrence is the expected number of future examples that are to be classified to a given class

m

Disjointness(A) = L D(A CO Frq(Ci) (3-7) i=l

where Frq(Cn is the expected frequency of examples of class Cit and it is assumed to be given

by the user The attribute ranking criteria in this case is defined by the LEF

ltCost d Disjointness t2 Importance 13 Normalized-IS 14 Dominance 15gt (3-8)

where the Cost denotes the evaluation cost of an attribute and is to be minimized while other

elementary criteria are treated the same way as in the default version

332 The AQDT-2 algorithm

The AQDT-2 algorithm constructs a decision structure from decision rules by recursively

selecting at each step the best test according to the ranking criteria described above and

assigning it to the new node The process stops when the algorithm creates terminal branches that

are assigned decision classes To facilitate such a process the system creates a special data

structure for each concept description (ruleset) This structure has fields such as the number of

rules the number of decision classes and the number of attributes present in the rules A set of

pointers connect this data structure to set of data structures each represents one decision class

The decision class structure contains fields with information on the number of rules belong to

that class the frequency of the decision class etc It is also connected to a set of data structures

representing the decision rules within each decision class The system creates independently a set

of data structures each is corresponding to one attribute Each attribute description contains the

attributes name domain type the number of legal values a list of the values the number of

rules that contain that attribute and values of that attribute for each rule The attributes are

arranged in an array in a lexicographic order flISt in the descending order of the number of rules

38

that contain that attribute and second in the ascending order of the number of the attributes

legal values

The system can work in two modes In the standard mode the system generates standard

decision trees in which each branch has a specific attribute-value assigned In the compact

mode the system builds a decision structure that may contain

A) or branches Le branches assigned an internal disjunction of attribute-values whenever it

leads to simpler structures For example if a node assigned attribute A has a branch marked by

values 1 v 2 then the control passes along this branch whenever A takes value 1 or 2 The

program creates or branches on the basis of the analysis of the value sets Vi while computing

the degree of attribute disjointness

B) nodes that are assigned derived attributes that is attributes that are certain logical or

mathematical combinations of the original attributes To produce decision structures with

derived attributes the input decision rules are generated by AQ17 (rather than AQI5) The

AQ17 rules may contain conditions involving attributes constructed by the program rather than

those originally given

To generate decision structures from rules the AQDT-2 method prefers either characteristic or

discriminant disjoint rule descriptions (given by an expert or learned by a system) Disjoint rules

are more suitable for building decision structures Assume that the description of each class is in

the form of a ruleset and that this set is the initial ruleset context The AQDT algorithm is

The AQDT2 Algorithm

Step 1 Evaluate each attribute occurring in the ruleset context using the LEF attribute ranking

measure Select the highest ranked attribute Let A represents this highest-ranked

attribute

Step 2 Create a node of the tree (initially the root afterwards a node attached to a branch) and

assign to it the attribute A In standard mode create as many branches from the node as

39

there are legal values of the attribute A and assign these values to the branches In

compact mode (decision structures) create as many branches as there are disjoint value

sets of this attribute in the decision rules and assign these sets to the branches

Step 3 For each branch associate with it a group of rules from the ruleset context that contain a

condition satisfied by the value(s) assigned to this branch For example if a branch is

assigned values i of attribute A then associate with it all rules containing condition [A=

i v ] If a branch is assigned values i v j then associate with it all rules containing

condition [A= i v j v ] Remove from the rules these conditions If there are rules in

the ruleset context that do not contain attribute A add these rules to all rule groups

associated with the branches stemming from the node assigned attribute A (This step is

justified by the consensus law [x=1] == [x=1] amp [y=a] v [x=1] amp [y=b] assuming

that a and b are the only legal values of y) All rules associated with the given branch

constitute a ruleset context for this branch

Step 4 If all the rules in a ruleset context for some branch belong to the same class create a leaf

node and assign to it that class If all branches of the trees have leaf nodes stop

Otherwise repeat steps 1 to 4 for each branch that has no leaf

To select an attribute to be a node in the decision tree (step 1 and 2 of the algorithm) the

algorithm performs two independent iterations In the first iteration it parses through all decision

rules and determine information about each attributes This information includes the importance

score of each attribute the number of rules containing a given attribute the disjoint value sets of

each attribute and the attribute values used in describing each decision class The second

iteration is only performed if the disjointness criterion is ranked first in the LEF function The

second iteration evaluates the attributes disjointness for each decision class against the other

decision classes

40

To determine the complexity of this process suppose that the maximum number of conditions

fonned by a single attribute value in one rule is s (where s ~ n the number of attributes) and r is

the total number of decision rules (in all decision classes)

m

r=LRt (m is the number of decision classes) 1=1

where Ri is the number of rules in decision class Ci bull The complexity of the frrst iteration can be

determined as

Cmpx(Itcl) =O(r s)

In the second iteration the disjointness is calculated between the decision classes and all

attributes The complexity of the second iteration can be given by

Cmpx(Itc2) = O(n m)

Assume that at each node 1 is the maximum of the number of decision classes to be classified at

this node and the number of rules associated with this node

l=max mr (3-9)

Since these two iteration are independent of each other the complexity of the AQDT algorithm

for building one node in the decision tree say Node Complexity NC(AQDT) is given by

NC(AQDT) = 0(1 n)

Usually I equals to the number of rules associated with the given node Thus the AQDT

complexity for building one node is a function of the number of attributes multiplied by the

number of rules associated with this node

At each level of the decision tree these two iterations are repeated for each non-leaf node The

Level Complexity of the AQDT algorithm LC(AQDT) can be given by

LC(AQDT) lt 0(1 n)

Which is less than the complexity of generating the root of the decision tree To explain this

consider the maximum possible number of non-leaf nodes at one level is half the number of the

initial decision rules (integer of rll) Figure 3-5-a shows an example of such situation where at

41

the lowest level each node classifies only two rules each belongs to different decision class This

decision tree could be a multimiddotvalues tree However the number of non-leaf nodes at each level

will be twice or more than the number of nonmiddot leaf nodes at the previous level Considering the

level complexity of the AQDT algorithm equal to (1 s 0) where 0 is the number of non-leaf

nodes at the given level In such cases it is either (1 0 S r) or (1 s lt r) In Figure 3middot5middota the

complexity of the AQDT algorithm at any lower level is given by

LC(AQDT) =0(2 s r(2) lt O(n I) =NC(AQDT)

a) per one level b) per one path

Figure 3 -5 Decision trees showing the maximum number of none leaf nodes

Note also that after selecting an attribute to be the root of the decision structure this attribute

and all conditions containing that attribute are removed from the data structure of the algorithm

Also if a leaf node is generated all rules belonging to the corresponding branch will not be

tested again

Since the disjointness criterion selects the attribute which minimizes the average number of tests

ANT it is true that the AQDT algorithm generates decision trees with the least number of levels

The number of levels per a decision tree is supposed to be less than or equal to the minimum of

both the number of attributes and the number of rules Consider k as the number of levels in a

given decision tree

k S min mr (3-10)

There are two cases represent the most complex situations Figure 3-5-a and 3-5-b In the fIrst

case where the decision rules were divided evenly the number of levels will be a function of the

42

logarithm of the number of rules In such case the complexity of the AQDT algorithm for

generating a decision tree from a set of decision rules is given by

Complexity(AQDT) = 0(1 n log r) (3-11)

The other situation is when the generated decision tree has the maximum number of levels The

maximum possible number of levels per a decision tree equals to one less than the number of

decision rules Figure 3-5-b Using the disjointness criterion it is not likely to get such a decision

tree because it has the maximum average number of test (ANT) that can be determined from the

same set of nodes and leaves However such a decision tree can be generated if the number of

decision classes is one less thanthe number of attributes In such case any disjQint decision rules

should have a maximum length that is less or equal to the floor of the logarithm of the number of

attributes Thus the level complexity of this decision tree is estimated as

LC(AQDT) =0(1 log n)

The maximum number of levels in such decision tree is k-l Thus the complexity of the AQDT

algorithm in such cases is given by

Complexity(AQDT) = 0(1 k log n) (3-12)

Since k is the minimum of n and r then n is the maximum of k and n Also since in (3-12)

r=n+1 then log r is greater than log n From 3-10 3-11 and 3-12 the complexity of the AQDT

algorithm is determined by

Cmplx(AQDT) = OCr k log 1) (3-13)

333 An example illustrating the algorithm

The following simple example illustrates how AQDT is used in selecting the optimal set of

testing resources for testing a software Suppose there are three tools for testing a software 1)

modeling (Tl) 2) checklist (T2) and 3) par_simul (T3) Also assume that there are four

different factors that affect the selection of any tool 1) the cost of using the tool (xl) 2) the

metrics that support the tool (x2) 3) the best phase for applying the tool (x3) and 4) type of tool

43

(automated semi-automated or manual) (x4) Table 3-1 shows these attributes and their possible

values

Suppose that the domain expert provided a set of rules to be used in testing a software Figure 3shy

6 shows a sample of these rules in AQ15c formats

Table 3-1 The available tools and the factors that affect the prclCes

Tl lt= [xl=2] amp [x2=2] v [xl=3] amp [x3=1 v 3] amp [x4=1] T2 lt= [xl=l v 2] amp [x2=3 v 4] v [xl=3] amp [x3=1 v 2] amp [x4=2] T3 lt= [xl=l] amp [x2=1] v [xl=4] amp [x3=2 v 3] amp [x4=3]

Figure 3-6 Decision rules for selecting the best tool for testing software

These rules can be interpreted as

Rule 1 Use the fltst tool for testing if you need average cost and the tool is

supported by the requirement metric

Rule 2 Use the fIrst tool for testing if you can afford high cost for testing either

in the requirement or the analysis phases and you need an automated tool

Rule 3 Use the second tool for testing if the cost limit is low or average and the

tool is supported by the system usage metric or the intractability metric

Rule 4 Use the second tool for testing if you can afford high cost for testing

either in the requirement or the design phases and you need a manual tool

Rule 5 Use the third tool for testing if the you are limited to low cost and the

tool is supported by the error rate metric

Rule 6 Use the third tool for testing if you can afford very-high cost for testing

either in the requirement or the system usage phases and you need a semishy

automated tool

44

Table 3-2 presents information on these rules and the disjointness values for all attributes For

each class the row marked Values lists values occurring in the ruleset for this class For

evaluating the disjointness of an attribute say A each rule in the ruleset above that does not

contain attribute A is characterized as having an additional condition [A= a vb ] where a b

are all legal values of A

The row Class disjointness specifies the class disjointness for each attribute The attribute xl

has the highest disjointness (11) and is assigned to the root of the tree For simplicity assume

the tolerances for each elementary criterion equal O

From the rules in Figure 3-6 we can also determine disjoint groupings of attribute-values used

for the compact mode of the algorithm This done as follows 1) determine for each attribute the

sets of values that the attribute takes in individual decision rules and remove those value sets

that subsume other value sets The remaining value sets are assigned to branches stemming from

the node marked by the given attribute For example x 1 has the following value sets in the

individual decision rules 2 3 I 2 I and 4 (Figure 3-6) Value set I 2 is

removed as it subsumes 2 and I In this case branches are assigned individual values of the

domain of x1 For attribute x2 the value sets are I 2 34 and I 2 3 4 In this case

branches are assigned value sets I 2 and 34

45

Attribute Xl ranks highest (as it is the highest disjointness) and is assigned to the root of the

tree Four branches are created each one corresponding to one of XIs possible values Since all

rules containing [x1=4] belong to class TI the branch marked by 4 is ended by a leafTI Rules

containing other values of X1 belong to more than one class This process is repeated for each

subset of rules until the decision tree is completed Figure 3-7 shows a decision structure learned

by AQDT-2 from the rules in Figure 3-6 The decision structure in Figure 3-7 can be used in

making decisions on which tools can be used for testing a given software

xl

Complexity No of nodes 4 No of leaves 7

Figure 3-7 A decision structure learned for classifying so tware testing tools

Figure 3-8-a shows the diagrammatic visualization of the decision rules and Figure 3-8-b shows

the visualization of the derived decision tree Each diagram in Figure 3-8 consists of cells

representing one combination of attribute-values Attributes and their legal values are shown on

scales surrounding the diagram (eg the horizontal scale for xl shows values 1 2 3 and 4)

Rules are represented by collections of cells in the intersection of the rows and columns

corresponding to the conditions in the rules

The shaded areas correspond to decision rules Rules of the same class have the same type of

shading Empty cells correspond to combination of attribute-values not assigned to any class For

illustration collections of cells corresponding to some of the initial rules are marked R 11 R 21

R31 and R32 Rll denotes the IlfSt rule of Class T1 ie [x1=2] amp [x2=2] R21 denotes the

IlfSt rule of class T2 ie [xl= 1 v 2] amp [x2= 3 v 4] R31 the first rule of class TI ie [x1=1]

46

amp [x2 =1] and denotes R32 denotes the second rule of class T3 Le [xl=4] amp [x3=2 v 3] amp

[x4=3]

a) Decision rules b) Derived decision tree

Comparing diagrams in Figure 3-8-a and 3-8-b AQDT-2 decision tree represents a slightly more

general description of Concepts Tl TI and T3 than the original rules Let us assume that it is

very costly to know which metrics suppon the required tools In other words suppose that we

would like to select the best tools independently of the metric they suppon (this is indicated to

AQDT-l by assigning very high cost to attribute x2) The algorithm again assigns attribute xl to

the root of the tree When xl takes value 1 or 2 it is impossible to assign a specific decision

without measuring attribute x2 For the value 1 of xl the recommended tool can be either TI or

T3 (see the diagram in Figure 3-9-a) and for the value 2 of xl the recommended tool can be

either Tl or T3 However for the value 3 of xl one can make a specific decision after

measuring attribute x4 For the value 1 of x4 tool is Tl and for the value 2 of x4 the

recommended tool is TI Figure 3-9-b shows another decision tree where the type of the tool

was ignored from the data Those decision trees are called indeterminate because some of their

leaves are assigned a disjunction of two or more class names

47

xl

xl

1 12

a) Ignoring the supporting metric b) Ignoring the type of the tooL Figure 3 -9 Decision trees learned ignoring the support metric and the type of the testing tool

It is clear that the given set of rules depend highly on amount of money can be spent Let us now

suppose that the cost attribute xl which was determined as the highest ranked attribute cannot

be measured The algorithm selected x4 to be the root of the new decision tree After continuing

the algorithm the tree in Figure 3-10 is obtainedThe nodes that are assigned one class indicate

situations in which it is possible to make a specific decision without knowing the value of

attribute xl

x4

Figure 3-10 A decision tree learned without the cost attribute

34 Tailoring Decision Structures to the Decision-making situation

Decision structures are among the simplest structures for organizing a decision-making process

A decision structure specifies explicitly the order in which attributes of an object or situation

need to be evaluated in the process of determining a decision A standard way to generate a

48

decision structure is to learn it from examples of decisions Such a process usually aims at

obtaining a decision structure that has the highest prediction accuracy that is the highest rate of

assigning correct decisions to given situations There can usually be a large number of logically

equivalent decision structures (Michalski 1990) As such they may have the same predictive

accuracy but differ in the way they organize the decision process and thus may differ in the cost

of arriving at a decision To minimize the average decision cost one needs to take into

consideration the distribution of the costs of attribute evaluation and the frequency of different

decisions This report presents an approach to building such Task-oriented decision structures

which advocates that they are built not from examples but rather from decision rules Decision

rules are learned from examples using the AQ15 or AQ17 inductive learning program or are

specified by an expert An efficient algorithm and a new system AQDT-2 that transforms

decision rules to task-oriented decision structures The system is illustrated by applying it to the

problem of learning decision structures in the area of construction engineering (for determining

the best wind bracing for tall buildings) In the experiments AQDT-2 outperformed all other

programs applied to the same data

Decision-making situations can vary in several respects In some situations complete

information about a data item is available (ie values of all attributes are specified) in others the

information may be incomplete To reflect such differences the user specifies a set of parameters

that control the process of creating a decision structure from decision rules AQDT-2 provides

several features for handling different decision-making problems 1) generating a decision

structure that tries to avoid unavailable or costly attributes (tests) 2) generating unknown

leaves in situations where there is insufficient information for generating a complete decision

structure 3) providing the user with the most likely decision when performing a required test is

impossible 4) providing alternative decisions with an estimate of the likelihood of their

correctness when the needed information can not be provided

49

341 Learning Costmiddot Dependent Decision Structures

As described in Sec 2 the LEF criterion enables the system to take into consideration the cost of

measuring tests (attributes) in developing a decision structure In the default setting of LEF the

tests cost is the [lIst criterion and its tolerance is O This means that only the least expensive

attributes pass to the next step of evaluation involving other elementary criteria If an attribute

has high cost or is impossible to measure (has an infinite cost) the LEF chooses another

cheaper attribute ifpossible

342 Assigning Decision Under Insufficient Information

In decision-making situations in which one or more attributes cannot be measured the system

may not be able to assign a definite decision for some cases If no more information can be

obtained but a decision has to be made it is useful to know the probability distribution for

different candidate decisions (Smyth Goodman amp Higgins 1990) The most probable decision is

then chosen The probability distribution can be estimated from the class frequency at the given

node Let us consider a node in the structure connected to the root by the sequence bl b2 bkof

attribute-values Let P(Ci I bl b2 bk) denote the conditional probability of class Ci i=I2 m

at that node given that an example to be classified has attribute-values assigned to branches bl

b2 bull bk in the decision structure Using Bayesian formula we have

P(Ci I bl~ bull bk) = P(Ci) P(bl bk I Ci) P(bl bk) (3-9)

where P(Ci) the apriori probability of class Ci P(bl bk I Ci) is the probability that the

example has attribute-values blbull bk given that it represents decision class Ci To approximate

these probabilities let us suppose that Wi is the number of training examples of Class Ci that

passed the tests leading to this node and twi - the total number of training examples of Class

Ci Let us use these numbers to estimate the probabilities in (9) Assuming that the apriori class

probabilities correspond to the frequency of training examples from different classes we have

50

m P(Ci) = twi IL twj (3-10)

j=l

P(b1 bkl Ci) = Wi I twi (3-11)

m m P(b1 bk) =L Wj IL twj (3-12)

j=l j=1

By substituting (1O) (11) and (12) in (9) we obtain

m P(Ci I b1 bk) = wilL Wj (3-13)

j=l

A related method for handling the problem of unavailability of an attribute is described by

Quinlan (1990) Quinlans method however assigns probabilities to the most probable decision

at the node associated with such an attribute The method does not restructure the decision tree

appropriately to fit the given decision-making situation (in this case to avoid measuring xl)

AQDT -2 assigns probabilities o~ly after it first created a decision structure that suits the given

decision-making si~ation An example is presented in section 42

343 Coping with noise in training data

The proposed methodology can be easily extended to handle the problem of learning from noisy

training data This is done based on ideas of rule truncation described in (Niblett amp Bratko 1986

Bergadanoet al 1992 Cestnik amp Karalic 1991) When noise is expected in the training data

the decision rules with t-weight below a certain threshold (reflecting the expected noise level) are

removed The rule truncation method seems to result in more accurate decision structures than

decision tree pruning method because truncation decisions are solely based on the importance of

the given rule or condition for the decision-making regardless of their evaluation order (unlike

the decision tree pruning which can only prunes attributes within a subtree and thus cannot

freely chose attributes to prune) Examples are presented in section 4

ix

4-5 The predictive accuracy of AQ15c and AQDT-2 for the MONK-2 problem 77

4-6 The predictive accuracy of AQ15c and AQDT-2 for the MONK-3 problem 81

4-7 The set of attributes and their values used in the trains problem 86

4-8 A tabular snmmary of the predictive accuracy of decision trees obtained

by AQDT -2 and C45 for the congressional voting data 88

4-9 Summary of the best parameter settings for the first subfunction of the approach

with different data characteristics 89

4-10 Summary of the performance of AQDT-2 and C45 on different problems 90

x

LIST OF FIGURES

No TITLE Page

2-1 An example to illustrate how attributes break rules 8

2-2 A decision tree learned from the decision table in Table 2-1 10

2-3 A decision tree learned using the gain criterion for selecting attributes 15

2-4 Decision rules and their Exception Directed Acyclic Graph (EDAG) 21

3-1 Architecture of the AQDT approach 24

3-2 A ruleset generated by AQ15 for the concept Voting pattern of

Democratic Representatives 27

3-3 Ven Diagrams ofPossible combinations of attribute values in two decision classes 33

3-4 Decision trees corresponding the Ven diagrams in Figure 3-3 33

3-5 Decision trees showing the maximum number of none leaf nodes 41

3-6 Decision rules for selecting the best tool for testing a software 43

3-7 A decision structure learned for classifying software testing tools 45

3-8 A diagrammatic visualization of the decision rules and the derived decision tree 46

3-9 Decision trees learned ignoring the support metric and the type of the testing tool 47

3-10 A decision tree learned without the cost attribute 47

3-11 Decision structures learned by AQDT-2 using different criteria 55

3-12 The Imams example Example where learning decision structures (trees)

from rules is better than learning them from examples 56

3-13 An example where decision rules are simpler than decision trees 57

4-1 Design of a complete experiment 59

Xl

4-2 Decision rules determined by AQ15c from the wind bracing data 61

4-3 A decision tree learned by C45 for the wind bracing data 63

4-4 A decision structure learned from AQ15c wind bracing rules 64

4-5 A decision structure that does not contain attribute xl 64

4-6 A decision structure without Xl with candidate decisions assigned to leaves 65

4-7 A decision structure determined from rules in Figure 4-4 under the

assumption of 10 classification error in the training data 65

4-8 Diagramatic visualization of decision trees learned for different decision

making situations for the wind bracing data 66

4-9 The accuracy of AQDT-2 and AQ15c with different parameter settings

for the wind bracing PIoblem 68

4-10 Analyzing different parameter setting of AQDT-2 using the wind bracing data 69

4-16 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-20 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-11 A comparison between AQ15c and AQDT-2 against C45 on the wind bracing data 69

4-12 A visualization diagram of the MONK-l problem 70

4-13 Decision rules learned by AQ15c for the MONK-1 problem 71

4-14 The decision tree for the MONK-1 problem generated both by AQDT-l 72

4-15 Compact decision structures generated by AQDT-2 for the MONK-1 problem 72

for the MONK-l Problem 74

4-17 Analyzing different parameter setting of AQDT-2 with the MONK-l data 75

4-18 Comparing AQ15c and AQDT-2 against C45 using the MONK-1 problem 75

4-19 A visualization diagram of the MONK-2 problem 76

for the MONK-2 Problem 78

4-21 Analyzing different parameter setting of AQDT-2 with the MONK-2 data 79

xii

4-22 Comparing AQ15c andAQDT-2 against C45 using the MONK-2 problem 79

4-24 The accuracy of AQDT-2 and AQ15c with different parameter settings

4-35 A visualization diagram of a decision tree learned by AQDT -2 after reducing

4-23 A visualization diagram of the MONK-3 problem 80

for the MONK-3 Problem 82

4-25 Analyzing different parameter setting of AQDT-2 with the MONK-3 data 82

4-26 Comparing AQ15c and AQDT-2 against C45 using the MONK-3 problem 83

4-27 Comparing AQ15c and AQDT-2 against C45 using the breast cancer problem 84

4-28 Comparing AQ15c and AQDT-2 against C45 using the mushroom problem 85

4-29 Decision structures learned by AQDT-2 for different decision-making situations 87

4-30 Comparing decision trees for the Cong Voting-84 data learned by C45 amp AQDT -2 88

4-31 A visualization diagram of decision rules learned by AQ15c for the MONK-2 91

4-32 A visualization diagram shows the testing errors for the rules in Figure 4-31 92

4-33 A visualization diagram of the decision tree learned by AQDT-2 for the MONK-2 93

4-34 A visualization diagram shows testing errors of AQDT-2 decision tree bull 94

the generalization degree to 1 94

DERIVING TASKmiddotORIENTED DECISION STRUCTURES

FROM DECISION RULES

Ibrahim M Fahmi Imam PhD

School of Information Technology and Engineering

Geotge Mason University Fall 1995

Ryszard S Michalski Advisor

ABSTRACT

This dissenation is concerned with research on learning task-oriented decision structures from

decision rules The philosophy behind this research is that it is more appropriate to learn

knowledge and store it in a declarative form and then when a decision making situation occurs

generate from this knowledge the decision structure that is most suitable for the given decision

making situation Learning decision structures from decision rules was first introduced by

Michalski (1978) The flIst implementation of this approach was done by Imam and Michalski

(1993 ab) called AQDT-l

This approach separates the function of generating a knowledge-base from the function of using

the knowledge-base for decision-making The first function focuses on learning accurate

consistent and complete concept description expressed in a declarative form The second function

is performed whenever a new decision-making situation occurrs a task-oriented decision structure

is obtained to suit that situation Thsk-oriented knowledge is defined as knowledge that is adapted

for solving a given decision-making situation (Imam amp Michalski 1994 Michalski amp Imam

1994)

The dissertation introduces the system AQJJT-2 for learning task -oriented decision structures from

decision rules or examples Each decision making situation is defined by a set of parameters that

controls the learning process of the AQDT-2 system The extensive experiments on AQJJT-2 show

that decision structures learned by it usually outperform in terms of accuracy and average size of

the decision structures those learned from examples by other well known systems The results

show also that the system does not work very well with noisy data The system is illustrated and

compared using applications of artificial problems such as the three MONKs problems (Thrun

Mitchell amp Cheng 1991) and the East-West 1iain problem (Michie et al 1994) It was also

applied to real world problems of learning decision structures in the areas of construction

engineering (for determining the best wind bracing design for tall buildings) medical

diagnosis(for learning decision rules for recognizing breast cancer) agriculture diagnosis (for

learning classification rules for distinguishing between poisonous and non-poisonous

mushrooms) and political data (for characterizing democratic and republican voting records)

CHAPTER 1 INTRODUCTION

11 Motivation and Overview

Learning and discovery systems should be able not only to generate and store knowledge but also

to use this knowledge for decision-making The main step in the development of systems for

decision-making is the creation of a knowledge structure that characterizes the decision-making

process The form in which knowledge can be easily obtained may however differ from the form

in which it is most readily used for decision-making It is therefore important to identify the form

of knowledge representation that is most appropIiate for learning (eg due to ease of its

modification) and the form that is most convenient for decision making

A simple and effective tool for describing decision processes is a decision structure which is a

directed acyclic graph that specifies an order of tests to be appliedto an object (or a situation) to

arrive at a decision about that object The nodes of the structure are assigned individual tests

(which may correspond to a single attribute a function of attributes or a relation) the branches are

assigned possible test outcomes or ranges of outcomes and the leaves are assigned a specific

decision a set of candidate decisions with corresponding probabilities or an undetermined

decision A decision structure reduces to a familiar decision tree when each node is assigned a

single attribute and has at most one parent when the branches from each node are assigned single

values of that attribute and when leaves are assigned single definite decisions Thus the problem

of generating a decision structure is a generalization of the problem of generating a decision tree

Decision trees are typically generated from a set of examples of decisions The essential

characteristic of any such method is the attribute selection criterion used for choosing attributes to

be assigned to the nodes of the decision tree being built Such criteria include the entropy

reduction the gain and the gain ratio (Quinlan 1979 83 86) the gini index ofdiversity (Breiman

et al 1984) and others (Cestnik amp Bratko 1991 Cestnik amp Karalic 1991 Mingers 1989a)

3

4

Adecision treedecision structure representations can be an effective tool for describing a decision

process as long as all the required tests can be performed and the decision-making situations it

was designed for remain constant (eg in a doctor-patient example the doctor should detennine

that answers for all symptoms appear in the decision tree) Problems arise when these assumptions

do not hold For example in some situations measuring certain attributes may be difficult or costly

(eg in the doctor-patient example a brain or blood test is needed which is very expensive or the

tools needed are not available) In such situations it is desirable to refonnulate the decision

structure so that the inexpensive attributes are evaluated frrst (assigned to the nodes close to the

root) and the expensive attributes are evaluated only if necessary (by assignment to the nodes far

away from the root) If an attribute cannot be measured at all it is useful to either modify the

structure so that it does not contain that attribute or-when this is impossible-to indicate

alternative candidate decisions and their probabilities A restructuring is also desirable if there is a

significant change in the frequency of occurrence of different decisions (eg in the doctor-patient

example the doctor may request a decision structure expressed in a specific set of symptoms

biased to classify one or more diseases or specify a certain order of testing)

Arestructuring ofa decision structure (or a tree) in order to suit new requirements could be quite

difficult This is because a decision structure is a procedural representation that imposes an

evaluation order on the tests In contrast no evaluation order is imposed by a declarative

representation such as a set of decision rules Tests (conditions) of rules can be evaluated in any

order Thus for a given set of rules one can usually build a huge number of logically equivalent

decision structures (trees) which differ in the test ordering Due to the lack of order constraints

a declarative representation (rules) is much easier to modify to adapt to different situations than a

procedural one (a decision structure or a tree) On the other hand to apply decision rules to make a

decision one needs to decide in which order tests are evaluated and thus needs a decision

structure

An attractive solution to these opposite requirements is to acquire and store knowledge in a

declarative form and transform it to a decision structure when it is needed for decision-making

5

This method allows one to create a decision structure that is most appropriate in a given decisionshy

making situation Because the number of decision rules per decision class is usually small (each

rule is a generalization of a set of examples) generating a decision structure from decision rules

can potentially be perfonned much faster than by generation from training examples Thus this

process could be done on line without any delay noticeable to the user Such virtual decision

structures are easy to tailor to any given decision-making situation

This approach allows one to generate a decision structure that avoids or delays evaluating an

attribute that is difficult to measure in some decision-making situation or that fits well a particular

frequency distribution of decision classes In other situations it may be unnecessary to generate

complete decision structure but it may be sufficient to generate only a part of it which concerns

only with decision classes of interest Thus such an approach has many potential advantages

This dissertation presents a new system called AQflf-2 The AWf-2 system generates a taskshy

oriented decision structure (decision structure that is adapted to the given decision-making

situation) from decision rules The decision rules are learned by either rule leaming system AQ15

(Michalski et al 1986) or system AQ17-DCI which has extensive constructive induction

capabilities (Bloedorn et al 1993)

To associate the decision rules with a given decision-making task AQflf-2 provides a set of

features including 1) enabling the system to include in the decision structure nodes corresponding

to new attributes constructed during the process of learning the decision rules 2) controlling the

degree of generalization needed during the development of the decision structure 3) providing four

new criteria for selecting an attribute to be a node in the decision structure that allow the system to

generate many different but equivalent decision structures from the same set of rules 4) generating

unknown nodes in situations when there is insufficient infonnation for generating a complete

decision structure 5) learning decision structures from discriminant rules as well as

characteristic rules and 6) providing the most likely decision when the decision process stops

due to inability to evaluate an attribute associated with an intennediate node

6

To test the methooology of generating decision structures from decision rules an extensive set of

planned experiments have been designed to test different aspects of the approach The experiments

include testing different combinations of parameters for each sub-function of the approach

analyzing the relatioship between decision rules and the decision structure learned from them and

comparing decision trees learned by the AQUf2 system the well known C45 (Quinlan 1993)

system for learning decision trees from examples Different experiments were designed to examine

the new features of the AQUf approach for learning task-oriented decision structures The

experiments were applied to artificial domains as well as real-world domains including MONK-I

MONK-2 and MONK-3 (Thrun Mitchell amp Cheng 1991) East-West trains (Michie et al

1994) Engineering Design-wind bracings (Arciszewski et al 1992) Mushrooms Breast

Cancer (Mangasarian amp WolbeJg 1990) and Congressional bting Records of 1984 The

MONKs problems are concerned with learning classification rules for robot-like figures MONK-l

requires learning a DNF-type description MONK-2 requires learning a non-DNF-type description

(one that cannot be easily described as a DNF rule using the original attributes) MONK-3 concerns

learning a DNF rule from noisy data The East-West trains dataset is a structural domain that

classifies two sets of trains (Eastbound and Westbound) The Engineering Design-wind

bracing data involves learning conditions for applying different types of wind bracing for ta11

buildings The Mushrooms data is concerned with learning classification rules for distinguishing

between poisonous and non-poisonous mushrooms The Breast Cancer data involves learning

concept descriptions for recognizing breast cancer The congressional voting data includes voting

records on different issues AQUf2 outperformed C45 on the average with respect to both the

predictive accuracy and tree size for most problems Apf-2 did not work very well with noisy

data or with problems that have many rules covering very few examples

12 The Problem Statement

There are many limitations and problems accompanied using decision trees for decision-making

CHAPTER 2 RELATED RESEARCH

21 Learning Decision Trees from Decision Diagrams

The AQDT-2 method proposed here is based on the earlier work by Michalski (1978) which

introduced an algorithm for generating decision trees from decision lists The method proposed

several attribute selection criteria These criteria are of increasing power of the main criterion

order cost estimate (the nth order cost estimates n=l 2 ) Michalski also analyzed two

specific criteria MAL and DMAL for selecting the optimal attribute for a node in the tree

based on properties extracted from the decision diagram In order to better explain the method

it is necessary to define some terms These terms will be used later in the dissertation

Definition 2-1 A ~ of a decision class C is a set of rules whose union includes the set of all

examples of class C and does not include any examples of other decision classes

Definition 2-2 A ~ of a decision class C is disjoint if all rules (rules) are pairwise logically

disjoint In other words for any two rules there exists a condition with the same attribute but

with different values in each rule

Definition 2-3 An minimal cover is a disjoint cover which has the smallest number of rules

among all possible covers

Definition 2-4 A diagram for a given cover is a table constructed graphically by representing

in two dimension space of all possible combination of attributes values and locating on the

diagram all the condition parts of the given rules and marking them with the action specified

with each rule

Michalskis algorithm requires construction of minimalmiddot cover The minimal cover should be

consistent and complete The method is based on the fact that if there are n decision classes

any decision tree that correctly classifies the given rules and has n distinct leaves is a minimal

7

8

decision tree (any consistent decision tree should have at least n leaves) Michalski (1978) has

shown that if only one rule is broken by a selected attribute then instead of having one leaf

(which could potentially represent this rule or the decision class in the tree) there will have to

be at least two leaves representing this rule in the fmal decision tree

The attribute selection criterion MAL introduced in (Michalski 1978) prefers attributes that do

not break any rules or break as few as possible An attribute breaks a rule if the attribute can

divide the rule into two or more sub-rules Figure 2-1 shows two examples of two sets of rules

In the fJIst xl and x3 break at least two rules each (xl breaks the rule [x4=2] amp [x1=lv3] amp

[x3=2v3] and the rule [x4=3] amp [x1=1v2] amp [x3=1] and x3 breaks three rules the rule [x4=2]

amp [xl=2] amp [x3=2v3]the rule [x4=l] amp [xl=3] amp [x3=lv3] and the rule [x4=3] amp [xl=4] amp

[x3=2v3]) In the diagram on the right xl is the only attribute that does not break any rule

Figure 2-1 An example to illustrate how attributes break rules

One of the criteria defmed by Michalski is the first degree cost estimate which assigns to each

attribute an integer equal to the number of rules broken by that attribute This criterion is also

called the static cost estimate of an attribute or the criterion of minimizing added leaves

(MAL)

-

C3

9

The MAL criterion (Minimizing Added Leaves) seeks an attribute that minimizes the

estimated number of additional nodes in the decision tree being generated over a hypothetical

minimal decision tree When there is a tie between two attributes the attribute to be selected is

the one which breaks smaller rules (rules that cover fewer examples or more specialized

rules) AQDT-2 uses an approximate version of this criterion (the attribute dominance)

Another criterion introduced by Michalski was the DMAL criterion The DMAL criterion

(Dynamically Minimizing Added Leaves) is based on a principle similar to that of MAL but

is more complex because once an attribute is selected as a node in the tree some rules andor

parts of the broken rules at each branch are merged into one rule The DMAL ensures that the

value of the total cost estimate of an attribute is decreased by a value equal to the number of

merged rules minus one

Example Learn a decision tree from the following decision table

The minimal cover consists of the following rules

Al lt [x2=O] v [x1=0][x2=2] A2 lt [x2=I] v [xl=2][x2=2] A3 lt [xl=I][x2=2]

The evaluations ofthe MAL criterion for these attributes are 2 for xl 0 for x2 5 for x3 and 5

for x4 The attribute xl divides the two rules [x2=O] and [x2=I] so selecting it will add two

leaves to the optimal number of leaves It is clear that the attribute to be selected as a root of

10

the decision tree is x2 Then three branches are attached to the root node and the decision rules

are divided into subsets each corresponding to one branch For x2 = 0 or 1 a leave node is

generated For x2=2 another attribute is selected to be a node in the tree In this case xl has the

minimum MAL value Figure 2-2 shows the decision tree obtained using the MAL criterion

Al A3 A2

Figure 2-2 A decision tree learned from the decision table in Table 2-1

22 Learning Decision Trees from Examples

Decision tree learning is a field that concerned with generating decision tree that classifies a set

of examples according to the decision classes they belong to The essential aspect of any

inductive decision tree method is the attribute selection criterion The attribute selection

criterion measures how good the attributes are for discriminating among the given set of

decision classes The best attribute according to the selection criterion is chosen to be assigned

to a node in the tree The fIrst algorithm for generating decision trees from examples was

proposed by Hunt Marin and Stone (1966) Hunts algorithm uses a divide and conquer

algorithm for building decision trees This algorithm has been subsequently modified by

Quinlan (1979) and applied by many researchers to a variety of learning problems

Attribute selection criteria can be divided into three categories These categories are logicshy

based information-based and statistics-based The logic-based criteria for selecting attributes

use logical relationships between the attributes and the decision classes to determine the best

attribute to be a node in the decision tree such as the MAL criterion minimizing added leaves

(Michalski 1978) which uses conjunction and disjunction operators The information-based

criteria are based on the information theory These criteria measure the information conveyed

11

by dividing the training examples into subsets Examples of such criteria include the

information measure 1M the entropy reduction measure and the gain criteria (Quinlan 1979

83) the gini index of diversity (Breiman et al 1984) Gain-ratio measure (Quinlan 1986) and

others (Clark amp Niblett 1987 Bratko amp Lavrac 1987 Cestnik amp Karalic 1991) The

statistics-based criteria measure the correlation between the decision classes and the attributes

These criteria use statistical distributions for determining whether or not there is a correlation

The attribute with the highest correlation is selected to be a node in the tree Examples of

statistics-based criteria include Chi-square and G-statistic (Sokal amp Rohlf 1981 Han 1984

Mingers 1989a)

Niblett and Bratko (1986) Quinlan (1987) and Bratko and Kononeko (1987) extended the

method of learning decision trees to also handle data with noise (by pruning) Handling noise

extended the process of learning decision trees to include the creation of an initial complete

decision tree tree pruning which is done by removing subtrees with small statistical Validity

and replacing them by leaf nodes (MiDgers 1989b) More recently pruning has also been used

for simplifying decision trees even for problems without noise (Bohanec amp Bratko 1994)

Pruning decision trees improves their simplicity but reduces their predictive accuracy on the

training examples Quinlan (1990) proposed also a method to handle the unknown attributeshy

value problem by exploring probabilities of an example belonging to different classes

The rest of this section includes a brief description of the attribute selection criterion used by

C45 learning system (Quinlan 1993) C45 uses an information-based criterion for selecting

an attribute to be a node in the tree Also the section includes a brief description of the Chishy

square method for attribute selection (Mingers 1989a) The method is a statistics-based

method for selecting an attribute to be a node in the tree

221 Building Decision Trees Using htformation-based Criteria

12

This section presents a description of the inductive decision tree learning system C4S The

C4S learning system is considered to be one of the most stable accurate and fastest program

for learning decision trees from examples

Learning decision trees from examples requires a collection of examples Each example is

represented by a fixed number of attribute-value pairs C45 (Quinlan 1993) is a learning

program that induces classification decision trees from a set of given examples The C4S

learning system is descended from the learning system ID3 (Quinlan 1979) which is based on

Hunts method for constructing decision tree from a set ofcases (Hunt Marin amp Stone 1966)

The C45 system uses an attribute selection criterion called the Gain Ratio This criterion

calculates the gain in classifying information based on the residual information needed to

classify cases in a set of training examples and the information yielded by the test based on the

relative frequencies of the possible outcomes (decision classes) The gain ratio criterion is

based on an earlier criterion used by ID3 called the Gain Criterion The Gain Criterion uses

the frequency of each decision class in the given set of training examples

Once an attribute is chosen to be a node in the tree the system generates as many links as the

number of its values and classifies the set of examples based on these values If all the

examples at a certain node belong to one decision class the system generates a leaf node and

assigns it to that class Otherwise the system searches for another attribute to be a node in the

tree

The Gain Criterion The gain criterion is based on the information theory That is the

information conveyed by a message depends on its probability and can be measured in bits as

minus the logarithm (base 2) of that probability To explain the gain criterion suppose that for

a given problem Xu bullbull X are given attributes and Cu c are decision classes Suppose S is

any set of cases and T is the initial set of training cases The frequency of class C1 in the set S

is the number of examples in S that belong to class C i bull

13

freq(Cit S) = Number of examples in S belong to CI (2-1)

Suppose that lSI is the total number of examples in S the probability that an example selected

at random from S belongs to class Ci is freq(CIbullS) IISI

The information conveyed by the message that a selected example belongs to a given decision

class Cit is determined by -log1 (freq(Ch S) IISI) bits

The expected information from such a message stating class membership is given by

info(S) = -k

L (freq(C S) I lSI) log1 (freq(C S) IISI) bits (2-2)I

info(S) is known also as the entrQpy of the set S When S is the initial set of training examples

info(T) determines the average amount of information needed to identify the class of an

example in T

Suppose that we selected an attribute X to be the root of the tree and suppose that X has k

possible values The training set T will be divided into k subsets each corresponding to one of

Xs values The expected information of selecting X to partition the training set T infoz(T) can

be found as the sum over all subsets of multiplying the information conveyed by each subset

by its probability

k

infoz(T) =L (ITII 1m) infO(TIraquo (2-3) I

The information gained by partitioning the training examples T into subset using the attribute

X is given by

gain(X) =inlo(T) - inoIT) (2-4)

The attribute to be selected is the attribute with maximum gain value

The Gain Ratio Criterion This criterion indicates the proportion of information generated by

the split that appears helpful for classification Quinlan (1993) pointed out that the gain

criterion has a serious deficiency Basically it is strongly biased toward attributes with many

outcomes (values) For example for any data that contains attributes such as social security

14

number the gain criterion will select that attribute to be the root of the decision tree However

selecting such attributes increases the size of the decision tree Quinlan provided a solution to

this problem by introducing the gain ratio criterion which takes the ratio of the information that

is gained by partitioning the initial set of examples T by the attribute X to the potential

information generated by dividing T into n subsets

Following similar steps to obtain the information conveyed by dividing T into n subsets The

expected information generated by dividing T into n subsets and by analogy to equation 2-2 is

determined by

split info(T) = - Lt

(ITII lTI) 10gl (ITII m ) (2-5) I

The gain ratio is given by

gain ratio(X) = gain(X) I split info(X) (2-6)

and it expresses the proportion of information generated by the split that is useful for

classification

Example Consider the following example presented by Quinlan (1993) Table 2-2 shows the

set of training examples

First determine the amount of information gained by selecting the attribute outlook to be a

root of the decision tree This attribute divides the training examples into three subsets

sunny-with five examples two of which belong to the class Play overcast-with four

examples all of which belong to the class Play and rain-with five examples three of

which belong to the class Play To determine info(11 the average information needed to

identify the class of an example in T there are 14 training examples and two decision classes

Nine of these examples belong to the class Play and five belong to the class Dont Play

Info(11 = - 914 10gl (914) - 514 10gl (514) = 094 bits

When using outlook to divide the training examples the information becomes

15

infO(T) = 514 -25 log2 (25) - 35 10g2 (35)

+ 414 -44 10g2 (44) - 04 log2 (04)

+ 514 -35 logl (35) - 25 logl (25) = 0694 bits

By substituting in equation 2-4 the gain of information results from using the attribute

outlook to split the training examples equal to 0246 The gain information for windy is

0048

Figure 2-3 shows a decision tree learned for this problem using the gain criterion The split

information for outlook is determined as follows

split info(T) = - 514 logl (514) - 414 log2 (414) - 514 log2 (514) = 1577 bits

The gain ratio for outlook = 02461577 =0156

16

Figure 2 -3 A decision tree learned using the gain criterion for selecting attributes

The C45 system handles discrete values as well as continuous values To handle an attribute

with continuous values C45 uses a threshold to transform the continuous domain into two

intervals In other words for each continuous attribute C45 generates two branches one

where the values of that attribute is greater than the determined threshold and the other if the

value is less than or equal to the threshold

Tree pruning in C45 is a process of replacing subtrees with small classification validity by

leaves The C45 system uses Laplace ratio for determining the error rate of different subtrees

This ratio is defined as (e+l) (n+2) where n is the number of the training examples and e is

the number of misclassified examples at a given leaf

222 Building Decision Trees Using Statistics-based Criteria

The Chi-square method for selecting attributes was used by Hart (1984) and Mingers (1989a)

in building decisionmiddot trees The method uses Chi-square statistics to measure the association

between two attributes When building decision trees the method is implemented such that it

determines the association between each attribute and the decision classes The attribute to be

selected is the one with greatest value

To determine the Chi-square value for an attribute consider aij is the number of examples in

class number i where the attribute A takes value numberj In other words aij is the frequency

of the combination of decision class number i and the attribute value number j The Chi-square

value for attribute A is given by

n m

Chi-square (A) =L L [ (aij - Eijl2 Eij ] (2shy

i=1 j=1

where n is the number of decision classes m is the number ofvalues of a given attribute Also

17

Eij = (TCi TVj ) T (2shy

8)

where T Ci and T Vj are the total number of examples belong to the decision class a and the total

number of examples where the attribute A takes value vj respectively T is the total number of

examples

Consider Quinlans example in Table 2-2 Table 2-3 shows the frequency of different

combination of values between the decision class and both the outlook and the Windy

attributes Table 2-4 shows the expected values TCi and T Vj of the frequencies in Table 2-3

of different attributes values for different decision classes

To determine the association value between the decision classes and both the attribute Windy

and the attribute Outlook the observed Chi-square values are

Chi-square (Windy Class) =[(3-39)239] + [(3-21)221] + [(6-51)232] + [(2-29)232]

=021 + 039 + 025 + 025 = 11

Chi-square (Outlook Class) = [(2-32)232] + [(4-26)226] + [(3-32)232] + [(3-18118]

+ [(0-14)214] + [(2-18)2 18] =045 + 075 + 02 + 08 + 14 + 002 = 362

18

Applying the same method to the other attributes the results will favor the attribute Outlook

Once that attribute is selected to be a node in the tree the remaining set of examples are divided

into subsets and the same process is repeated on each subset

Table 2-5 shows a summary of these criteria and their basic evaluation function

Table 2-5 Attribute selection criteria and their basic evaluation measure

Info Measure (IM) Gain Gshystatistics and Gain Ratio Chi-square

II

Entropy(S) = - L (freq(~ 5) 1151) logz (freq(CIt 5) 1151 ) I

G-Statistics =2N 1M (N-No of examples) n m

Chi-square (A B) = L L [(aij - ~j)21Eij ] i=l

223 Analysis of Attribute Selection Criteria

This subsection introduces briefly the analysis of different selection criteria which was done by

Mingers (1989a) Mingers compared six attribute selection criteria that are used in decision

tree programs These criteria are the Information Measure (IM) Chi-square G statistics Gin

index of diversity Marshal correction and Gain RaJio The overall results show that the Gain

Ratio criterion had the strongest xesults

In the flISt experiment Minger tested the six criteria on a problem with ambiguous examples

(ie examples may belong to more than one decision class) to observe how the selected criteria

evaluate the given attributes The problem has two decision classes and two attributes X and

Y It was assumed that attribute X is better for classifying the examples than attribute Y The

training examples were unevenly spread between the two values of X Attribute Y has three

values and the examples were spread randomly among them Table 2-6 (a and b) shows the

contingency tables for both attributes Table 2-7 shows a summary of the goodness of split

provided by the six criteria Mingers noted that the measures that are not based on information

19

theory give radiation (attribute X here) less weighL This may be because the zero in the fIrst

row of radiation has a greater influence in the log calculation In the case of using the Chishy

square criterion the value zero adds the maximum association between any two attributes

because the Chi-square value of a zero cell is the expected value of this cell

b)

Now let us demonstrate results from another experiment done by Mingers In this experiment

Mingers used four different data sets to generate decision trees for eleven different criteria In

the final results he compared the total number of nodes and the total error rate provided by

each criterion over all given problems Table 2-8 shows the final results for five selected

criteria only

20

U lgt ~middot results for comparing the total accuracy and size of decision trees different attribute selection criteria from four VVLU

This experiment was performed on four real world data sets These data are concerned with

profiles of BA Business Studies degree students recurrence of breast cancer classifying types

of Iris and recognizing LCD display digits The data was divided randomly 70 for training

and 30 for testing For more details see Mingers (1989a)

23 Learning Decision Structures

Considering the proposed defInition of decision structures given above two related lines of

research are described in this section Gaines (1994) and Kohavi (1994) proposed two

approaches for generating decision structures that share some of the earlier ideas by Imam and

Michalski (1993b)

In the first approach Brian Gaines introduces a method for transforming decision rules or

decision trees into exceptional decision structures The method builds an Exception Directed

Acyclic Graph (EDAG) from a set of rules or decision trees The method starts by assigning

either a rule say RO or a conclusion say CO or both to the root node If it assigns a conclusion

to the root node it places it on a temporary conclusion list Then it generates a new child node

and add to it a new rule say Rl The method evaluates the new rule Rl if it satisfies the

conclusion CO in its parent node it builds a new child and repeats the processes Otherwise if

the rule Rl does not satisfy the conclusion CO it changes the temporary memory with the new

conclusion Cl which is satisfied by the rule RO The same process is repeated until all rules

that have common conditions with the rule at the root are evaluated The method then creates a

21

new child node from the root and repeat the process until all rules are evaluated In the decision

structure nodes containing rules only represent common conditions to all of its children

The main disadvantages of this approach is that it requires discriminant rules to build such a

decision structure Also such a structure is more complex than the traditional decision trees

that are used for decision-making

Figure 2-4 shows an example of set of rules and their equivalent exceptional decision structure

The decision structure can be read as follows It is Safe except if xl=1 amp x2=1 amp x3=1 and

either x4=3 or x5=I then it is Lost except if x6=1 it is Safe except if x7=1 it is Lost

The second approach introduced by Ronny Kohavi learns decision structures from examples

using a bottom-up algorithm The method uses a greedy Hill-climbing algorithm for inducing

Oblivious read-Once Decision Graphs (HOODO) A read-once decision graph is defined as a

decision graph where each attribute occurs at most once along any computational path In other

words for each path from the root to the leaves of the decision structure attributes may occur

as a node once at most However there may be more than one node with the same attribute in

the decision graph Kohavi also defined a leveled decision graph in which the nodes are

partitioned into a sequence of pairwise disjoint sets such that outgoing edges from each level

terminate at the next level An oblivious decision graph is a decision graph where all nodes at

a given level are labeled by the same attribute

22

Safe lt [xl=2] Safe lt [x2=2] Safe lt [x3=2] Safe lt [x4=1] amp [x5=2] Safe lt [x4=1] amp [x5=3] Safe lt [x6=l] amp [x7=2] Safe lt [x6=1] amp [x7=3] Safe lt [x4=2] amp [x5=2] Safe lt [x4=2] amp [x5=3]

Lost lt [x6=2] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x6=3] amp [x5=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x5=1] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x7=1] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=2] amp [x3=1] amp [x2=I] amp [xl=l] Lost lt [x4=3] amp [x6=3] amp [x3=1] amp [x2=I] amp [xl=l]

The algorithm starts by generating a leaf node for each decision class Then it applies a

nondeterministic method to select an attribute to build new level of the decision structure This

attribute is removed from the data and the data is divided into subsets each corresponding to a

complex combination of that attributes values For each subset the process is repeated until all

the examples of a given subset belong to one decision class For example if the selected

attribute say A has two values (0 and 1) and there are two decision classes (CO and CI) The

data is divided into two subsets the fIrst subset contains examples where A takes value 0 and

belong to class CO or takes value 1 and belong to class Cl The second subset of examples is

the set where A takes value 0 and belong to class CI or takes value 1 and belong to class CO

The number of nodes of the fIrst level (after the leave nodes) is expected to be less than or

equal to ka where k is the number of decision classes and n is the number of values of the

selected attribute and it can increase exponentially before that number is reduced

exponentially to one

It is easy for the reader to fIgure out some major disadvantages of such approach including

The average size of such decision structures is estimated to be very large especially when there

23

is no similarity (ie strong patterns) or logical relationship in the data The time used to learn

such decision structure is relatively very high comparing to systems for learning decision trees

from examples and fmally it could be better to search for attribute which reduces the number

of generated subsets of the data instead of nondeterministically selecting an attribute to build a

new level of the decision structure Kohavi provided a comparison between the C45 system

and his system that can be found in (Kohavi 1994) Later Kohavi and Li (1995) used a

deterministic method for attribute selection which minimizes the width of the penultimate level

of the graph

Table 2-9 shows a comparison between the proposed approach and those two approaches The

EDAG and HOODG systems are unreleased prototype systems

or Decision structures are easy to Decision structures are Decision structures are easy to understand difficult to read understand

CHAPTER 3 DESCRIPTION OF THE APPROACH

31 General Methodology

In the proposed approach the function of learning or discovery is separated from the function of

using the discovered knowledge for decision-making The frrst function is performed by an

inductive learning program that searches for knowledge relevant to a given class of decisions

and stores the learned knowledge in the form of decision rules The second function is performed

when there is a need for assigning a decision to new data points in the database (eg bull a

classification decision) by a program that transforms obtained knowledge into a decision

structure optimized according to the given decision-making situation

The Learning Task GivenA set of training examples describing the concept to be learned Learning goal which specifies the decision classes to be learned from the training examples A background knowledge to control the learning process Determine A concept description in a declarative form of knowledge (decision rules) that satisfies the learning goal

The Decision-making Task GivenA set of decision rules in a conjunctive form A description of the new decision-making situation (eg attributes costs and order preference importance or frequency of decision classes etc) One or more examples that needed to be tested under the given decisionshymaking situation A set of parameters to control the learning process Determine A decision structure that suits the given decision-making situation

The decision rules used here are learned by either the AQ1S (Michalski et al 1986) or AQ17

(Bloedorn et aI 1993) learning systems The reasons for selecting decision rules as a form of

23

24

declarative knowledge are that they do not impose any order on the evaluation of the attributes

and due to the lack of order constraints decision rules can be evaluated in many different

ways which increases the flexibility of adapting them to the different tasks of decision-making

(Bergadano et al 1990) Since the number of rules learned per class is typically much smaller

than the number of examples per class generating a decision structure from decision rules can

potentially be done on line

Such virtual decision structures can be tailored to any given decision-making situation The

needed decision rules have to be generated only once and then they can be used many times for

generating decision structures according to changing requirements of decision-making tasks The

method uses the AQDT-2 system (Imam amp Michalski 1994) for learning decision structures

from decision rules Decision structures represent a procedural form of knowledge which makes

them easy to implement but also harder to change Consequently decision structures can be quite

effective and useful as long as they are used in decision-making situations for which they are

optimized and the attributes specified by the decision structure can be measured without much

cost Figure 3-1 shows an architecture of the proposed methodology

Data

Decision -

- Learning knowledge from database The decision making process

Figure 3 -1 Architecture of the AQDT approach

25

It is assumed that the database is not static but is regularly updated A decision-making problem

arises when there is a case or a set of cases to which the system has to assign a decision based on

the knowledge discovered Each decision-making situation is defmed by a set of attribute-values

Some attribute-values may be missing or unknown A new decision structure is obtained such

that it suits the given decision-making problem The learned decision structure associates the

new set of cases with the proper decisions

32 A Brief Description of the AQIS and AQ17 Rule Learning Programs

The decision rules are generated from examples by an AQ-type inductive learning system

specifically by AQ15 (Michalski et albull 1986) or AQI7-DCI (Bloedorn at al 1993) The rest of

this subsection includes a brief description of the inductive learning systems AQ15 and AQI7

AQ15 learns decision rules for a given set of decision classes from examples of decisions using

the STAR methodology (Michalski 1983) The simplest algorithm based on this methodology

called AQ starts with a seed example of a given decision class and generates a set of the

most general conjunctive descriptions of the seed (alternative decision rules for the seed

example) Such a set is called the star of the seed example The algorithm selects from the star

a description that optimizes a criterion reflecting the needs of the problem domain If the

criterion is not defined the program uses a default criterion that selects the description that

covers the largest number of positive examples (to minimize the total number of rules needed)

and with the second priority that involves the smallest number of attributes (to minimize the

number of attributes needed for arriving at a decision)

If the selected description does not cover all examples of a given decision class a new seed is

selected from uncovered examples and the process continues until a complete class description

is generated The algorithm can work with few examples or with many examples and can

optimize the description according to a variety of easily-modifiable hypothesis quality criteria

26

The learned descriptions are represented in the form of a set of decision rules expressed in an

attributional logic calculus called variable-valued logic 1 or VLl (Michalski 1973) A

distinctive feature of this representation is that it employs in addition to standard logic operators

the internal disjunction operator (a disjunction of values of the same attribute in a condition) and

the range operator (to express conditions involving a range of discrete or continuous values)

These operators help to simplify rules involving multi valued discrete attributes the second

operator is also used for creating logical expressions involving continuous attributes

AQ15 can generate decision rules that represent either characteristic or discriminant concept

descriptions depending on the settings of its parameters (Michalski 1983) A characteristic

description states properties that are true for all objects in the concept The simplest

characteristic concept description is in the form of a single conjunctive rule (in general it can be

a set of such rules) The most desirable is the maximal characteristic description that is a rule

with the longest condition part ie stating as many common properties of objects of the given

class as can be determined A discriminant description states properties that discriminate a given

concept from a fixed set of other concepts The most desirable is the minimal discriminant

descriptions that is a rule with the slwnest condition part For example to distinguish a given

set of tables from a set of chairs one may only need to indicate that tables have large flat top

A characteristic description of the tables would include also properties such as have four legs

have no back have four corners etc Discriminant descriptions are usually much shorter than

characteristic descriptions

Another option provided in AQ15 controls the relationship among the generated descriptions

(rulesets or covers) of different decision classes In the IC (Intersecting Covers) mode

rulesets of different classes may logically intersect over areas of the description space in which

there are no training examples In the DC (Disjoint Covers) mode descriptions of different

classes are logically disjoint The DC mode descriptions are usually more complex both in the

number of rules and the number of conditions There is also a DL mode (a Decision List mode

27

also called VL modeM-for variable-valued logic mode) in which the program generates rule sets

that are linearly ordered To assign a decision to an example using such rulesets the program

evaluates them in order If ruleset is satisfied by the example then the decision is made

otherwise the program proceeds to the evaluation of the ruleset +1 In IC and DC modes

rule sets can be evaluated in any order

Alternatively the system can use rules from the AQ17-DCI program for Data-driven

Constructive Induction AQ17-DCI differs from AQ15 mainly in that it contains a module for

generating additional attributes These attributes are various logical or mathematical

combinations the original attributes The program generates a large number of potential new

attributes and seleets from them those most promising based on an attribute quality criterion

To illustrate the format of rules generated by AQ15 (or AQ17-DCI) an exemplary ruleset is

shown in Figure 3-2 The ruleset (that can be re-represented as a disjunctive normal form

expression) describes a voting record of Democratic Representatives in the US Congress

Each rule is a conjunction of elementary conditions Each condition expresses a simple relational

statement For example the condition [State = northeast v northwest] states that the attribute

State (of the Representative) should take the value northeast or northwest to satisfy the

condition

Rl [Gas_concban = yes] amp [Soc_see_cut = no v not registered] R2 [Draft = Yes v not registered]amp[AlaslaLparks = yes v not registered] amp

[Food_stamp_cap = no] amp [State = northeast v northwest] R3 [Chrysler =yes v not registered] amp [Income =low] R4 [Education =yes] amp [Occupation = yes]

Figure 3-2 A ruleset generated by AQ15 for the concept Voting pattern of Democratic Representatives

The above rules were generated from examples of the voting records For illustration below is

an example of a voting record by a Democratic representative

Draft registration=no Ban of aid to Nicaragua=no Cut expenditure on mx missiles=yes Federal subsidy to nuclear power stations=yes Subsidy to national parks

28

in Alaska=yes Fair housing bill=yes Limit on Pac contributions=yes Limit onood stamp program=no Federal help to education=no StateFrom=north east State Population=large Occupation =unknown Cut in social security spending=no Federal help to Chrysler corp=not registered

By expressing elementary statements in the example as conditions and linking conditions by

conjunction the examples can be re-expressed as decision rules Thus decision rules and

examples formally differ only in the degree of generality

33 Generating Decision Structurestrees from Decision Rules

This section describes the AQDT-1 system for learning decision structures (trees) from decision

rules (Imam amp Michalski 1993ab) Also a description of the AQDT-2 method for learning taskshy

oriented decision structure from decision rules is included and fmally the methodology is

illustrated by two examples

Methods for learning decision trees from examples have been very popular in machine learning

due to their simplicity Decision trees built this way can be quite efficient as long as they are

used in decision-making situations for which they are optimized and these situations remain

relatively stable Problems arise when these situations significantly change and the assumptions

under which the tree was built do not hold anymore For example in some situations it may be

difficult to determine the value of the attribute assigned to some node One would like to avoid

measuring this attribute and still be able to classify the example if this is potentially possible

(Quinlan 1990) If the cost of measuring various attributes changes it is desirable to restructure

the tree so that the inexpensive attributes are evaluated f11st A tree restructuring is also

desirable if there is a significant change in the frequency of occurrence of examples from

different classes A restructuring of a decision tree to suit the above requirements is however

difficult to do The reason for this is that decision trees are a form of decision structure

representation and imposes constraints on the evaluation order of the attributes that are not

logically necessary

29

One problem in developing a method for generating decision structures from decision rules is to

design an attribute selection criterion that is based on the properties of the rules rather than of

the training examples A decision rule normally describes a number of possible examples Only

some of them are examples that have actually been observed ie training examples An attribute

selection criterion is needed to analyze the role of each attribute in the rules It cannot be based

on counting the numbers of training examples covered by each attribute-value and the

frequency of decision classes in the training examples as is done in learning decision trees from

examples because the training examples are assumed to be unavailable

Another problem in learning decision trees from decision rules stems from the fact that decision

rules constitute a more powerful knowledge representation than decision trees They can directly

represent a description in an arbitrary disjunctive normal form while decision trees can represent

directly only descriptions in the disjoint disjunctive normal form In such descriptions all

conjunctions are mutually logically disjoint Therefore when transforming a set of arbitrary

decision rules into a decision tree one faces an additional problem of handling logically

intersecting rules

The solution to both problems (attribute selection and logically intersected rules) in the AQDT-2

system is based on the earlier work by Michalski (1978) which introduced a general method for

generating decision trees from decision rules The method aimed at producing decision trees

with the minimum number of nodes or the minimum cost (where the cost was defined as the

total cost of classifying unknown examples given the cost of measuring individual attributes and

the expected probability distribution of examples of different decision classes) More

explanations are provided in the following section

331 Tbe AQDT-2attribute selection metbod

This section describes the AQDT -2 method for building a decision structure from decision rules

The method for building a single-parent decision structure is similar to that used in standard

30

methods of building a decision tree from examples The major difference is that it assigns tests

(attributes) to the nodes using criteria based on the properties of the decision rules (this includes

statistics about the examples covered by each rule in the case of learning rules from examples)

rather than statistics characterizing the frequency of training examples per decision classes per

attribute-values or per conjunctions of both Other differences are that the branches may be

assigned an internal disjunction of values (not only a single value as in a typical decision tree)

and leaves may be assigned a set of alternative decisions with probabilities Also the tests can be

attributes or names standing for logical or mathematical expressions that involve several

attributes or variables In the following we use the terms test and attribute interchangeably

(to distinguish between an attribute and a name standing for an expression the latter is called a

constructed attribute)

At each step the method chooses the test from an available set of tests that has the highest utility

(see below) for the given set of decision rules This test is assigned to the node The branches

stemming from this node are assigned test values or disjoint groups of values (in the form of

logical disjunction if such occur in the rules subsumed groups of values are removed) Each

branch is associated with a reduced set of rules determined by removing conditions in which the

selected attribute assumes value(s) assigned to this branch If all rules in the reduced ruleset

indicate the same decision class a leaf node is created and assigned this decision class The

process continues until all nodes are leaf nodes If it is not possible to reduce further the rule set

because some attribute is declared as unavailable (infmite cost) then the leaf is assigned a set of

candidate decisions with associated probabilities (see Sec 42)

The test (attribute) utility is a combination of one or more of the following elementary criteria 1)

cost which indicates the cost of using each attribute for making decision 2) disjointness which

captures the effectiveness of the test in discriminating among decision rules for different decision

classes 3) importance which determines the importance of a test in the rules 4) value

31

distribution which characterizes the distribution of the test importance over its of values and 5)

dominance which measures the test presence in the rules These criteria are dermed below

~ The cost of a test expresses the effort or cost needed to measure or apply the test

Djsjointness The disjointness of a test is defined as the sum of the class disjointness-the

disjointness of the test for each decision class Suppose decision classes are Cl C2 bull Cm and

decision rule sets for these classes have been determined Given test A let V 1 V 2 bull V m denote

sets of the values (outcomes) of A that are present in rule sets for classes C 1 C2bullbull Cm

respectively If a ruleset for some class say Ct contains a rule that does not involve test A than

Vt is the set of all possible values of A (the domain of A)

Definition 3 -1 The degree ofclass disjointness D(A Ci ) of test A for the ruleset ofclass Ci is

the sum of the degrees of disjointness D(A Cit Cj) between the ruleset for Ci and rulesets for

Cj j=l 2 m j J L The degree of disjointness between the ruleset for Ci and the ruleset for Cj is

dermed by

if Vi Vj ifVj gtVj (3-1) ifVinVj -tljgt amp -tVi amp -tVj ifVinVj=1jgt

where cjgt denotes the empty set Note that exchanging the second and third conditions of equation

(3-1) may seem to be an improved criteria However it does not clearly distinguish between both

cases (Le for both situations the disjointness will be similar) The current equation is better

because it gives higher scores to attributes that classify different subsets of the two decision

classes than attributes that classify only a subset of one decision class

Definition 3-2 The disjointness of the test A for evaluating a given set of decision rules is the

sum of the degrees of class disjointness of each decision class

32

m m Disjointness(A) = L D(A cn where D(A cn = L D(A Ci Cj) (3-2)

i=l i=l i~j

The disjointness of a test ranges from 0 when the test values in rulesets of different classes are

all the same to 3m(m-l) when every rule set of a given class contains a different set of the test

values If two tests have the same disjointness value the attribute to be selected is the one with

less number of values

Definition The A vera~ Number of Tests (ANT) required to make decision from a decision tree

is dermed as the average number of tests (attributes) to be examined from the root of the tree to

any leaf node in order to reach a decision

Definition A decision structure is a one-node-per-level decision structure if at each level there is

only one node and zero or more leaves

Such decision structure can be generated by combining together all branches that associated with

each one a set of decision rules belongs to more than one decision class

Theorem 1 Consider learning one-node-level decision structure for a database with two

decision classes The disjointness criterion ranks first attributes that add minimum number of

tests to the decision tree

Proof Consider all possible distributions of an attributes values within two decision classes Ci

and Cj There are three cases 1) subset (same as super set) 2) non-empty intersection but not

subset 3) no intersection Figure 3-3 shows all possible distributions (the case of having the

same set of values in both classes is trivial one) Consider that branches leading to one subset

with the same decision class are combined into one branch In the first case there will be two

branches only The first leads to a leaf node and the other leads to an intermediate node where

another attribute to be selected The minimum ANT in this case is 53 In the second case three

branches should be created Two branches leads to a leaf node where all values at each branch

belong to only one and different decision class The third branch leads to an intermediate node

33

where another attribute should be selected furtur classifies the decision classes The minimum

ANT in this case is 64 In the third case only two branches will be generated where each leads

to leaf node with different decision class In this case the minimum ANT is 1

Figure 3-4 shows decision trees equivalent to selecting the corresponding attribute Note that in

case of having more than one attribute-value at branches lead to leaves belonging to one decision

class they will be combined into one branch in the decision structure The symbol 1 means

that an attribute is needed to classify the two decision classes In such cases there will be at least

two additional paths

D(A Ci) = 0 D(A CJ) = 1 D(A Ci) =2 D(A Cj) = 2 D(A Ci) =3 D(A Cj) =3

Figure 3-3 Yen Diagrams of Possible combinations of attribute values in two decision classes

The average number of tests required for making a decision in each possible case is determined

in Figure 3-4 It is clear that in the case of two decision classes the disjointness criterion ranks

highly attributes that reduces the average number of tests required for decision-making The

theorem can be proved in the general case

i ~ i Cj C1middot bull Ci Cj

ANT=32middotANT=53 ANT=l

1 means at least one attribute is needed to complete the decision tree Figure 3-4 Decision trees corresponding the Yen diagrams inFigure 3-3

34

Theorem 2 The attribute with the highest non~zero disjointness is the best attribute to classify

the decision classes

Proof Suppose that the number of decision classes is n Assume also that there are two attributes

A and B where D(A) lt D(B) Since the disjointness criterion considers the mutual relationship

between any two decision classes this means that there are more decision classes where D(A

Ci) lt D(B Ci) than those where D(A Ci) gt D(B Ci) (ignoring classes with equal disjointness

for bCth attributes) Hence there are more decision classes where D(A Ci Cj) lt D(B Ci Cj)

than those where D(A Ci Cj) gt D(B Cit Cj) Let us frrst prove that if D(A Ci Cj) lt D(B Cit

Cj) then B classifies the decision classes better than A

For each pair of decision classes Ci and Cj the possible values of the disjointness of any attribute

are 0 14 or 6 For all positive values ofD(B)= 14 or 6 it is clear that attribute B should have

smaller ANT than attribute A which has lower disjointness This means that if D(A Cit Cj) lt

D(B Cit Cj) then attribute B can classify both decision classes better than attribute A

Similarly B is better for classifying more pairs of decision classes than A This implies that B is

a better classifier than A

Importance The second elementary criterion the importance of a test is based on the

importance score (IS) introduced in (Imam Michalski amp Kerschberg 1993) In the obtained

rules each test is assigned a score that represents the total number of training examples that

are covered by the rules involving this test Decision rules learned by an AQ learning program

are accompanied with information on their strength Rule strength is characterized by t-weight

and u~weight The t-weight (total-weight) of a rule for some class is the number of examples of

that class covered by the rule The importance score of a test is the aggregation of the totalshy

weights of all rules that contain that test in their condition part Given a set of decision rules for

m decision classes Cl Cm and involving n tests AlbullAn and the number of rules associated

with class Ci is denoted by tlrit The importance score is defmed as follow

35

Definition 3 -3 The importance score IS(Aj) of the test A j is detennined by

m IS(Aj)= L IS(Aj Ci) (3-31)

i=l

where ri IS(Aj Ci) = L Rik(Aj) (3-32)

k=l

and Rik the weight of a test Aj in the rule Rk of class Ci is given by

t - weight if A j belongs to rule Rik Rik (Amiddot) = (3-4)J 0 otherwise

where i=I n ik=I ri j=I m

The importance score method has been separately compared as a feature selection method with

a genetic algorithm-based method (Imam amp Vafaie 1994) The importance score method

produced an equal or higher accuracy on three real-world problems than those reported by the

GA method while selecting fewer attributes In addition the IS method was significantly faster

than the GA

yalue distribution The third elementary criterion value distribution concerns the number of

legal values of tests Given two tests with equal importance scores this criterion prefers the test

with the smaller number of legal values Experiments have shown that this criterion is especially

useful when using discriminant decision rules

Definition 34 A value distribution VD(Aj) of a test A j is defined by

VD(Aj) = IS(Aj) I Vj (3-5)

where v is the number of legal values of Aj

Dominance The fourth elementary criterion dominance prefers tests that appear in large

numbers of rules as this indicates their high relevance for discriminating among the rulesets of

given decision classes Since some conditions in the rules have values linked by internal

disjunction counting such rules directly would not reflect properly their relevance Therefore

36

for computing the dominance the rules are counted as if they were converted to rules that do not

have internal disjunction Such a conversion is done by multiplying out the condition parts of

the rules containing internal disjunction For example the condition part [x3=1 v 3]amp[x4=1] is

multiplied out to two rules with condition parts [x3=1]amp[x4=1] and [x3=3]amp[x4=I]

The above criteria are combined into one general test measure using the lexicographic

evaluation functional with tolerances (LEF) (Michalski 1973) LEF is a list of some or all of

the above elementary criteria each associated with a Ittolerance threshold It in percentage The

criteria are applied to tests in the order defined by LEF A test passes to the next criterion only if

it scores on the previous criterion within the range defined by the tolerance (from the top value)

The default LEF is

ltCost d Disjointness t2 Importance t3 Value distr t4 Dominance 0gt (3-6)

where d a t3 t4 and t5 are tolerance thresholds (in percentages) their default values are O

The default value of the cost of each test is 1

The above LEF ranks attributes this way First attributes are evaluated on the basis of their cost

If two or more attributes have the same cost they ranked by their degree of disjointness If two

or more attributes still share the same top score or their scores differ less than the assumed

tolerance threshold t2 the method evaluates these attributes using the second (Importance)

criterion If again two or more attributes share the same top score or their scores differ less than

the tolerance threshold t3 then the third criterion normalized IS is used then similarly the

fourth criterion (dominance) If there is still a tie the method selects the best attribute

randomly

If there is a non-uniform frequency distribution of examples of different classes then the

selection criterion uses a modified definition of the disjointness Namely the previously defmed

37

disjointness for each class is multiplied by the frequency of the class occurrence The class

occurrence is the expected number of future examples that are to be classified to a given class

m

Disjointness(A) = L D(A CO Frq(Ci) (3-7) i=l

where Frq(Cn is the expected frequency of examples of class Cit and it is assumed to be given

by the user The attribute ranking criteria in this case is defined by the LEF

ltCost d Disjointness t2 Importance 13 Normalized-IS 14 Dominance 15gt (3-8)

where the Cost denotes the evaluation cost of an attribute and is to be minimized while other

elementary criteria are treated the same way as in the default version

332 The AQDT-2 algorithm

The AQDT-2 algorithm constructs a decision structure from decision rules by recursively

selecting at each step the best test according to the ranking criteria described above and

assigning it to the new node The process stops when the algorithm creates terminal branches that

are assigned decision classes To facilitate such a process the system creates a special data

structure for each concept description (ruleset) This structure has fields such as the number of

rules the number of decision classes and the number of attributes present in the rules A set of

pointers connect this data structure to set of data structures each represents one decision class

The decision class structure contains fields with information on the number of rules belong to

that class the frequency of the decision class etc It is also connected to a set of data structures

representing the decision rules within each decision class The system creates independently a set

of data structures each is corresponding to one attribute Each attribute description contains the

attributes name domain type the number of legal values a list of the values the number of

rules that contain that attribute and values of that attribute for each rule The attributes are

arranged in an array in a lexicographic order flISt in the descending order of the number of rules

38

that contain that attribute and second in the ascending order of the number of the attributes

legal values

The system can work in two modes In the standard mode the system generates standard

decision trees in which each branch has a specific attribute-value assigned In the compact

mode the system builds a decision structure that may contain

A) or branches Le branches assigned an internal disjunction of attribute-values whenever it

leads to simpler structures For example if a node assigned attribute A has a branch marked by

values 1 v 2 then the control passes along this branch whenever A takes value 1 or 2 The

program creates or branches on the basis of the analysis of the value sets Vi while computing

the degree of attribute disjointness

B) nodes that are assigned derived attributes that is attributes that are certain logical or

mathematical combinations of the original attributes To produce decision structures with

derived attributes the input decision rules are generated by AQ17 (rather than AQI5) The

AQ17 rules may contain conditions involving attributes constructed by the program rather than

those originally given

To generate decision structures from rules the AQDT-2 method prefers either characteristic or

discriminant disjoint rule descriptions (given by an expert or learned by a system) Disjoint rules

are more suitable for building decision structures Assume that the description of each class is in

the form of a ruleset and that this set is the initial ruleset context The AQDT algorithm is

The AQDT2 Algorithm

Step 1 Evaluate each attribute occurring in the ruleset context using the LEF attribute ranking

measure Select the highest ranked attribute Let A represents this highest-ranked

attribute

Step 2 Create a node of the tree (initially the root afterwards a node attached to a branch) and

assign to it the attribute A In standard mode create as many branches from the node as

39

there are legal values of the attribute A and assign these values to the branches In

compact mode (decision structures) create as many branches as there are disjoint value

sets of this attribute in the decision rules and assign these sets to the branches

Step 3 For each branch associate with it a group of rules from the ruleset context that contain a

condition satisfied by the value(s) assigned to this branch For example if a branch is

assigned values i of attribute A then associate with it all rules containing condition [A=

i v ] If a branch is assigned values i v j then associate with it all rules containing

condition [A= i v j v ] Remove from the rules these conditions If there are rules in

the ruleset context that do not contain attribute A add these rules to all rule groups

associated with the branches stemming from the node assigned attribute A (This step is

justified by the consensus law [x=1] == [x=1] amp [y=a] v [x=1] amp [y=b] assuming

that a and b are the only legal values of y) All rules associated with the given branch

constitute a ruleset context for this branch

Step 4 If all the rules in a ruleset context for some branch belong to the same class create a leaf

node and assign to it that class If all branches of the trees have leaf nodes stop

Otherwise repeat steps 1 to 4 for each branch that has no leaf

To select an attribute to be a node in the decision tree (step 1 and 2 of the algorithm) the

algorithm performs two independent iterations In the first iteration it parses through all decision

rules and determine information about each attributes This information includes the importance

score of each attribute the number of rules containing a given attribute the disjoint value sets of

each attribute and the attribute values used in describing each decision class The second

iteration is only performed if the disjointness criterion is ranked first in the LEF function The

second iteration evaluates the attributes disjointness for each decision class against the other

decision classes

40

To determine the complexity of this process suppose that the maximum number of conditions

fonned by a single attribute value in one rule is s (where s ~ n the number of attributes) and r is

the total number of decision rules (in all decision classes)

m

r=LRt (m is the number of decision classes) 1=1

where Ri is the number of rules in decision class Ci bull The complexity of the frrst iteration can be

determined as

Cmpx(Itcl) =O(r s)

In the second iteration the disjointness is calculated between the decision classes and all

attributes The complexity of the second iteration can be given by

Cmpx(Itc2) = O(n m)

Assume that at each node 1 is the maximum of the number of decision classes to be classified at

this node and the number of rules associated with this node

l=max mr (3-9)

Since these two iteration are independent of each other the complexity of the AQDT algorithm

for building one node in the decision tree say Node Complexity NC(AQDT) is given by

NC(AQDT) = 0(1 n)

Usually I equals to the number of rules associated with the given node Thus the AQDT

complexity for building one node is a function of the number of attributes multiplied by the

number of rules associated with this node

At each level of the decision tree these two iterations are repeated for each non-leaf node The

Level Complexity of the AQDT algorithm LC(AQDT) can be given by

LC(AQDT) lt 0(1 n)

Which is less than the complexity of generating the root of the decision tree To explain this

consider the maximum possible number of non-leaf nodes at one level is half the number of the

initial decision rules (integer of rll) Figure 3-5-a shows an example of such situation where at

41

the lowest level each node classifies only two rules each belongs to different decision class This

decision tree could be a multimiddotvalues tree However the number of non-leaf nodes at each level

will be twice or more than the number of nonmiddot leaf nodes at the previous level Considering the

level complexity of the AQDT algorithm equal to (1 s 0) where 0 is the number of non-leaf

nodes at the given level In such cases it is either (1 0 S r) or (1 s lt r) In Figure 3middot5middota the

complexity of the AQDT algorithm at any lower level is given by

LC(AQDT) =0(2 s r(2) lt O(n I) =NC(AQDT)

a) per one level b) per one path

Figure 3 -5 Decision trees showing the maximum number of none leaf nodes

Note also that after selecting an attribute to be the root of the decision structure this attribute

and all conditions containing that attribute are removed from the data structure of the algorithm

Also if a leaf node is generated all rules belonging to the corresponding branch will not be

tested again

Since the disjointness criterion selects the attribute which minimizes the average number of tests

ANT it is true that the AQDT algorithm generates decision trees with the least number of levels

The number of levels per a decision tree is supposed to be less than or equal to the minimum of

both the number of attributes and the number of rules Consider k as the number of levels in a

given decision tree

k S min mr (3-10)

There are two cases represent the most complex situations Figure 3-5-a and 3-5-b In the fIrst

case where the decision rules were divided evenly the number of levels will be a function of the

42

logarithm of the number of rules In such case the complexity of the AQDT algorithm for

generating a decision tree from a set of decision rules is given by

Complexity(AQDT) = 0(1 n log r) (3-11)

The other situation is when the generated decision tree has the maximum number of levels The

maximum possible number of levels per a decision tree equals to one less than the number of

decision rules Figure 3-5-b Using the disjointness criterion it is not likely to get such a decision

tree because it has the maximum average number of test (ANT) that can be determined from the

same set of nodes and leaves However such a decision tree can be generated if the number of

decision classes is one less thanthe number of attributes In such case any disjQint decision rules

should have a maximum length that is less or equal to the floor of the logarithm of the number of

attributes Thus the level complexity of this decision tree is estimated as

LC(AQDT) =0(1 log n)

The maximum number of levels in such decision tree is k-l Thus the complexity of the AQDT

algorithm in such cases is given by

Complexity(AQDT) = 0(1 k log n) (3-12)

Since k is the minimum of n and r then n is the maximum of k and n Also since in (3-12)

r=n+1 then log r is greater than log n From 3-10 3-11 and 3-12 the complexity of the AQDT

algorithm is determined by

Cmplx(AQDT) = OCr k log 1) (3-13)

333 An example illustrating the algorithm

The following simple example illustrates how AQDT is used in selecting the optimal set of

testing resources for testing a software Suppose there are three tools for testing a software 1)

modeling (Tl) 2) checklist (T2) and 3) par_simul (T3) Also assume that there are four

different factors that affect the selection of any tool 1) the cost of using the tool (xl) 2) the

metrics that support the tool (x2) 3) the best phase for applying the tool (x3) and 4) type of tool

43

(automated semi-automated or manual) (x4) Table 3-1 shows these attributes and their possible

values

Suppose that the domain expert provided a set of rules to be used in testing a software Figure 3shy

6 shows a sample of these rules in AQ15c formats

Table 3-1 The available tools and the factors that affect the prclCes

Tl lt= [xl=2] amp [x2=2] v [xl=3] amp [x3=1 v 3] amp [x4=1] T2 lt= [xl=l v 2] amp [x2=3 v 4] v [xl=3] amp [x3=1 v 2] amp [x4=2] T3 lt= [xl=l] amp [x2=1] v [xl=4] amp [x3=2 v 3] amp [x4=3]

Figure 3-6 Decision rules for selecting the best tool for testing software

These rules can be interpreted as

Rule 1 Use the fltst tool for testing if you need average cost and the tool is

supported by the requirement metric

Rule 2 Use the fIrst tool for testing if you can afford high cost for testing either

in the requirement or the analysis phases and you need an automated tool

Rule 3 Use the second tool for testing if the cost limit is low or average and the

tool is supported by the system usage metric or the intractability metric

Rule 4 Use the second tool for testing if you can afford high cost for testing

either in the requirement or the design phases and you need a manual tool

Rule 5 Use the third tool for testing if the you are limited to low cost and the

tool is supported by the error rate metric

Rule 6 Use the third tool for testing if you can afford very-high cost for testing

either in the requirement or the system usage phases and you need a semishy

automated tool

44

Table 3-2 presents information on these rules and the disjointness values for all attributes For

each class the row marked Values lists values occurring in the ruleset for this class For

evaluating the disjointness of an attribute say A each rule in the ruleset above that does not

contain attribute A is characterized as having an additional condition [A= a vb ] where a b

are all legal values of A

The row Class disjointness specifies the class disjointness for each attribute The attribute xl

has the highest disjointness (11) and is assigned to the root of the tree For simplicity assume

the tolerances for each elementary criterion equal O

From the rules in Figure 3-6 we can also determine disjoint groupings of attribute-values used

for the compact mode of the algorithm This done as follows 1) determine for each attribute the

sets of values that the attribute takes in individual decision rules and remove those value sets

that subsume other value sets The remaining value sets are assigned to branches stemming from

the node marked by the given attribute For example x 1 has the following value sets in the

individual decision rules 2 3 I 2 I and 4 (Figure 3-6) Value set I 2 is

removed as it subsumes 2 and I In this case branches are assigned individual values of the

domain of x1 For attribute x2 the value sets are I 2 34 and I 2 3 4 In this case

branches are assigned value sets I 2 and 34

45

Attribute Xl ranks highest (as it is the highest disjointness) and is assigned to the root of the

tree Four branches are created each one corresponding to one of XIs possible values Since all

rules containing [x1=4] belong to class TI the branch marked by 4 is ended by a leafTI Rules

containing other values of X1 belong to more than one class This process is repeated for each

subset of rules until the decision tree is completed Figure 3-7 shows a decision structure learned

by AQDT-2 from the rules in Figure 3-6 The decision structure in Figure 3-7 can be used in

making decisions on which tools can be used for testing a given software

xl

Complexity No of nodes 4 No of leaves 7

Figure 3-7 A decision structure learned for classifying so tware testing tools

Figure 3-8-a shows the diagrammatic visualization of the decision rules and Figure 3-8-b shows

the visualization of the derived decision tree Each diagram in Figure 3-8 consists of cells

representing one combination of attribute-values Attributes and their legal values are shown on

scales surrounding the diagram (eg the horizontal scale for xl shows values 1 2 3 and 4)

Rules are represented by collections of cells in the intersection of the rows and columns

corresponding to the conditions in the rules

The shaded areas correspond to decision rules Rules of the same class have the same type of

shading Empty cells correspond to combination of attribute-values not assigned to any class For

illustration collections of cells corresponding to some of the initial rules are marked R 11 R 21

R31 and R32 Rll denotes the IlfSt rule of Class T1 ie [x1=2] amp [x2=2] R21 denotes the

IlfSt rule of class T2 ie [xl= 1 v 2] amp [x2= 3 v 4] R31 the first rule of class TI ie [x1=1]

46

amp [x2 =1] and denotes R32 denotes the second rule of class T3 Le [xl=4] amp [x3=2 v 3] amp

[x4=3]

a) Decision rules b) Derived decision tree

Comparing diagrams in Figure 3-8-a and 3-8-b AQDT-2 decision tree represents a slightly more

general description of Concepts Tl TI and T3 than the original rules Let us assume that it is

very costly to know which metrics suppon the required tools In other words suppose that we

would like to select the best tools independently of the metric they suppon (this is indicated to

AQDT-l by assigning very high cost to attribute x2) The algorithm again assigns attribute xl to

the root of the tree When xl takes value 1 or 2 it is impossible to assign a specific decision

without measuring attribute x2 For the value 1 of xl the recommended tool can be either TI or

T3 (see the diagram in Figure 3-9-a) and for the value 2 of xl the recommended tool can be

either Tl or T3 However for the value 3 of xl one can make a specific decision after

measuring attribute x4 For the value 1 of x4 tool is Tl and for the value 2 of x4 the

recommended tool is TI Figure 3-9-b shows another decision tree where the type of the tool

was ignored from the data Those decision trees are called indeterminate because some of their

leaves are assigned a disjunction of two or more class names

47

xl

xl

1 12

a) Ignoring the supporting metric b) Ignoring the type of the tooL Figure 3 -9 Decision trees learned ignoring the support metric and the type of the testing tool

It is clear that the given set of rules depend highly on amount of money can be spent Let us now

suppose that the cost attribute xl which was determined as the highest ranked attribute cannot

be measured The algorithm selected x4 to be the root of the new decision tree After continuing

the algorithm the tree in Figure 3-10 is obtainedThe nodes that are assigned one class indicate

situations in which it is possible to make a specific decision without knowing the value of

attribute xl

x4

Figure 3-10 A decision tree learned without the cost attribute

34 Tailoring Decision Structures to the Decision-making situation

Decision structures are among the simplest structures for organizing a decision-making process

A decision structure specifies explicitly the order in which attributes of an object or situation

need to be evaluated in the process of determining a decision A standard way to generate a

48

decision structure is to learn it from examples of decisions Such a process usually aims at

obtaining a decision structure that has the highest prediction accuracy that is the highest rate of

assigning correct decisions to given situations There can usually be a large number of logically

equivalent decision structures (Michalski 1990) As such they may have the same predictive

accuracy but differ in the way they organize the decision process and thus may differ in the cost

of arriving at a decision To minimize the average decision cost one needs to take into

consideration the distribution of the costs of attribute evaluation and the frequency of different

decisions This report presents an approach to building such Task-oriented decision structures

which advocates that they are built not from examples but rather from decision rules Decision

rules are learned from examples using the AQ15 or AQ17 inductive learning program or are

specified by an expert An efficient algorithm and a new system AQDT-2 that transforms

decision rules to task-oriented decision structures The system is illustrated by applying it to the

problem of learning decision structures in the area of construction engineering (for determining

the best wind bracing for tall buildings) In the experiments AQDT-2 outperformed all other

programs applied to the same data

Decision-making situations can vary in several respects In some situations complete

information about a data item is available (ie values of all attributes are specified) in others the

information may be incomplete To reflect such differences the user specifies a set of parameters

that control the process of creating a decision structure from decision rules AQDT-2 provides

several features for handling different decision-making problems 1) generating a decision

structure that tries to avoid unavailable or costly attributes (tests) 2) generating unknown

leaves in situations where there is insufficient information for generating a complete decision

structure 3) providing the user with the most likely decision when performing a required test is

impossible 4) providing alternative decisions with an estimate of the likelihood of their

correctness when the needed information can not be provided

49

341 Learning Costmiddot Dependent Decision Structures

As described in Sec 2 the LEF criterion enables the system to take into consideration the cost of

measuring tests (attributes) in developing a decision structure In the default setting of LEF the

tests cost is the [lIst criterion and its tolerance is O This means that only the least expensive

attributes pass to the next step of evaluation involving other elementary criteria If an attribute

has high cost or is impossible to measure (has an infinite cost) the LEF chooses another

cheaper attribute ifpossible

342 Assigning Decision Under Insufficient Information

In decision-making situations in which one or more attributes cannot be measured the system

may not be able to assign a definite decision for some cases If no more information can be

obtained but a decision has to be made it is useful to know the probability distribution for

different candidate decisions (Smyth Goodman amp Higgins 1990) The most probable decision is

then chosen The probability distribution can be estimated from the class frequency at the given

node Let us consider a node in the structure connected to the root by the sequence bl b2 bkof

attribute-values Let P(Ci I bl b2 bk) denote the conditional probability of class Ci i=I2 m

at that node given that an example to be classified has attribute-values assigned to branches bl

b2 bull bk in the decision structure Using Bayesian formula we have

P(Ci I bl~ bull bk) = P(Ci) P(bl bk I Ci) P(bl bk) (3-9)

where P(Ci) the apriori probability of class Ci P(bl bk I Ci) is the probability that the

example has attribute-values blbull bk given that it represents decision class Ci To approximate

these probabilities let us suppose that Wi is the number of training examples of Class Ci that

passed the tests leading to this node and twi - the total number of training examples of Class

Ci Let us use these numbers to estimate the probabilities in (9) Assuming that the apriori class

probabilities correspond to the frequency of training examples from different classes we have

50

m P(Ci) = twi IL twj (3-10)

j=l

P(b1 bkl Ci) = Wi I twi (3-11)

m m P(b1 bk) =L Wj IL twj (3-12)

j=l j=1

By substituting (1O) (11) and (12) in (9) we obtain

m P(Ci I b1 bk) = wilL Wj (3-13)

j=l

A related method for handling the problem of unavailability of an attribute is described by

Quinlan (1990) Quinlans method however assigns probabilities to the most probable decision

at the node associated with such an attribute The method does not restructure the decision tree

appropriately to fit the given decision-making situation (in this case to avoid measuring xl)

AQDT -2 assigns probabilities o~ly after it first created a decision structure that suits the given

decision-making si~ation An example is presented in section 42

343 Coping with noise in training data

The proposed methodology can be easily extended to handle the problem of learning from noisy

training data This is done based on ideas of rule truncation described in (Niblett amp Bratko 1986

Bergadanoet al 1992 Cestnik amp Karalic 1991) When noise is expected in the training data

the decision rules with t-weight below a certain threshold (reflecting the expected noise level) are

removed The rule truncation method seems to result in more accurate decision structures than

decision tree pruning method because truncation decisions are solely based on the importance of

the given rule or condition for the decision-making regardless of their evaluation order (unlike

the decision tree pruning which can only prunes attributes within a subtree and thus cannot

freely chose attributes to prune) Examples are presented in section 4