chapter – 7shodhganga.inflibnet.ac.in/bitstream/10603/36602/17/17_chapter7.pdf · chapter-7:...
TRANSCRIPT
Chapter-7: Prototype Implementation and Evaluation Page | 150
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
CHAPTER – 7
Prototype Implementation and
Evaluation
7.1 Prototype Implementation of Proposed
System
7.2 Evaluation of Proposed System
7.3 Summary
Chapter-7: Prototype Implementation and Evaluation Page | 151
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
CHAPTER – 7
PROTOTYPE IMPLEMENTATION ANDEVALUATION
This chapter gives conceptual system design used for prototype
implementation and data sets used to evaluate the developed
prototype. It also describes in brief Precision, Recall, and F-Measure,
widely used metrics for evaluating the correctness of such algorithm.
Finally, it provides the analysis of result for the sample data sets while
proposed approach is applied on them in prototype.
7.1 Prototype Implementation of ProposedSystem
A GUI based system, called AI-ATOM, is developed to implement the
proposed algorithm in prototype. As the system is developed in
prototype using Rapid Application Development, the efficiency of the
system is not given a well thought. The main objective is to
demonstrate effectiveness of the proposed algorithm. The following
section describes abstract system requirement specification, system
module chart, database design, class diagram, and sample screens of
user interface of the prototype system.
7.1.1 System Requirement Specification
I. User Requirement (User should be able to):
a. Ontology Management
i. Create New Ontology
ii. View Ontology
iii. Update Ontology
iv. Delete Ontology
Chapter-7: Prototype Implementation and Evaluation Page | 152
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
v. Import Ontology from File
vi. Export Ontology to RDF File
b. Ontology Mapping Project Management
i. Create New Ontology Mapping Project
ii. View Ontology Mapping
iii. Update Ontology Mapping Project
iv. Delete Ontology Mapping Project
v. Assign users to Ontology Mapping Project
c. Mapping Element Management
i. Generate mapping elements
ii. Store mapping elements
iii. View all mapping elements
iv. View suggested mapping elements
v. View un-processed mapping elements
vi. Accept selected mapping elements
vii. Reject selected mapping elements
viii. Add new mapping elements
d. System Settings
i. Auxiliary Resource Setting
Set Context Directory
Set Stop Word List
Set Domain Specific Abbreviation List
Set Domain Specific Synonym List
ii. Threshold Management
Set threshold values for matchers
iii. Algorithm Configuration and Parameter Setting
Enable/Disable matchers
Set execution order of matchers
iv. Matcher Rules Management
Enable/Disable specific matcher rule
Set parameters for matcher rule
Chapter-7: Prototype Implementation and Evaluation Page | 153
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
e. User Management
i. Add user
ii. Delete user
iii. Update user
II. System Requirement (Develop Code Library for):
a. Generating all possible mapping elements by cross joining
classes of both the ontologies
b. Filtering Mapping Elements based on incompatible data
types and constraints.
c. Filtering Mapping Elements based on previously rejected
mappings by user.
d. Reusing previously accepted mappings by user and
eliminate all Mapping Elements involving either entity of
accepted mapping.
e. Jaccard Similarity coefficient
f. Vector Space Model Engine
i. Create feature vector
ii. Calculate VSM similarity score
iii. Select potential mapping elements for further
processing
g. Label Matcher
i. Edit Distance
ii. N-Gram
iii. Prefix
iv. Suffix
v. Soundex
vi. Heuristic Rules
Equality
Vowel less
Right most digit less
Digit less
Chapter-7: Prototype Implementation and Evaluation Page | 154
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Non Alpha Numeric less
Same Character Order
h. Linguistic Matchers
i. Domain Specific Synonym
ii. WordNet Synonym
iii. WordNet Gloss
i. Structure Matchers
i. Path Label Matching
ii. Upward Cotopic Distance
iii. Anchor-Prompt Path Propagation
iv. Heuristic Rules
All children to all children
All children to class label
Parent and some children to parent and some
children
j. Language Processing Activities
i. Spell Corrector
ii. Expand Abbreviation
iii. Remove Stop words
iv. Stemming
v. Get Domain Synonym
vi. Get WordNet Synonym
vii. Get WordNet Gloss
7.1.2 System Module Chart
The Figure 33 shows module hierarchy chart for the majorcomponents of the system and major modules of these components.
Chapter-7: Prototype Implementation and Evaluation Page | 155
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 33: System Module Chart
7.1.3 Data Structure
The Data Structure used by Algorithm is listed below:
User (User ID, User Name, Password)
AI-ATOM
UserManagement
Chapter-7: Prototype Implementation and Evaluation Page | 155
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 33: System Module Chart
7.1.3 Data Structure
The Data Structure used by Algorithm is listed below:
User (User ID, User Name, Password)
Ontology
New Ontology
View and Maintain Ontology
Import From File
OntologyMappingProject
New Ontology Mapping Project
View and Maintain OntologyMapping Project
System Settings
AuxiliaryResource Setting
Set Context Directory
Set Stop Words
Set Abbreviation List
Set Synonym ListThreshold
Management
AlgorithmConfiguration
Matcher RulesManagement
OntologyMappingEngine
Label Matcher
Edit Distance
Label Huerstic RulesVSM
LanguageProcessing
LinguisticMatcher
get Domain Synonym
get WordNet Synonym
get WordNet Gloss
StructureMatcher
Upward Cotopic Distance
Anchor-Prompt
Structure Heuristic RulesUserManagement Add User
Update User
Delete User
Chapter-7: Prototype Implementation and Evaluation Page | 155
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 33: System Module Chart
7.1.3 Data Structure
The Data Structure used by Algorithm is listed below:
User (User ID, User Name, Password)
Set Context Directory
Set Stop Words
Set Abbreviation List
Set Synonym List
Edit Distance
N-Gram
Prefix
Suffix
Equality
Label Huerstic Rules
get Domain Synonym
get WordNet Synonym
get WordNet Gloss
Path Label
Upward Cotopic Distance
Anchor-Prompt
Structure Heuristic Rules
Chapter-7: Prototype Implementation and Evaluation Page | 156
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Ontology (Ontology ID, Ontology Name, Description, Owner User
ID, Date of Creation)
Class (Class ID, Class Name, Description, Ontology ID)
Class Has Sub Class (CHSC ID, Class ID, Sub Class ID)
Ontology Mapping Project (OMP ID, OMP Name, Description,
From Ontology ID, To Ontology ID, Created By User ID, Start
Date, Due Date, Completed Date)
Ontology Mapping Project User (OMPU ID, OMP ID, User ID)
Domain Specific Label (DSL ID, Label, Meaning, Frequency of
Use)
Domain Specific Abbreviation (DSA ID, Abbreviation, Expanded
Text)
Domain Specific Synonym (DSS ID, Synonym, Synonym Group
ID)
Stop word (Stop word ID, Stop word)
Algorithm Configuration (AC ID, AC Name, Description, For
OMP ID, Precision, Recall, F Measure)
Parameter (Parameter ID, Parameter Name, Description, Default
Value Text, Default Value Number)
Parameter Possible Value (PPV ID, Parameter ID, Possible Value
Text, Possible Value Number)
Algorithm Parameter Value (APV ID, For AC ID, Parameter ID,
User PPV ID)
Mapping Element (ME ID, OMP ID, From Class ID, From Class
Name, To Class ID, To Class Name, Status, Relation,
Confidence, Explanation, Processed By System YN, Processed
By User YN, User ID)
o Status= Candidate/Potential/System
Generated/Accepted/Rejected
Accepted Rejected Mapping Group (ARMG ID, Class Name,
Relation, Group ID, Accepted or Rejected)
Chapter-7: Prototype Implementation and Evaluation Page | 157
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
The Figure 34 shows the Database Relationship Diagram for the
prototype system.
Figure 34: Database Relationship Diagram
7.1.4 Class Diagram
The Figure 35 and Figure 36 shows sample class diagrams for fewmatchers and language processing activities, and Figure 37 showssample abstract class diagram for the GUI component of the system.
Chapter-7: Prototype Implementation and Evaluation Page | 157
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
The Figure 34 shows the Database Relationship Diagram for the
prototype system.
Figure 34: Database Relationship Diagram
7.1.4 Class Diagram
The Figure 35 and Figure 36 shows sample class diagrams for fewmatchers and language processing activities, and Figure 37 showssample abstract class diagram for the GUI component of the system.
Chapter-7: Prototype Implementation and Evaluation Page | 157
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
The Figure 34 shows the Database Relationship Diagram for the
prototype system.
Figure 34: Database Relationship Diagram
7.1.4 Class Diagram
The Figure 35 and Figure 36 shows sample class diagrams for fewmatchers and language processing activities, and Figure 37 showssample abstract class diagram for the GUI component of the system.
Chapter-7: Prototype Implementation and Evaluation Page | 158
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 35: Sample Class Diagram of the System – I
Chapter-7: Prototype Implementation and Evaluation Page | 159
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 36: Sample Class Diagram of the System - II
Chapter-7: Prototype Implementation and Evaluation Page | 160
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 37: Abstract Class Diagram for GUI of the System
7.1.5 User Interface
7.1.5.1. Sample Screens for Menu Design
Opening Screen
The Figure 38 shows opening screen of the system, which allows the
user to navigate around the system modules with the help of
horizontal main menu.
Figure 38: Opening Screen of the System
Chapter-7: Prototype Implementation and Evaluation Page | 160
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 37: Abstract Class Diagram for GUI of the System
7.1.5 User Interface
7.1.5.1. Sample Screens for Menu Design
Opening Screen
The Figure 38 shows opening screen of the system, which allows the
user to navigate around the system modules with the help of
horizontal main menu.
Figure 38: Opening Screen of the System
Chapter-7: Prototype Implementation and Evaluation Page | 160
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 37: Abstract Class Diagram for GUI of the System
7.1.5 User Interface
7.1.5.1. Sample Screens for Menu Design
Opening Screen
The Figure 38 shows opening screen of the system, which allows the
user to navigate around the system modules with the help of
horizontal main menu.
Figure 38: Opening Screen of the System
Chapter-7: Prototype Implementation and Evaluation Page | 161
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Ontology Management
The Ontology Menu shown in Figure 39 allows the user to perform
tasks specific to ontology management. It allows the user to create
new ontology, to open existing ontology, and to close the currently
opened ontology. It also allows importing ontology from text file.
Figure 39: Ontology Management Screen
Ontology Mapping Project ManagementThe Ontology Mapping Menu shown in Figure 40 allows the user to
perform task specific to ontology mapping project management. It
allows the user to create new ontology mapping project, to open
existing ontology mapping project, and to close the currently opened
ontology mapping project. It also allows assigning users to ontology
mapping project.
Figure 40: Ontology Mapping Project Management Screen
Chapter-7: Prototype Implementation and Evaluation Page | 161
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Ontology Management
The Ontology Menu shown in Figure 39 allows the user to perform
tasks specific to ontology management. It allows the user to create
new ontology, to open existing ontology, and to close the currently
opened ontology. It also allows importing ontology from text file.
Figure 39: Ontology Management Screen
Ontology Mapping Project ManagementThe Ontology Mapping Menu shown in Figure 40 allows the user to
perform task specific to ontology mapping project management. It
allows the user to create new ontology mapping project, to open
existing ontology mapping project, and to close the currently opened
ontology mapping project. It also allows assigning users to ontology
mapping project.
Figure 40: Ontology Mapping Project Management Screen
Chapter-7: Prototype Implementation and Evaluation Page | 161
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Ontology Management
The Ontology Menu shown in Figure 39 allows the user to perform
tasks specific to ontology management. It allows the user to create
new ontology, to open existing ontology, and to close the currently
opened ontology. It also allows importing ontology from text file.
Figure 39: Ontology Management Screen
Ontology Mapping Project ManagementThe Ontology Mapping Menu shown in Figure 40 allows the user to
perform task specific to ontology mapping project management. It
allows the user to create new ontology mapping project, to open
existing ontology mapping project, and to close the currently opened
ontology mapping project. It also allows assigning users to ontology
mapping project.
Figure 40: Ontology Mapping Project Management Screen
Chapter-7: Prototype Implementation and Evaluation Page | 162
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Application Setting and Algorithm Configuration Management
The Setting Menu shown in Figure 41 allows the user to set auxiliary
resources such as context dictionary, acronym dictionary, domain
specific synonym list, and stop word list. It also allows configuring
algorithm, setting threshold values for different matchers, and setting
parameters for matcher rules.
Figure 41: System Setting Screen
7.1.5.2. Sample Screen for Forms Design
Use of Context Dictionary to suggest domain specific labels to user
The Figure 42 shows the benefit of the context dictionary. When user
creates new ontology using system, it suggests the domain specific
labels according to its usage frequency for partially entered label by
user. This reduces the problem of word sense disambiguation.
Depending on his intended meaning, he can directly select word from
suggested list. This helps the ontology mapping process greatly.
Importing Ontology from Text File
The Figure 43 shows the option for importing the ontology from the
text file. It is assumed that ontology tree is stored in this text file
using the parenthesized representation for general tree as shown in
Figure 32. When user selects the file and clicks Open button, ontology
Chapter-7: Prototype Implementation and Evaluation Page | 162
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Application Setting and Algorithm Configuration Management
The Setting Menu shown in Figure 41 allows the user to set auxiliary
resources such as context dictionary, acronym dictionary, domain
specific synonym list, and stop word list. It also allows configuring
algorithm, setting threshold values for different matchers, and setting
parameters for matcher rules.
Figure 41: System Setting Screen
7.1.5.2. Sample Screen for Forms Design
Use of Context Dictionary to suggest domain specific labels to user
The Figure 42 shows the benefit of the context dictionary. When user
creates new ontology using system, it suggests the domain specific
labels according to its usage frequency for partially entered label by
user. This reduces the problem of word sense disambiguation.
Depending on his intended meaning, he can directly select word from
suggested list. This helps the ontology mapping process greatly.
Importing Ontology from Text File
The Figure 43 shows the option for importing the ontology from the
text file. It is assumed that ontology tree is stored in this text file
using the parenthesized representation for general tree as shown in
Figure 32. When user selects the file and clicks Open button, ontology
Chapter-7: Prototype Implementation and Evaluation Page | 162
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Application Setting and Algorithm Configuration Management
The Setting Menu shown in Figure 41 allows the user to set auxiliary
resources such as context dictionary, acronym dictionary, domain
specific synonym list, and stop word list. It also allows configuring
algorithm, setting threshold values for different matchers, and setting
parameters for matcher rules.
Figure 41: System Setting Screen
7.1.5.2. Sample Screen for Forms Design
Use of Context Dictionary to suggest domain specific labels to user
The Figure 42 shows the benefit of the context dictionary. When user
creates new ontology using system, it suggests the domain specific
labels according to its usage frequency for partially entered label by
user. This reduces the problem of word sense disambiguation.
Depending on his intended meaning, he can directly select word from
suggested list. This helps the ontology mapping process greatly.
Importing Ontology from Text File
The Figure 43 shows the option for importing the ontology from the
text file. It is assumed that ontology tree is stored in this text file
using the parenthesized representation for general tree as shown in
Figure 32. When user selects the file and clicks Open button, ontology
Chapter-7: Prototype Implementation and Evaluation Page | 163
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
in imported from the text file on this screen. By clicking on Save
button, user can save this ontology in native form of the system.
Figure 42: Using Context Dictionary to suggest Domain Specific Labels
Figure 43: Importing Ontology from Text File
Chapter-7: Prototype Implementation and Evaluation Page | 163
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
in imported from the text file on this screen. By clicking on Save
button, user can save this ontology in native form of the system.
Figure 42: Using Context Dictionary to suggest Domain Specific Labels
Figure 43: Importing Ontology from Text File
Chapter-7: Prototype Implementation and Evaluation Page | 163
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
in imported from the text file on this screen. By clicking on Save
button, user can save this ontology in native form of the system.
Figure 42: Using Context Dictionary to suggest Domain Specific Labels
Figure 43: Importing Ontology from Text File
Chapter-7: Prototype Implementation and Evaluation Page | 164
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Ontology Mapping Project Management
The Figure 44 shows screen for ontology mapping project
management. It shows both the ontologies considered for mapping
tasks. When user clicks on Generate button, it fills the grid displayed
in the middle of both the ontologies with system generated mappings
as shown in Figure 45. When user clicks on any of this mapping
element, it shows the explanation for this mapping element in text
boxes shown above the Generate button. The user can select mapping
elements using Checkbox against the mapping element in the grid,
and can accept or reject such checked mapping elements. The user
can also add new mapping element by selecting nodes from both the
ontologies and clicking on Add button.
Figure 44: Ontology Mapping Project Management
Chapter-7: Prototype Implementation and Evaluation Page | 164
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Ontology Mapping Project Management
The Figure 44 shows screen for ontology mapping project
management. It shows both the ontologies considered for mapping
tasks. When user clicks on Generate button, it fills the grid displayed
in the middle of both the ontologies with system generated mappings
as shown in Figure 45. When user clicks on any of this mapping
element, it shows the explanation for this mapping element in text
boxes shown above the Generate button. The user can select mapping
elements using Checkbox against the mapping element in the grid,
and can accept or reject such checked mapping elements. The user
can also add new mapping element by selecting nodes from both the
ontologies and clicking on Add button.
Figure 44: Ontology Mapping Project Management
Chapter-7: Prototype Implementation and Evaluation Page | 164
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Ontology Mapping Project Management
The Figure 44 shows screen for ontology mapping project
management. It shows both the ontologies considered for mapping
tasks. When user clicks on Generate button, it fills the grid displayed
in the middle of both the ontologies with system generated mappings
as shown in Figure 45. When user clicks on any of this mapping
element, it shows the explanation for this mapping element in text
boxes shown above the Generate button. The user can select mapping
elements using Checkbox against the mapping element in the grid,
and can accept or reject such checked mapping elements. The user
can also add new mapping element by selecting nodes from both the
ontologies and clicking on Add button.
Figure 44: Ontology Mapping Project Management
Chapter-7: Prototype Implementation and Evaluation Page | 165
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
System Generated Mapping Elements
Figure 45: System Generated Mapping Elements
System Setting and Configuration
The Figure 46 shows the System Setting screen. It allows the user to
do following type of system settings
Enable or disable specific feature of the algorithm such as
whether to reuse previously accepted mappings or not
Enable or disable specific matcher and sub matcher
Enable or disable the use of threshold value for a specific
matcher
To specify threshold value for a specific matcher
To set execution order of the matchers
Chapter-7: Prototype Implementation and Evaluation Page | 165
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
System Generated Mapping Elements
Figure 45: System Generated Mapping Elements
System Setting and Configuration
The Figure 46 shows the System Setting screen. It allows the user to
do following type of system settings
Enable or disable specific feature of the algorithm such as
whether to reuse previously accepted mappings or not
Enable or disable specific matcher and sub matcher
Enable or disable the use of threshold value for a specific
matcher
To specify threshold value for a specific matcher
To set execution order of the matchers
Chapter-7: Prototype Implementation and Evaluation Page | 165
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
System Generated Mapping Elements
Figure 45: System Generated Mapping Elements
System Setting and Configuration
The Figure 46 shows the System Setting screen. It allows the user to
do following type of system settings
Enable or disable specific feature of the algorithm such as
whether to reuse previously accepted mappings or not
Enable or disable specific matcher and sub matcher
Enable or disable the use of threshold value for a specific
matcher
To specify threshold value for a specific matcher
To set execution order of the matchers
Chapter-7: Prototype Implementation and Evaluation Page | 166
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 46: System Setting Screen
7.2 Evaluation of proposed system
7.2.1 Evaluation Measure
The effectiveness of the system in processing the mapping elements ismeasured by looking at Precision and Recall [67] [24]. The ontologymapping systems are evaluated with respect to the notion ofcorrectness perception – a judgment by a human that a mappingelement found by ontology mapping algorithm is correct or not.
A system’s ability to retrieve correct mapping elements is assessedwith a Recall measure that is defined as below:
Recall = | Relevant and Retrieved | / | Relevant |
A system can achieve 100% recall by simply returning all the possiblemapping elements between two ontologies.
A system’s accuracy is based on how many of the mapping elementsgenerated by system are actually correct as per user’s decision, whichcan be assessed by a Precision metric and is defined below.
Precision = | Relevant and Retrieved | / | Retrieved |
Chapter-7: Prototype Implementation and Evaluation Page | 166
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 46: System Setting Screen
7.2 Evaluation of proposed system
7.2.1 Evaluation Measure
The effectiveness of the system in processing the mapping elements ismeasured by looking at Precision and Recall [67] [24]. The ontologymapping systems are evaluated with respect to the notion ofcorrectness perception – a judgment by a human that a mappingelement found by ontology mapping algorithm is correct or not.
A system’s ability to retrieve correct mapping elements is assessedwith a Recall measure that is defined as below:
Recall = | Relevant and Retrieved | / | Relevant |
A system can achieve 100% recall by simply returning all the possiblemapping elements between two ontologies.
A system’s accuracy is based on how many of the mapping elementsgenerated by system are actually correct as per user’s decision, whichcan be assessed by a Precision metric and is defined below.
Precision = | Relevant and Retrieved | / | Retrieved |
Chapter-7: Prototype Implementation and Evaluation Page | 166
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 46: System Setting Screen
7.2 Evaluation of proposed system
7.2.1 Evaluation Measure
The effectiveness of the system in processing the mapping elements ismeasured by looking at Precision and Recall [67] [24]. The ontologymapping systems are evaluated with respect to the notion ofcorrectness perception – a judgment by a human that a mappingelement found by ontology mapping algorithm is correct or not.
A system’s ability to retrieve correct mapping elements is assessedwith a Recall measure that is defined as below:
Recall = | Relevant and Retrieved | / | Relevant |
A system can achieve 100% recall by simply returning all the possiblemapping elements between two ontologies.
A system’s accuracy is based on how many of the mapping elementsgenerated by system are actually correct as per user’s decision, whichcan be assessed by a Precision metric and is defined below.
Precision = | Relevant and Retrieved | / | Retrieved |
Chapter-7: Prototype Implementation and Evaluation Page | 167
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Ideally both should be 100%. But, most of the system scarifies one forthe other. Hence, to measure the optimum balance between these twomeasures, F-Measure is used which is defined as below:
F-Measure = 2 * (Precision * Recall) / (Precision + Recall)
The Precision and Recall can be understood from the Figure 47, Where:
A = False Positives B = True Positives C = False Negatives D = True Negatives.
Figure 47: Precision, Recall, and F-Measure
Using these notions; Precision, Recall, and F-Measure can be definedas following.
Recall = | Relevant and Retrieved | / | Relevant |
= B / (B+C)
Precision = | Relevant and Retrieved | / | Retrieved |
= B / (A+B)
F-Measure = 2 * (Precision * Recall) / (Precision + Recall)
= 2 * B/ ((A+B) + (B+C))
System
Matches
User
Matches
D
A B C
Chapter-7: Prototype Implementation and Evaluation Page | 168
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
7.2.2 Data Sets used for evaluation
The system is evaluated with two data sets, which are shown in Figure
48 and Figure 49. The first data set represents sample ontologies
represented by two different academic institutes, whereas second data
set represents snapshot of database schema from two academic
institutes.
Figure 48: Data Set-1: Sample Ontologies from two different Academic
Institutes
Both the data sets are selected from the academic domain as one of
the objectives of study is to analyze and to present the significance of
domain knowledge in the automated ontology mapping process.
Chapter-7: Prototype Implementation and Evaluation Page | 169
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 49: Data Set-2: Sample Database Schema from two different
Academic Institutes
In experimental setup, these data sets are given to few users having
varying knowledge regarding the ontology mapping. The information
about manual mappings identified by different users for above two
data sets are aggregated and summarized in the Table 6.
Table 6: User Performance for example Data sets
DataSet
UserPerformance
FP TP FN P R FM
DS-1 Actual 0 23 0 100 100 100
Maximum 12 23 6 88.46 100 93.88
Minimum 3 17 0 58.62 73.91 65.38
Average 6 19.33 3.67 77.60 84.06 80.36
DS-2 Actual 0 13 0 100 100 100
Maximum 5 12 4 100 92.31 88.89
Minimum 0 9 1 64.29 69.23 72
Average 2.30 10 2.30 83.33 81.12 80.90FP=False Positive, TP=True Positive, FN=False Negative, P=Precision, R=Recall, FM=F-Measure
Chapter-7: Prototype Implementation and Evaluation Page | 170
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
7.2.3 Analysis of Result
The basic objective of work is to improve automation of ontology
mapping process which yields quality mappings. Thus, system is
tested with different algorithm configuration and parameter settings
for its effectiveness only. The efficiency of the system with respect to
memory and/or CPU is not given due importance for this prototype
system. The following section describes experimental study performed
for the prototype system.
7.2.3.1 Overall Performance of the System
The Figure 50 and Figure 51 represent overall performance of the
system with actually expected number of mappings by the creator of
the ontology. It is observed that system offers reasonably good
performance without any fine tuning of the algorithm. It provides
100% precision for one of the data set. It proves the system’s potential
ability to be used as a completely automated system.
Figure 50: Overall System Performance for Data set - 1
Precision Recall F-Measure
Actual 100.00 100.00 100.00
System 100.00 43.48 60.61
0.00
20.00
40.00
60.00
80.00
100.00
120.00
Perf
orm
ance
in %
Graph-1.1: Overall SystemPerformance
Actual
System
Chapter-7: Prototype Implementation and Evaluation Page | 171
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 51: Overall System Performance for Data set - 2
7.2.3.2 System Performance against User
The comparative study between system and user for the selected data
sets are depicted in the Figure 52 and Figure 53. The study shows
that system can be rated as better when it is compared with user
driven manual activity for the establishment of ontology mappings for
a given data sets. That is, system gives F-Measure value of 65.54%
against actual result. But, when system’s average performance (not
best) is compared to user’s average performance; it gives F-Measure
value of 71.49%, which may be considered as good result.
7.2.3.3 System Performance for different Threshold Values
The system is run for five threshold values 0.6, 0.7, 0.8, 0.9, and 1.0.
The performance of system for these threshold values for data set-1
and data set-2 is shown in the Figure 54 and Figure 55 respectively. It
is observed that precision of the system increases when threshold
value is set to high value for both the data sets.
Precision Recall F-Measure
Actual 100.00 100.00 100.00
System 55.56 76.92 64.52
0.0020.0040.0060.0080.00
100.00120.00
Perf
orm
ance
in %
Measures
Graph-2.1: Overall SystemPerformance
Actual
System
Chapter-7: Prototype Implementation and Evaluation Page | 172
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 52: System Performance against User for Data set - 1
Figure 53: System Performance against User for Data set - 2
The Recall and F-Measure slightly decreases for data set-1, and
decreases moderate for data set-2. It also reveals that as the demand
for precision increases, the recall of the system decreases, and vice
versa. Another interesting effect of setting high threshold is that it
Precision RecallF-
Measure
User Average 77.60 84.06 80.36
System Average 90.00 42.61 57.45
System against User 115.98 50.69 71.49
0.0020.0040.0060.0080.00
100.00120.00140.00
Perf
orm
ance
in %
Graph-1.2: System Performance AgainstUser
User Average
System Average
System against User
Precision RecallF-
Measure
User Average 83.33 81.12 80.90
System Average 65.02 52.31 56.14
System against User 78.02 64.48 69.39
0.0010.0020.0030.0040.0050.0060.0070.0080.0090.00
Perf
orm
ance
in %
Measures
Graph-2.2: System Performance AgainstUser
User Average
System Average
System against User
Chapter-7: Prototype Implementation and Evaluation Page | 173
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
yields 100% precision which helps for total automation of the ontology
mapping process.
Figure 54: System Performance for different Threshold Values (DS - 1)
Figure 55: System Performance for different Threshold Values (DS - 2)
0.60 0.70 0.80 0.90 1.00
Threshold
Precision 83.33 66.67 100.00 100.00 100.00
Recall 43.48 43.48 43.48 43.48 39.13
F-Measure 57.14 52.63 60.61 60.61 56.25
0.0020.0040.0060.0080.00
100.00120.00
Perf
orm
ance
in %
Graph-1.3: The effect of differentThreshold Values on Performance
Precision
Recall
F-Measure
0.60 0.70 0.80 0.90 1.00
Threshold
Precision 55.56 66.67 60.00 71.43 71.43
Recall 76.92 61.54 46.15 38.46 38.46
F-Measure 64.52 64.00 52.17 50.00 50.00
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
Perf
orm
ance
in %
Graph-2.3: The effect of differentThreshold Values on Performance
Precision
Recall
F-Measure
Chapter-7: Prototype Implementation and Evaluation Page | 174
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
7.2.3.4 The effect of Language Processing Activities on
System Performance
The prototype system developed implements several language
processing activities. The effect of using/performing selected
language-processing activities, viz., removal of stop words, stemming
the words, and spelling correction; on system performance for data
set-1 and data set-2 is shown in Figure 56 and Figure 57 respectively.
It is observed that all language processing activities improves the
precision of the system. At the same time there is no adverse effect on
the F-Measure, except that of stemming for the data set-1.
Figure 56: The effect of Language Processing Activities on System
Performance for Data set - 1
With AllLanguageProcessing
WithoutStop words
WithoutStemming
WithoutSpell
Corrector
Precision 100.00 100.00 100.00 100.00
Recall 43.48 39.13 52.17 39.13
F-Measure 60.61 56.25 68.57 56.25
0.00
20.00
40.00
60.00
80.00
100.00
120.00
Perf
orm
ance
in %
Graph-1.4: The effect of LanguageProcessing Activities on Performance
Precision
Recall
F-Measure
Chapter-7: Prototype Implementation and Evaluation Page | 175
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 57: The effect of Language Processing Activities on System
Performance for Data set - 2
7.2.3.5 The effect of Auxiliary Resources on System
Performance
The system used the concept of semantic similarity with the help of
domain specific knowledge and WordNet knowledge. The fig- and fig-
depicts the effect of excluding the specific type of external resources
for the data set-1 and data set-2 respectively. It shows that F-measure
is highest when all auxiliary resources are used. For data set-2, the F-
Measure reduces to almost 50% when none of the auxiliary resources
is used. This signifies the importance of Linguistic Matcher used by
the system.
With AllLanguageProcessing
WithoutStop words
WithoutStemming
WithoutSpell
Corrector
Precision 75.00 55.56 62.5 62.5
Recall 69.23 38.46 76.92307692 38.46153846
F-Measure 72.00 45.45 68.96551724 47.61904762
0.0010.0020.0030.0040.0050.0060.0070.0080.0090.00
Perf
orm
ance
in %
Graph-2.4: The effect of LanguageProcessing Activities on Performance
Precision
Recall
F-Measure
Chapter-7: Prototype Implementation and Evaluation Page | 176
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
Figure 58: The effect of Auxiliary Resources on System Performance for
Data set - 1
Figure 59: The effect of Auxiliary Resources on System Performance for
Data set - 2
WithAuxiliary
Resources
WithoutDomain
Knowledge
WithoutWordNet
WithoutAuxiliary
Resources
Precision 100.00 100.00 100.00 100.00
Recall 43.48 39.13 34.78 30.43
F-Measure 60.61 56.25 51.61 46.67
0.00
20.00
40.00
60.00
80.00
100.00
120.00Pe
rfor
man
ce in
%
Graph-1.5: The effect of AuxiliaryResources on Performance
Precision
Recall
F-Measure
WithAuxiliary
Resources
WithoutDomain
Knowledge
WithoutWordNet
WithoutAuxiliary
Resources
Precision 75.00 100.00 100.00 100.00
Recall 69.23 23.08 30.77 23.08
F-Measure 72.00 37.50 47.06 37.50
0.00
20.00
40.00
60.00
80.00
100.00
120.00
Perf
orm
ance
in %
Graph-2.5: The effect of AuxiliaryResources on Performance
Precision
Recall
F-Measure
Chapter-7: Prototype Implementation and Evaluation Page | 177
An Integrated Approach to Improve Ontology Mapping Process in Semantic Web
The values shown in all the graphs shown here is obtained by keeping
all other parameter constant just to see the effect of specific
parameter. Moreover, a particular parameter setting is applied in
totality here. For example, if stemming is disabled, it is disabled for all
matchers and sub matchers. During trial run of the system it is
observed that this does not give the optimum performance. For
example, the N-Gram similarity improves the result if stemming is
used on class label before they are passed to it. Similarly, the effect of
different parameters is measured with constant threshold values, 0.7
for data set-1 and 0.8 for data set-2. Moreover, this threshold value is
used uniformly for all matcher and sub matchers. Using different
threshold values for different matchers and sub matcher may reveal
other interesting behavior of the system.
7.3 Summary
This chapter presented comparative evaluation of the proposed
integrated approach against user driven manual mapping activity. The
effectiveness of the proposed system is analyzed with two data sets
from an academic domain. The proposed algorithm is highly
configurable with many possibilities of combining different parameter
values. Some selected combination is used to test the performance of
the system. In best case, algorithm yields extremely encouraging
result. Though, the algorithm is yet to be tested for their optimum
configuration setting which is a gigantic task. The engineering
contribution is made by developing a prototype system, which can be
further used or extended to add scientific contribution.