evaluation of ontologies

Evaluation of OntologiesAsuncion Gomez-Perez*´ ´ ´Facultad de Informatica, Universidad Politecnica de Madrid, Campus de´ ´Montegancedo sn., Boadilla del Monte, 28660, Madrid, Spain

The evaluation of ontologies is an emerging field. At present, there is an absence of adeep core of preliminary ideas and guidelines for evaluating ontologies. This paperpresents a brief summary of previous work done on evaluating ontologies and the criteriaŽ .consistency, completeness, conciseness, expandability, and sensitiveness used to evalu-ate and to assess ontologies. It also addresses the possible types of errors made whendomain knowledge is structured in taxonomies in an ontology and in knowledge bases:circularity errors, exhaustive and nonexhaustive class partition errors, redundancy errors,grammatical errors, semantic errors, and incompleteness errors. It also describes theprocess followed to evaluate the standard-units ontology already published at theOntology Server. � 2001 John Wiley & Sons, Inc.

I. INTRODUCTION

During recent years, considerable progress has been made in developingthe conceptual bases for building technology that allows knowledge reuse andsharing. Ontologies generally specify conceptualizations with a high degree offormality. However, the process of building ontologies is often anarchisticbecause each development team usually follows its own set of principles, designcriteria, and steps in the ontology development process.

Until now, few domain-independent methodological approaches4,7,15,19 havebeen reported for building ontologies. These methodologies have in commonthat they start from the identification of the purpose of the ontology, the needfor domain knowledge acquisition, and the need for ontology evaluation. How-ever, evaluation is performed differently in each one. Uschold’s methodology19

includes the evaluation activity but does not state how it is to be carried out.Gruninger and Fox15 propose to evaluate the ontology by identifying a set of¨

* e-mail: [email protected]; [email protected] grant sponsor: Ministerio de Educacion y Ciencia in Spain.Ćontract grant number: PF94-9921929.

Ž .Contract grant sponsor: Comision Interministerial de Ciencia y Tecnologıa CICYT´ ín Spain.

Contract grant number: TIC 96-1226-E.Contract grant sponsor: Universidad Politecnica de Madrid.Ćontract grant number: AM98-19.

Ž .INTERNATIONAL JOURNAL OF INTELLIGENT SYSTEMS, VOL. 16, 391�409 2001� 2001 John Wiley & Sons, Inc.

´ ´GOMEZ-PEREZ392

competency questions, which are the basis for a rigorous characterization of theknowledge that the ontology has to cover. Finally, METHONTOLOGY4,7 pro-poses that evaluation be carried out throughout the entire lifetime of theontology development process. It proposes that most of the evaluation becarried out during the conceptualization phase to prevent errors and theirpropagation in the implementation phase.

However, the ontological engineering community is lack of interest inevaluation issues. There exist: a lack of mechanisms to evaluate knowledgesharing technology in general; a lack of specific methods for evaluating existingontologies, and ontologies under construction; a lack of papers describing how

Ž 16very well-known and large ontologies Cyc Ontologies, ontologies at the3 18 .Ontology Server, SENSUS etc. have been evaluated before being made

public; there is no deep knowledge of how tools for building ontologies performevaluation; and a lack of application-dependent and end-user methods to judgethe usability and utility of an ontology to be used in an application. All of themare obstacles to their use in companies.

Ž .Over the years, the knowledge-based systems KBS verification and valida-tion community has developed a wide range of methods, techniques, and toolsfor verifying and validating KBS whose KB is formalized as rules. However,ontologies are formalized using concepts organized in taxonomies, instances,relations, functions, and axioms in languages like Ontolingua,13 CycL,16 LOOM,17

etc. Therefore, we need specific methods for evaluating them.Since the evaluation of ontologies is an emerging field and there is an

absence of a deep core of preliminary ideas, this paper is organized in two parts.The first part offers a brief review of the work done on evaluating knowledge

Ž . Ž .sharing technology Section II , ontologies Section III , and assessing ontologiesŽ .Section IV . The second part presents the evaluation of taxonomic knowledge

Ž . 13in ontologies Section V built using primitives from the frame ontology and12 Ž .the evaluation of the standard-units ontology Section VI , both of which are

available at the Ontology Server. This paper does not propose to show how toevaluate an ontology as it is built. Its goal is:

Ž .1 To present what types of errors should be avoided when building taxonomicknowledge into an ontology under a frame-based approach or, from anotherviewpoint, the errors to be looked for if an existing ontology is to be reused inanother application or ontology.

Ž .2 To show what activities should be performed to evaluate an existing ontologybefore its use by other ontologies and other applications. The standard-unitsontology was judged bearing in mind its future use by the chemical-elementsontology.4

II. EVALUATION OF KNOWLEDGE SHARING TECHNOLOGY

A study of KBS evaluation ideas served as a precedent for the evaluation ofontologies to learn from KBS successes and mistakes.11 Briefly, the main ideas

Ž .taken from this study to prepare a framework for ontology evaluation are: 1division into two evaluation types: technical evaluation, carried out by develop-

EVALUATION OF ONTOLOGIES 393

Ž .ers, and user evaluation; 2 provision of a set of terms and standard definitionsŽ .of such terms; 3 definition of a set of criteria to carry out the technical and

Ž .user evaluation processes; 4 inclusion of evaluation activities in methodologiesŽ .for building ontologies; 5 construction of tools for evaluating existing ontolo-

Ž .gies; and 6 inclusion of evaluation modules in tools used to build ontologies.With regard to terminology and definitions of terms in the knowledge

Žsharing technology field, the main terms borrowed from the KBS evaluation. 9field are : ‘‘evaluation,’’ ‘‘verification,’’ ‘‘validation,’’ and ‘‘assessment.’’E�aluation of knowledge sharing technology means to judge the ontologies

Ž .and problem solving methods PSMs , their software environments, and docu-mentation technically against a reference framework during each phase andbetween phases of the life cycle. Examples of reference frameworks are the realworld, a set of requirements, or a set of competency questions. The termevaluation subsumes verification and validation.

Verification of knowledge sharing technology refers to the technical activitythat guarantees the correctness of an ontology and PSMs, their associatedsoftware environments and documentation with respect to a reference frame-work during each phase and between phases of the life cycle.

Validation of knowledge sharing technology guarantees that the ontologiesand PSMs, their software environments, and documentation correspond to thesystems that they are supposed to represent.

Assessment of knowledge sharing technology refers to the usability andusefulness of the ontologies and PSMs, their software environments, and theirdocumentation when they are reused or shared in applications.

In relation to the criteria for evaluating knowledge sharing technology, afew criteria and examples of evaluation of class definitions, hierarchies andrelations, and axioms appear in Refs. 10 and 8. With regard to the tools, the

Ž .ontology server provides a parser for legal knowledge interchange format KIFsentences. It analyzes whether definitions are well formed and generates areport on undefined concepts and intra-ontology dependencies.

III. EVALUATION OF ONTOLOGIES

The evaluation of ontologies8 refers to the correct building of the contentŽof the ontology, that is, ensuring that its definitions a definition is written in

.natural language and in a formal language correctly implement ontologyrequirements and competency questions or perform correctly in the real world.

Ž .The goal is to prove compliance of the world model if it exists and is knownwith the world modeled formally. Ontology evaluation includes:

� Each individual definition and axiom.� Collection of definitions and axioms that are stated explicitly in the ontology.� Definitions that are imported from other ontologies.� Definitions that can be inferred using other definitions and axioms.

´ ´GOMEZ-PEREZ394

The goal of the evaluation process is to determine what the ontology definescorrectly, does not define, or even defines incorrectly. We also have to look atthe scope of the definitions and axioms by figuring out what can be inferred,cannot be inferred, or can be inferred incorrectly. To evaluate a given ontology,the following criteria were identified: consistency, completeness, conciseness,expandability, and sensitiveness. For examples that show how to deal with thesecriteria, see Refs. 10 and 8.

Consistency refers to whether it is possible to obtain contradictory conclu-sions from valid input definitions. A given definition is consistent if and only ifthe individual definition is consistent and no contradictory sentences can beinferred using other definitions and axioms.

� A given definition is individually consistent if and only if:

� The formal definition is metaphysically consistent, that is, if there is nocontradiction in the interpretation of the formal definition with respect to the

Žreal world. The goal is to prove compliance of the world model if it exists and.is known with the world modeled formally.

� The informal definition is metaphysically consistent, that is, if there is nocontradiction in the interpretation of the informal definition with respect to thereal world.

� The entire definition is internally consistent, that is, the formal and informaldefinition have the same meaning.

� A definition is inferentially consistent if it is impossible to obtain contradictoryconclusions using the meaning of all the definitions and axioms in the ontology,and the ontologies included by this ontology.

Completeness. Incompleteness is a fundamental problem in ontologies. Infact, we cannot prove either the completeness of an ontology or the complete-ness of its definitions, but we can prove both the incompleteness of an individualdefinition, and thus deduce the incompleteness of an ontology, and the incom-pleteness of an ontology if at least one definition is missing with respect to theestablished reference framework. So, an ontology is complete if and only if:

� All that is supposed to be in the ontology is explicitly set out in it, or can beinferred.

� Ž .Each definition is complete. This is determined by figuring out: a what knowl-Ž .edge the definition defines or does not explicitly define about the world; and b

for all the knowledge that is required but not explicit, check whether it can beinferred using other definitions and axioms. If it can be inferred, the definition iscomplete. Otherwise, it is incomplete.

To provide a mechanism to evaluate completeness, the following activitiescan be of assistance in finding incomplete definitions.

� Check completeness of the class hierarchy. Errors appear when: the superclassesof a given class are imprecise or overspecified, and when information about


subclasses that are subclass partition† or about exhaustive subclass partitions‡ ismissing.

� Check the completeness of the domains and ranges of the functions and relations.The goal is to figure out whether the domain and range of each argument of eachfunction or relation exactly and precisely delimits the classes that are appropriatefor that argument. Errors appear when the domains and ranges are imprecise oroverspecified.

� Check the completeness of the classes. The aim is to ascertain whether the classcontains as much information as required. Errors appear when: there are proper-ties missing in the definition of a class, when different classes have the sameformal definition, etc.

Conciseness. An ontology is concise if it does not store any unnecessary oruseless definitions, if explicit redundancies do not exist between definitions, andredundancies cannot be inferred using other definitions and axioms.

Expandability refers to the effort required in adding new definitions to anontology and more knowledge to its definitions, without altering the set ofwell-defined properties that are already guaranteed.

Sensiti�eness relates to how small changes in a definition alter the set ofwell-defined properties that are already guaranteed.

IV. ASSESSMENT OF ONTOLOGIES

Today, it is easy to get information about organizations that have ontologiesusing the WWW. There are even specific points that gather information aboutontologies and have links to other web pages containing more explicit informa-tion about such ontologies: The Ontology Page§ and the yellow pages of

�1ontologies. Additionally, there are ontology servers, like the OntolinguaServer,3 Cycorp’s Upper CYC Ontology Server,16 or Ontosaurus18 that collect ahuge number of very well-known ontologies.

When developers search for candidate ontologies for their application, theyface a complex multicriteria choice problem. Apart from the dispersion of

Ž .ontologies over several servers, a ontology content formalization differs de-Ž .pending on the server at which it is stored, b ontologies on the same server are

Ž .usually described with different detail levels, c there is no common format forpresenting relevant information about the ontologies so that users can decide

Ž .which ontology best suits their purpose, d there is no technical documentdescribing how the ontology was evaluated. Choosing an ontology that does notmatch system needs properly or whose usage is expensive may force future users

† A subclass-partition of a class C is a set of subclasses of C that are mutuallydisjoint.

Ž‡ An exhaustive subclass partition of a class C is a set of mutually disjoint classes a.subclass partition which covers C. Every instance of C is an instance of exactly one of

the subclasses in the partition.§ http:��www.medg.lcs.mit.edu�doyle�top.� http:��delicias.dia.fi.upm.es�OntoAgent�.

´ ´GOMEZ-PEREZ396

to stop reusing the ontology and oblige them to formalize the same knowledgeagain.

So, assessment is focused on judging the understanding, usability, useful-ness, abstraction, quality, and portability of the definitions from the user pointof view, and different kinds of users and different kinds of applications requiredifferent means of assessing an ontology. Some work has been done on identify-ing a set of features to characterize ontologies from the user point of view.1,20

Ž .2 1There is also an ontology-based WWW broker, named ONTO Agent, forselecting ontologies that satisfy a given set of constraints.

For example, before reusing definitions from an ontology to build a KBS,the knowledge engineer should evaluate questions like: Does the ontologydevelopment environment provide methods and tools that help to design thenew knowledge base? Does the ontology server include a tutorial or case studiesthat reduce the cost required to learn and to assimilate ontology content? Byhow much does the ontology reduce the bottleneck in the knowledge acquisitionphase? Does the ontology server have translators that transform the ontologycontent into the target language that I need? How much knowledge is lost inthis translation process? Has the ontology been technically evaluated? Howtrustworthy is the translation of the definitions into the chosen target languageof the KB? Additionally, is it possible to integrate the definitions into the KBwithout making significant modifications to the KB?

V. EVALUATION OF TAXONOMIC KNOWLEDGEIN ONTOLOGIES

Most of the existing ontologies are formalized under a frame-based ap-proach and domain knowledge is structured in taxonomies. This section presentsa set of errors that can be made when building taxonomies. This study is basedon the experience of evaluating ontologies at the Ontology Server, and shouldbe used to prevent these kinds of errors occurring when developing newontologies. Throughout this section, we use the semantics of the primitivesdefined at the frame ontology.13

A. The Frame Ontology Primitives

The Ontology Server is one of the most commonly used tools for ontologydevelopment. Ontology Server ontologies are written in Ontolingua. Ontolinguaontologies were originally written in KIF.5 A knowledge representation ontology,named the frame ontology, was built on KIF. Its goal was to unify the semanticsof the most commonly used primitives in the frame paradigm and to enablefuture ontology developers to develop ontologies using a frame-based approach.When building a taxonomy using the frame ontology vocabulary, the followingprimitives are used:

� Ž .Subclass-of ?child-class ?parent-class . the class ?child-class is a subclass of theclass ?parent-class.


� Ž .Superclass-of ?parent-class ?child-class . the class ?parent-class is a superclass ofthe class ?child-class.

� Ž .Instance-of ?indi�idual ?class . the instance ?individual is an instance of theclass ?class.

� Ž .Class-partition ?set-of-classes . defines a set of disjoint classes. A partition of theclass C into the set of classes class p , . . . , class p , where class p � class p1 n i k� � � �for every i � k is defined when any instance or subclass of any class class pi�cannot belong to any other class class-p . When this occurs, the classeskclass p , . . . , class p are a partition of class C.1 n� �

� Ž .Subclass-partition ?C ?class-partition . defines the set of disjoint subclasses?class-partition as subclasses of the class ?C. This classification does not necessar-ily have to be complete, that is, there may be instances of ?C that are notincluded in any of these subclasses.

� Ž .Exhausti�e-subclass-partition ?C ?class-partition . defines the set of disjoint sub-classes ?class-partition as subclasses of the class ?C, where the class ?C can bedefined as a union of all the classes that make up the partition.

Partitions can define concept classifications in a disjoint and�or completemanner. As exhaustive subclass partitions merely add the completeness con-straint to the established subsets, they have been distinguished as: nonexhaus-tive subclass partition errors and exhaustive subclass partition errors.

B. Errors in Developing Taxonomies

This section presents a set of possible errors that can be made by ontolo-gists when building taxonomic knowledge into an ontology or by knowledgeengineers when building KBs under a frame-based approach. They are classedunder the criteria previously identified in Section III as: inconsistency errors,incompleteness errors, and redundancy errors.

1. Inconsistency Errors

Circularity Errors. They occur when a class is defined as a specialization orgeneralization of itself. Depending on the number of relations involved, circular-

Ž .ity errors can be classed as: circularity errors at distance zero a class with itself ,circularity errors at distance 1 and circularity errors at distance n.

Partition Errors. Partitions can define concept classifications in a disjointand�or complete manner. As exhaustive subclass partitions merely add thecompleteness constraint to the established subsets, subclass partition errors andexhaustive subclass partition errors have been distinguished:

� Subclass partition with common instances. It occurs when one or several instancesbelong to more than one subclass of the defined partition. For example, if dogsand cats form a subclass partition of the set of mammals, an error of this typewould occur if we define Pluto as an instance of both classes. The developershould remove the wrong relation to solve this problem.

´ ´GOMEZ-PEREZ398

� Subclass partition with common classes occurs when there is a partitionclass p , . . . , class p defined in a class class A and one or more classes1 n� � �class B , . . . , class B are subclasses of more than one subclass class p of the1 k i� � �partition. For example, if dogs and cats form a subclass partition of the set ofmammals, an error of this type would occur if we define the class Doberman as asubclass of both classes. The developer should remove the wrong relation to solvethe problem.

� Exhausti�e subclass partition with common instances. It occurs when one or severalinstances belong to more than one subclass of the defined exhaustive partition.For example, having defined the classes odd and e�en as an exhaustive subclasspartition of the class number, an error of this type appears if the number four isan instance of the odd and e�en numbers.

� Exhausti�e subclass partition with common classes occurs when there is a partitionclass p , . . . , class p defined in a class class A and one or more classes1 n� � �class B , . . . , class B are subclasses of more than one subclass class p of the1 k i� � �partition. For example, having defined the classes odd and e�en as an exhaustivesubclass partition of the class number, an error of this type appears if the classprime is a subclass of the odd and e�en numbers. So, if we define the numberThree as a prime number, we get the inconsistency since three would be aninstance of odd and even numbers.

� Exhausti�e subclass partition with external instances. These errors occur whenŽ .having defined an exhaustive subclass partition of the base class Class A into�

the set of classes class-p . . . class-p , there are one or more instances of the1 nclass A that do not belong to any class class p of the exhaustive partition. Fori� �example, if the numbers classed as odd and e�en had been defined as forming anexhaustive subclass partition and the number four were defined as an instance of

Ž .the class numbers instead of the class e�en , we would have an error of this type.

Semantic Inconsistency Errors. They usually occur because the developermakes an incorrect semantic classification, that is, classes a concept as a subclassof a class of a concept to which it does not really belong; for example, classesthe concept dog as a subclass of the concept house. The same would occur withthe instances; for example, if the instance Pluto would be an instance-of of theclass house.

2. Incompleteness Errors

Errors of this type are made when:Incomplete Concept Classification. Generally, an error of this type is made

whenever concepts are classified without accounting for them all, that is,concepts existing in the domain are overlooked. An error of this type occurs if aconcept classification musical instruments is defined considering only the classesformed by string instruments and wind instruments and overlooking, for example,the percussion instruments.

Partition Errors. They could appear when the definition of the partitionbetween a set of classes is omitted. We have identified two types of errors:

� Subclass partition omission. The developer identifies the set of subclasses of agiven class, but omits that the subclasses are disjoint. An example would be todefine dogs and cats as a subclass of mammals and to omit that dogs and cats

Ž .form a subclass partition though not complete of the set of mammals.


� Exhausti�e subclass partition omission. The developer defines a partition of a classand omits the completeness constraint to the established subsets. Exampleswould be to define odd and e�en as a subclass of numbers or to define odd ande�en as a subclass partition of numbers. In both cases, it is omitted that the

Žnumbers classed as odd and e�en form an exhaustive subclass partition that is,.complete .

3. Redundancy Errors

Redundancy is a type of error that occurs when redefining expressions thatwere already explicitly defined or that can be inferred using other definitions.

Grammatical Redundacy Errors. These errors occur in taxonomies whenthere is more than one explicit definition of any of the hierarchical relations.

� Redundancies of subclass-of relations occur between classes when subclass-ofrelations are repeated. We can distinguish direct and indirect repetition. Thereexists a direct repetition, when two or more subclasses of relations between thesame source and target classes are defined, that is, including the subclass ofrelation between the classes dog and mammals twice. There exists an indirectrepetition, for example, if we define the class dog as a subclass of pet, and pet as asubclass of animal, when dog is also defined as a subclass of animal.

� Redundancies of instance-of relations. As in the above case, there are two possibil-ities. There exists a direct repetition if two instance-of relations between the sameinstance and class are defined. There exists an indirect repetition, for example, ifwe define the instance Clyde as an instance of real elephant and real elephant as asubclass of the class elephant. The definition of an instance of relation betweenClyde and elephant would lead to a redundancy in the taxonomy.

Identical Formal Definition of Some Classes. It occurs when there are two ormore classes in the ontology with the same formal definition, that is, the onlydifference between the subclasses is the name. This error type could also beclassified as an example of classes with incomplete knowledge. The developercould solve this problem by adding what distinguishes the classes of the partitionor, otherwise, realize that it does not make sense to have classes with identicalformal definitions in the partition and delete one of them.

Identical Formal Definition of Some Instances. It occurs when there are twoor more instances in the ontology with the same formal definition, that is, theonly difference between them is the name.

VI. EVALUATION OF THE ONTOLOGY STANDARD-UNITS

A. Scope

This section presents how the approach outlined in the previous sections isŽ . 12applied to the evaluation of the standard-unit SU ontology. Gruber’s five

´ ´GOMEZ-PEREZ400

design criteria.¶14 and the ontological distinction principle.�2 that have proveduseful in the development of ontologies were also applied in the evaluation ofthe SU ontology.

The SU ontologies were selected for evaluation due to the followingreasons. First, the SU ontology is a simple ontology that only provides classesand instances. Second, we came to revise the SU ontology because it wasincluded in an environmental pollutant ontology, which uses units of measure of

Ž .possible indicators of pollution. To evaluate the standard-units SU ontology,we took its Ontolingua implementation. Then, we obtained a possible concep-tual model and we analyzed it. After this, we corrected and restructured theinitial conceptual model and we created a new conceptual model, which wasre-implemented. The new implementation was also evaluated against the estab-lished reference framework.

B. Background Knowledge

The SU ontology defines a series of SI units of measurement and othercommonly used units that do not belong to the SI units. It includes the

Žstandard-dimensions ontology, which defines a series of physical dimensions i.e.,.mass, time, length, temperature, and electrical current for different quantities.

It also includes other dimensions, derived from the above five, including pres-sure, volume, etc. Depending on the system of units used, the physical quantitiesdefined at the standard-dimensions ontology can be expressed in different unitsusing the vocabulary of the SU ontology; for example, length can be expressedin meters, miles, inches, etc. Both the SU and the standard-dimensions ontolo-gies include the physical-quantities ontology, which defines the basic vocabularyfor describing physical quantities in a general form, making explicit the relation-ship between quantities of various orders, units of measure, and physicaldimensions. A quantity is a hypothetically measurable amount of something. Forexample, the term meter, defined in the SU ontology, is an instance of the classunit-of-measure defined in the physical-quantities ontology.

The SU ontology was analyzed on the basis of its Ontolingua implementa-tion. Figure 1 shows a preliminary conceptual model that possibly originatedsuch implementation. It is important to note that this ontology contains neitherrelations, functions, nor axioms. The hierarchy illustrates that there are two

¶ Clarity and objecti�ity, which means that the ontology should provide the meaningof defined terms by providing objective definitions and also natural language documenta-tion; completeness, which means that a definition expressed by a necessary and sufficient

Žcondition is preferred over a partial definition defined only by a necessary or sufficient.condition ; coherence, to permit inferences that are consistent with the definitions;

maximize monotonic extendibility, which means that new general or specialized termsshould be included in the ontology in a such way as does not require the revision ofexisting definitions; and minimal ontological commitments, which means making as fewclaims as possible about the world being modeled, giving the parties committed to theontology freedom to specialize and instantiate the ontology as required.

� Classes with no common identity criteria must be disjoint.


Unit-Of-Measure Instance-of

Subclass-of

System-of-Units

Si-Unit

Ampere Candela Degree-Kelvin Identity-Unit Kilogram Meter Mole Second-Of-Time

.

.

Physical-Quantities

Standard-Units

Ampere Amu Angstrom ... Coulomb ... Degree-Celsius ... Electronvolt

Figure 1. Preliminary hierarchy of the standard-units ontology.

classes: unit-of-measure and system-of-units, both defined in the physical-quan-tities ontology. In this manner, a series of units of measure which are instancesof the class unit-of-measure are defined in SU, as well as a class, SI-unit, whichgroups all the SI units. In the SU ontology, all the units have a property thatindicates the dimension of the aforesaid unit. These dimensions are defined inthe standard-dimensions ontology. In this ontology, there is a class, calledphysical-dimension, which is also defined in the physical-quantities ontology, ofwhich all the dimensions defined are instances in the standard-dimensions

Ž .ontology Fig. 2 .We came to revise the SU ontology because it was included in chemical-ele-

ments. We needed to check that the units of measurement of certain attributesin chemical-elements befitted the knowledge and usual practice of experts. Theexperts had drawn up the inspection document setting out the properties to bechecked, and we transformed them into the vocabulary of the ontology. Forexample, check that the semidisintegration-period of the concept elements is

Physical-Dimension

Amount-Of-Substance-Dimension ... Density-Dimension ... Length-Dimension ... Mass-Dimension ... Resistivity-Dimension... Specific-Heat-Dimension Voltage-Dimension

Instance-of

Instance-of

Physical-Quantities

Standard-Dimensions

Figure 2. One of the hierarchies of the standard-dimensions ontology.

´ ´GOMEZ-PEREZ402

Standard-Units

Semidisintegration Period

Year

Time Quantity Time Dimension

Chemicals-Elements

Standard-Dimensions

Value Type

Unit of Measure Dimension

Figure 3. Relation between the standard-units and standard-dimensions ontologies.

Žfilled in with a value type time-quantity and its unit of measurement is year see.Fig. 3 .

C. Evaluation of the Standard-Unit Ontology

This ontology was evaluated in two steps�analysis and synthesis. It is inthe analysis phase when the ontology is evaluated and assessed. The synthesisphase seeks to correct the ontology and document any changes made. Twoontologists and two domain experts were involved in analyzing and synthesizingthe SU ontology.

1. Analysis

ŽTaking into account Figure 1 and the SU Ontolingua code version avail-.able at the Ontology Server as of December 1997 , the SU ontology contains 61

instances and one class. As we said before, all the units are defined as instancesof the class unit-of-measure and the SI units are defined also as an instance ofthe class SI-units. Therefore, this ontology provides a poor taxonomy of organiz-ing ontological knowledge. This situation causes at least two problems. First, theinstances cannot be classified by similar characteristics. Second, part of theinference power allowing some concepts to inherit properties from more generalconcepts in a properly diversified hierarchy is lost. It would be advisable to buildbranched taxonomies to take advantage of the above-mentioned benefits.

Ž .A Analysis of the taxonomy. Following the approach outlined in SectionV.B, we can say the following about the hierarchical organization of theontological knowledge.

� Analysis of inconsistencies. The ontology is consistent. There are no circularityerrors at distance zero, there are no partition errors since it is impossible todefine a partition using a unique class, and there is no semantic inconsistency.

� Analysis of incompleteness. The ontology is incomplete since there is no taxo-nomic organization classifying the units of measurement according to some


criterion that divides the general concepts into other more specific concepts allthe way down to instances. By contrast, there is a single class to which all theinstances are subordinated.

� Analysis of conciseness. The ontology is concise since hierarchical relations be-tween each instance and the class is not repeated. This ontology does not provideidentical formal definitions of some instances.

Ž .B Analysis of the ontology. Taking into account the criteria presented inRef. 14 and the evaluation examples presented in Ref. 8, the inspection of thewhole ontology code has shown other problems that are obstacles to understandand to extend the ontology. They are:

Ž .1 Understandability and extendibility problems. Definitions that should be made inthe same manner, as they refer to similar concepts, are made differently in theOntolingua code. For example, the SI base unit called ‘‘ampere’’ was defined:

Ždefine-frame ampere: own-slotsŽŽ .documentation ‘‘SI electrical current unit.’’Ž .instance-of unit-of-measureŽ ..quantity.dimension electrical-current-dimension:axiomsŽŽ Ž . ...� quantity.dimension ampere electrical-current-dimension

However, the definition of the SI base unit named ‘‘meter’’ is:

Ž Ž .define-instance meter unit-of-measure‘‘SI length unit. No conversion is given because this is a standard.’’: axiom-defŽ Ž Ž . .and � quantity.dimension meter length-dimensionŽ ...SI-unit meter

It would be advisable for definitions that are children of the same parentŽ .SI-unit to be defined according to the same templates to improve ontologyunderstanding and to ease the inclusion of new definitions. So, the lack ofpatterns for doing similar definitions makes difficult the assessment of theontology by the end-user and also the expandability of the ontology.

Ž .2 The choice of names for the different instances does not comply with a fixedstandard. For example, the different multiples and divisors of ampere are called:milli-amp, nano-ampere, and pico-ampere. To ease ontology understanding andto improve its clarity, the same naming conventions should be used to namerelated terms. Therefore, the above-mentioned names should be standardizedand denoted as follows: milli-ampere, nano-ampere, and pico-ampere.

Ž .3 The multiples of the base units do not appear to have been chosen systemati-cally. For instance, kilo-ohm and millimeter are omitted. When restructuring theSU ontology, our framework was the set of units of interest for the chemical-ele-ments ontology. As kilo-ohm and millimeter will not be used in our framework,we can say that the SU ontology is complete in this framework of reference.

Ž .4 Incompleteness in formal definitions. The ontology includes factors of conversionbetween different units of the same dimension. However, this conversion is notalways made from one particular unit to the unit that is considered the base unitat the SI. For example, taking the base unit of time ‘‘second,’’ each definition of

Ž . Žits multiples minutes, hours, etc. and its submultiples millisecond, microsec-.ond, etc. should contain the appropriate factor of conversion to seconds.

´ ´GOMEZ-PEREZ404

However, definitions appear in the SU ontology with factors of conversion asfollows:

Ždefine-frame day: own-slotsŽŽ .documentation ‘‘one day, i.e., 24 h’’Ž .instance-of unit-of-measureŽ ..quantity.dimension time-dimension: axiomsŽŽ Ž .. Ž Ž ....� day * 24 h � year * 365 day

In this definition, the factors of conversion of the unit ‘‘day’’ are established inŽ . Ž .relation to nonbase units hours and years , but not with the base unit seconds .

The following factor of conversion should be added to the formal definition:

� day * 86,400 second-of-timeŽ .Ž .Ž .

The conversion should always be made to the base unit to improve the clarity ofthe ontology. Other commonly used factors of conversion between units can alsobe added, but the conversion to the base unit should never be missing.

Ž .5 Incompleteness in the NL documentation. Some definitions have quite a poorinformal language description, which provides the user with no information.This is the case of the natural language definition of meter, which states: ‘‘SIlength unit. No conversion is given because this is a standard.’’ An extremeexample is kilometer, for which no informal definition is given at all. A naturallanguage definition should be included whenever possible to give a betterunderstanding of the more formal definition made later. In the example, ‘‘Ameter is 1,650,763.73 wavelengths in vacuo of the unperturbed transition 2 p �105d in 86Kr.’’5

Ž .6 Neither was the vocabulary of the standard-dimensions ontology used in astandardized manner. Thus, for example, the dimension mega-Pascal is definedas a ‘‘pressure-dimension’’

quantity.dimension mega-Pascal pressure-dimensionŽ .

Whereas, the dimension Pascal is said to be:

� quantity.dimension PascalŽ .Ž* force-dimension expt length-dimension � 2Ž .Ž . .

As the pressure-dimension definition exists in the standard-dimensions ontology,which is used by SU, it would be more rational and clearer to define all the unitsof pressure using this dimension, instead of using its equivalent in units oflength and force. Therefore, the dimension Pascal should be defined as

quantity.dimension Pascal pressure-dimensionŽ .

Ž . Ž .7 In the SU ontology, the number pi � is defined as an instance of the realnumbers because a factor of conversion between angles and radians appears inthe definition of ‘‘angular-degree.’’

Ž Ž .define-instance angular-degree unit-of-measure‘‘Angular measurement unit.’’

Ž Ž ..:� * radian �the-number-pi 180Ž Ž . ..:axiom-def � quantity.dimension angular-degree identity-dimension


As this is an ontology of units of measure, definitions that have nothing to dowith the above units must not be included. This problem could be solved in twoways: one possible solution would be to delete the definition of the number � inthe factor of conversion and to enter the real number 3.1415926535897936.However, a better solution is to include the real number � in the KIF-numbersontology, which could be included in the SU ontology and thus this definitioncould be used.

Synthesis. After analyzing the SU Ontolingua code and obtaining a possibleunderlying conceptual model of the ontology and after considering the problemsexplained above, we restructured it following the next criteria:

Standardize naming con�entions. We gave standard names to the new classes and in-stances. The names of the classes in the SU ontology were chosen taking intoaccount the type of units represented and the names of the dimensions found in thestandard-dimensions ontology.

Specialization of concepts. The goal was to identify general concepts that are specializedinto more specific and disjoint concepts down to domain instances. The criterionused to specialize was to group units according to the base unit. So, all the units formeasuring length are grouped within the same class. The name of this class is

Ž .‘‘length-unit,’’ and its instances are: meter SI base unit , ‘‘angstrom,’’ ‘‘centimeter,’’‘‘foot,’’ ‘‘inch,’’ ‘‘kilometer,’’ and ‘‘mile.’’ In this case, the new ontology includes oneclass for each type of SI base unit. We created 20 new classes. Therefore, we cansay that the restructured ontology complies with the ontological distinction princi-ple2 which states that classes should be disjoint. We also maximized the monotonicextendibility 13 of the SU ontology because the inclusion of classes and instancesdoes not require the revision of existing definitions.

Branched taxonomies. Whenever possible, the hierarchy must be sufficiently branched bysimilar characteristics to increase the power provided by inheritance mechanismsbetween classes and instances. Figure 4 illustrates the new hierarchy. It should beinterpreted as the concept classification tree, in which all the concepts of the firstbranch are subclasses of the unit-of-measure class and the terms represented inboxes are instances of the concept to which they are linked by an arrow.

Inclusion of new properties and changes to existing properties. In this ontology, the propertyabbreviation was added to each unit defined for the purpose of extending the use ofthis international standard for this attribute, ruling out widespread, though notabsolutely correct, uses.

Clarity of the natural language documentation. The documentation accompanying eachdefinition must be clear, useful, and must give a better understanding of the formaldefinition of the term defined.

Minimize the semantic distance between sibling concepts.1 Similar concepts are usuallygrouped and represented as subclasses of one class and should be defined using thesame set of primitives, whereas concepts which are less similar are presentedfurther apart in the hierarchy. All the terms in the restructured ontology weredefined using the same pattern to give a clearer understanding of the ontology. TheSI units were all defined following the same template. As an example:

Ždefine-okbc-frame kilogram:direct-typesŽ .SI-unit mass-unit:own-slot-specsŽŽdocumentation ‘‘SI mass unit. It is equal to the mass of the

.international prototype of the kilogram.’’Ž .quantity.dimension mass-dimensionŽ ...abbreviation ‘‘kg’’

´ ´GOMEZ-PEREZ406

Unit-Of-Measure

Electrical-Current-Unit Mass-Unit Pressure-Unit

Si-Unit

System-Of-Units

Ampere Milli-Ampere Nano-Ampere Pico-Ampere

Amu Gram Kilogram Pound-Mass Slug

Pascal Mega-Pascal Meter

Kilogram Second-Of-Time Ampere Degree-Kelvin Mole Candela Identity-Unit

Subclass-of

Instance-of

... ... ...

Figure 4. Taxonomy for the modified conceptual model.

D. Implementation and Re-Evaluation

On the basis of these considerations, the SU ontology was re-implementedin Ontolingua using the Ontology Server editor. The same people evaluated there-implemented ontology, and an external person that did not participate on thetwo previous phases compared them using the previous evaluation frameworkand common sense knowledge. The conclusions were:

� The ontology is syntactically correct, as it successfully passes the Ontology Serveranalyze tests.

� The ontology is complete for the use to which it is to be put in the chemicalontology.

� The ontology is consistent, as the knowledge formalized was checked against theabove-mentioned sources of knowledge.

� The ontology is concise, as there is no redundant knowledge.� The new implementation is more structured and understandable than the old

one.

For the purpose of assuring information about the evolution of the SU ontology,a rigorous change control6 was performed. The goal is to have all the changesdocumented and to facilitate future modifications and changes to the ontology.Change control is what allows ontology developers to control the evolution ofthe ontology, ruling out any errors caused by undesired effects in the finalimplementation, such as inconsistency and incompleteness. Change control alsohelps end-users to determine which version of the ontology they require fortheir system or for the new ontology they are to develop. Figure 5 presents anexample of a control report for a change made to the standard-units ontology.


Description of the Change: Modify the hierarchy of the Standard-Units

ontology shown in figure 1, as it does not include intermediate classes that

represent each type of SI base unit. In this model, all the instances of the

ontology depend on one class.

Need for Change: It is not technically correct to have a class from which all

the instances of the ontology hang. This structure prevents concepts being

classed by similar characteristics, and some of the inference power allowing

concepts to inherit properties from other more general concepts in a properly

diversified hierarchy is lost.

Effects of the Change: The hierarchy has been satisfactorily branched, as

shown in figure 4. In this case, one class has been created for each type of SI

base unit. This change affects all the instances of the ontology, as the Unit-Of-

Measure class has to be replaced in the formal definition of the instances by the

new class representing the SI base unit to which they belong.

Alternatives: None.

Date of change: 27/03/98.

Change made: Changes are shown in figure 4.

Figure 5. Change control report.

The team took 108 h to evaluate, re-implement, re-evaluate, and to performthe change control documents.

VII. CONCLUSIONS

Although there are a host of papers on knowledge-based systems evalua-tion, the ontology evaluation field is only just emerging. As ontologies are usedfor several different purposes, diverse kinds of evaluation are required. Evalua-tion of their content and evaluation of their development process are critical inthe process of using ontologies for knowledge management, natural languageprocessing, intelligent integration information, agents, etc. In this sense, we cansay that it is unwise to publish an ontology, to reuse an existing ontology to builda new ontology, and to implement an application that relies on ontologies

Ž .written by others or even yourself , without first evaluating and assessing it.

´ ´GOMEZ-PEREZ408

In this paper, we presented the conceptual bases of ontology evaluationfrom a technical point of view and examples of how evaluation of taxonomiesshould be performed. It includes:

� A brief summary of previous work done on ontology evaluation and the criteriaŽ .consistency, completeness, conciseness, expandability, and sensitiveness used toevaluate and to assess ontologies.

� Identification of a set of errors that can be made when domain knowledge isstructured in taxonomies. Errors are classed in the following categories: circular-

Žity errors, partition errors including exhaustive subclass partitions and nonex-.haustive subclass partitions , redundancy errors, and semantic errors.

� It also describes the SU ontology evaluation process following the approachpresented in this paper and bearing in mind the future use of this ontology by aenvironmental ontology.

Future work will include primarily:

� Addressing in more depth the theoretical foundations of ontology evaluation, thatcould include the evaluation of other components.

� Introducing activities related to ontology evaluation into ontology developmentmethodologies. The purpose of these activities will be to raise ontologist aware-ness of the fact that evaluation should be performed throughout the entireontology life cycle to detect errors at the earliest possible time and should not beleft until the end when the ontology has been implemented.

� Evaluation of very well-known and large ontologies like Cyc, SENSUS, etc.� Develop independent tools for evaluating ontologies built under a given knowl-

edge representation paradigm.� Develop evaluation tools to be integrated in the environments used to build

ontologies.� Create application-dependent and end-user methods to judge the adequacy of an

ontology for a given application.

Ž .The first part Sections II�IV of this research work was performed at the Knowl-edge Systems Laboratory in Stanford University. The author thanks the followingpersons: Almudena Galan and Rosario Garcıa, for their knowledge of chemistry and the´ ´environment; Maria Dolores Rojas for the help on the analysis and synthesis of thestandard units ontology, and Mariano Fernandez, for the evaluation of the new version of´the standard-units ontology.

References

Ž .21. Arpirez J, Gomez-Perez A, Lozano A, Pinto S. ONTO Agent: An ontology-based´ ´WWW broker to select ontologies. In: Workshop on Applications of Ontologies andPSMs, Brighton, United Kingdom, August 1998, p 16�24.

2. Borgo S, Guarino N, Masolo C. Stratified ontologies: The case of physical objects. In:Proceedings of the Workshop on Ontological Engineering. Held in conjunction withthe Eleventh Conference on Artificial Intelligence 1996, Budapest, 1996, p 5�15.

3. Farquhar A, Fikes R, Rice J. The Ontolingua server: A tool for collaborativeontology construction. In: Proceedings of the 10th Knowledge Acquisition for Knowl-edge-Based Systems Workshop, Banff, Alberta, Canada, 1996, p 44.1�44.19.

4. Fernandez M, Gomez-Perez A, Pazos J, Pazos A. Building a chemical ontology using´ ´ ´methontology and the ontology design environment. IEEE Intell Syst Appl 1999;Ž .4 1 :37�45.


5. Genesereth M, Fikes R. Knowledge interchange format; Technical Report; Com-puter Science Department, Stanford University, Logic-92-1, 1992.

6. Gomez-Perez A, Rojas-Amaya MD. Ontological reengineering for reuse. Knowledge´ ´acquisition modeling and management. In: 11th European Workshop, EKAW’99.Dagstuhl Castle, Germany, May 26�29, 1999, p 139�156.

7. Gomez-Perez A. Knowledge sharing and reuse. In: Liebowitz J, editor. The hand-´ ´book of applied expert systems. Boca Raton: CRC Press; 1998.

8. Gomez-Perez A. A framework to verify knowledge sharing technology. Expert Syst´ ´Ž .Appl 1996;11 4 :519�529.

9. Gomez-Perez A, Juristo N, Pazos J. Evaluation and assessment of the knowledge´ ´sharing technology. In: Mars NJL, editor. Towards very large knowledge bases.Amsterdam: IOS Press; 1995.

10. Gomez-Perez A. Some ideas and examples to evaluate ontologies; Technical Report´ ´KSL-94-65; Knowledge System Laboratory, Stanford University. Also in Proceedingsof the 11th Conference on Artificial Intelligence for Applications, CAIA94, 1994.

11. Gomez-Perez A. From knowledge based systems to knowledge sharing technology:´ ´Evaluation and assessment; Technical Report KSL-94-73; Knowledge Systems Labo-ratory, Stanford University, December 1994.

12. Gruber T, Olsen R. An ontology for engineering mathematics; Technical ReportKSL-94-18; Knowledge Systems Laboratory, Stanford University, CA, 1994.

13. Gruber T. A translation approach to portable ontology specification. KnowledgeAcquisition 1993;5:199�220.

14. Gruber T. Toward principles for the design of ontologies used for knowledge sharingInt J Human Comput Stud 1993;43:907�928.

15. Gruninger M, Fox M. Methodology for the design and evaluation of ontologies. In:Proceedings of the International Joint Conference on Artificial Intelligence 1995Workshop on Basic Ontological Issues in Knowledge Sharing, 1995.

16. Lenat DB, Guha RV. Building large knowledge-based systems. Representation andinference in the cyc project. Reading, MA: Addison-Wesley; 1990.

Ž .17. MacGregor R. Inside the LOOM classifier. SIGART Bull 1991;2 3 :70�76.18. Swartout B, Patil R, Knight K, Russ T. Towards distributed use of large-scale

ontologies. In: Spring Symposium Series on Ontological Engineering, Stanford Uni-versity, CA, 1997, p 138�148.

19. Uschold M, Gruninger M. ONTOLOGIES: Principles, methods and 16 applications.¨Ž .Knowledge Eng Rev 1996;11 2 :1996.

20. Uschold M. Where are killer apps? In: Workshop on Applications of Ontologies andProblem Solving Methods, Eleventh Conference on Artificial Intelligence 1998,Brighton, United Kingdom, 1998.

evaluation of ontologies

Documents