semantic web

THEME ISSUE PAPER

Evaluation of semantic retrievalsystems on the semantic web

Jorge Luis Morato, Sonia Sanchez-Cuadrado and Christos DimouComputer Science Department, University Carlos III of Madrid, Leganes, Spain

Divakar YadavComputer Science and Engineering, Jaypee Institute of Information Technology,

Noida, India, and

Vicente PalaciosComputer Science Department, University Carlos III of Madrid, Leganes, Spain

Abstract

Purpose This paper seeks to analyze and evaluate different types of semantic web retrievalsystems, with respect to their ability to manage and retrieve semantic documents.

Design/methodology/approach The authors provide a brief overview of knowledge modelingand semantic retrieval systems in order to identify their major problems. They classify a set ofcharacteristics to evaluate the management of semantic documents. For doing the same the authorsselect 12 retrieval systems classified according to these features. The evaluation methodology followedin this work is the one that has been used in the Desmet project for the evaluation of qualitativecharacteristics.

Findings A review of the literature has shown deficiencies in the current state of the semantic webto cope with known problems. Additionally, the way semantic retrieval systems are implementedshows discrepancies in their implementation. The authors analyze the presence of a set offunctionalities in different types of semantic retrieval systems and find a low degree of implementationof important specifications and in the criteria to evaluate them. The results of this evaluation indicatethat, at the moment, the semantic web is characterized by a lack of usability that is derived by theproblems related to the management of semantic documents.

Originality/value This proposal shows a simple way to compare requirements of semanticretrieval systems based in DESMET methodology qualitatively. The functionalities chosen to test themethodology are based on the problems as well as relevant criteria discussed in the literature. Thiswork provides functionalities to design semantic retrieval systems in different scenarios.

Keywords Semantic web, Semantic search engines, Problems in the semantic web,Qualitative evaluation, Requirements analysis

Paper type Research paper

IntroductionThe concept of the semantic web has emerged out of the need to facilitate access,management and retrieval of knowledge. Although different authors interpret the

The current issue and full text archive of this journal is available at

www.emeraldinsight.com/0737-8831.htm

MINCIN funded this research project HAR2011-27540 and TIN2011-27244.An earlier version of this paper was presented at the LOV symposium, held in Madrid, Spain,

on 18 June, 2012.

LHT31,4

638

Received 4 March 2013Revised 22 August 2013Accepted 3 September 2013

Library Hi TechVol. 31 No. 4, 2013pp. 638-656q Emerald Group Publishing Limited0737-8831DOI 10.1108/LHT-03-2013-0026

semantic web in a different way (Bizer et al. 2009) the idea that underlies is a web ofmachine-readable, semantically related data, which means the navigation can beachieved by means of semantic relationships among concepts instead of hyperlinksbetween documents. Therefore, the semantic web requires semantic documents andresources that permit knowledge representation and inferences.

Semantic documents, in some application domains, are referred to as compositeinformation resources composed of uniquely identified, semantically annotated, andsemantically interlinked document data units of different granularity (Nesic, 2010). Inothers, such as the Swoogle search engine, a SWD (stands for semantic web document)is defined as an RDF document. Considering an SWD just as an RDF document isproblematic, as Pastor-Sanchez et al. (2012) point out that the RDF employment in theDataHub documents is frequently scarce and sometimes anecdotal.

Vocabularies of metadata, ontologies and schemas are also considered semanticdescription resources, because all of them are key elements for providinginteroperability in the context of the linked data cloud and semantic webimplementations. Vocabularies are defined as the concepts and relationships usedto describe and represent an area of concern. They are used to classify the terms thatcan be used in a particular application, characterize possible relationships, and definepossible constraints on using those terms (W3C, 2013). Vocabularies are not only moreuseful when they can be retrieved, accessed and used easily, but also when they areformalized in a more precise way. They can be formalized in various levels from XML(eXtensible Markup Language) schemas or RDF to ontologies. Their use can also varyfrom a simple resource description (e.g. in HTML code) to an application profile thatcombines different vocabularies.

In the following sections, we proceed with a review of literature related to theinfrastructure of the semantic web. Next, we identify characteristics and relevantproblems in other studies about semantic retrieval evaluation. Finally, we propose aqualitative evaluation of different functionalities for distinct types of semantic searchsystems in the context of the Web.

1. Discrepancies of the semantic webMany projects on the semantic web have distinct approaches to represent knowledge ina formalized way. For example, ontologies are described with other formal languages,such as KIF, RIF or Common Logic. Moreover, some initiatives focus on the inclusionof metadata in HTML, through, for instance, the projects RDFa, microformats ordataspaces (Bizer et al., 2009). In the case of the ISO Topic Map standard defines a datamodel, expressed with XML syntax and the TCML language. There are other similarinitiatives based on XML Schema, for example XMI for UML, MPEG-7 or TEI (Bikakiset al. 2013).

Each representation model has different syntax and semantics that can produceconflicts when we try to fuse them together. In the semantic web the prevailingtendency uses RDF as a standard data model and a serialization format based on XMLsyntax (Allemang and Hendler, 2011), where data resources and vocabulary conceptscan be accessed by URIs. Despite this tendency, there is a need to establishtransformations from other languages and models such as XML Schema, KIF, etc.

The W3C has proposed some recommendations for reducing the gap between thedifferent languages of ontologies, annotation and XML Schema (W3C , 2007, 2012). It is

Semanticretrieval systems

639

important to note that the semantics in XML Schema are informal, in contrast to RDFwhere semantics are more formalized. An additional main aspect is that unlike RDF,querying XML Schema takes a closed world assumption (Bikakis et al., 2013).

One of the most successful initiatives is the Linked Data project (Bizer et al., 2009).The main proposal of this project is based on four principles:

(1) identity for each entity (e.g. URIs);

(2) accessibility to each object (e.g. HTTP URI);

(3) structure information in a formalized way with standards (e.g. RDF andSPARQL); and

(4) integration of entities through relationships between them (e.g. applicationprofiles, metadata vocabularies, and so on).

Linked data enables integrating distributed data, and leveraging the generation ofrepositories for semantic resources and datasets (DBpedia, GeoNames, UMBEL). Thesuccess of the linked data initiative is determined by the ability of practitioners toidentify, reuse, or link to other available sources of linked data (W3C, 2011).

It must be noted that although in the linked data project the files that describeresources are usually RDF documents, in practice we observe other formats as well, forexample Xlink (W3C, 2010), a required format for topic maps until 2006, or JSON linkeddata (Lanthaler and Gutl, 2012). There have been many initiatives that return to thesimplicity of the Web 1.0 to overcome problems with b-nodes and RDF molecules, forinstance hypernotation (Milicic, 2011). The heterogeneity of document formats in thesemantic web explains why, for example, when we submit a search query to theDataHub repository, we retrieve a large variety of CSV, XML or HTML files, instead ofonly RDF files.

As a conclusion, the specifications of a retrieval system for semantic web resourcesmust not be restricted to RDF documents only, but instead they should include othertypes of resources and languages as well.

1.1 Knowledge modeling on the webThe main consequence of modeling knowledge in the semantic web context is thenecessity to select metadata vocabularies for elaborating semantic documents.However, selecting the appropriate metadata vocabulary involves several difficultiesand challenges. This process includes:

(1) Selecting a vocabulary from the linked open data (LOD) Cloud.

(2) Selecting the appropriate approach for describing the subject.

(3) Defining queries and conceptual and semantic navigation.

We have found some difficulties to carry out these tasks.1.1.1 Selecting a metadata vocabulary from the LOD cloud. Selecting a metadata

vocabulary from the Cloud involves several difficulties. One problem is a lack ofcriteria to select an adequate metadata vocabulary. According to the resourceDatasets in the next LOD Cloud, by the Research Group Data and Web Science fromthe University of Mannheim (2013), there are 25.2 billion RDF triples in W3C linkingopen data (aka LOD); datasets like Yago alone have 19 million triples. According toPastor-Sanchez et al. (2012), 347 millions of triples are associated with controlled

LHT31,4

640

vocabularies in the Data Hub. For example, there are many vocabularies that can beused to describe resources, persons and institutions. However, there is a lack of criteriato select the best candidate. Palacios (2010) has suggested some criteria such as degreeof standardization, stability, number of elements, and usage statistics and popularity.

Other problems are lack of consensus between similar metadata element set due tooverlapped definitions, lack of descriptions, formalization problems and differentconceptualizations. Specifically there exist various vocabularies for description ofpersons and organizations. Many of them have a large degree of overlapping, as forexample FOAF, PIM, MADS (xsd), LID, DOAC, VCard, hcard, BIO or XFN. Similarly,there are many different alternative vocabularies for semantic relationships, such asSKOS, VDEX, BS 8723, ISO 25964, Zthes, PSI TopicMaps (Morato et al., 2007). Thisoverlapping should not be a problem, provided that the definitions are coherent whenthey describe the same concepts. On the contrary, it could prove beneficial for LOD. Arigorous analysis will show us typical problems in the determination of exactcorrespondences. For example, one of the obvious elements would be birthday; if wesee, however, its description in two different widely used vocabularies, like Foaf andVcard we observe that the correspondence between elements is not obvious.

. Foaf:birthday, definition: the birthday of this agent, represented in mm-ddstring form, eg. 12-31

. Foaf:agent, definition: Used to describe any agent related to bibliographic items.Such agents can be persons, organizations or groups of any kind.

. Vcard:bday, definition Date of birth of the individual associated with the vCard

. Vcard:agent, Information about another person who will act on behalf of thevCard object

On the one hand, the meaning of the term "agent" is different between twovocabularies. And on the other hand, one of the vocabularies specifies a date format,while the other one does not (although it is usually associated with the xsd datatypeof date).

In addition to the above problem, there are other resources that present metadatawithout definitions, updates or nonsense elements (e.g. PIM or synonyms anddeprecated elements like foaf:lastName, foaf:familyName orfoaf:family_name). It must be noted that, according to the ISO 11179 datamodel, metadata registry can include different vocabularies and their semantics.However, metadata handling is implemented by the original system, not in the registry.In other words, in order to create a new registry and be able to alter the definitions, weshould define a new local resource.

1.1.2 Selecting the appropriate approach for describing the subject. Selecting theappropriate approach for describing the subject implies problems with URIs such as theever changing nature of the internet and the absence of URIs for some elements isanother problem. The absence of these URIs is usually caused either by higher orderrelationships or by RDF molecules. Typically, many of these problems are solved byblank nodes (bnodes). A similar approach is widely found in knowledge managementunder the name of reification. These bnodes lack in URIs to identify them, impeding thelinking of triples. This problem is also similar to the absence of URIs for RDF molecules(Ding et al., 2005). RDF molecules are atomic units that can be larger than an RDF triple


641

and whose division would imply significant loss of semantics. As an example we can seethat if we have a person whose name is Tim and his surname is Berners-Lee, we needa higher level of granularity to group name and surname together.

Finally, many nodes provide literal text as their value, which produces manyproblems during their semi-automatic linking. Various works (such as Waitelonis andSack, 2009 and Kobilarov et al., 2009) that link profiles with Dbpedia and Foaf presenta success ratio of 12 percent for persons and 18 percent in organizations. Theseauthors, however, suggest that DBpedia could not guarantee correct and exhaustiveresults. In the case of Kobilarov et al. (2009), the results for persons and organizationswere at 30 percent. These results agree with the work of Howarth (2000) and Buscaldiet al. (2003).

1.1.3 Queries and conceptual navigation. In addition to the problems that we havealready mentioned, queries are formalized in SPARQL language ( Jain et al., 2010). Inorder to represent the query, it is necessary not only to know this language, but also toknow in detail the concepts that are present in each resource and the properties withwhich the concepts have been analyzed. In other words, it is improbable that a genericquery on various triples of different dataset would give the desired results.

Semantic navigation is defined as a transversal through relationships betweenconcepts. However, many linked vocabularies have mistakes in their hierarchy(e.g. circles), disjoint properties, different classification criteria, granularity, and so on(Palacios, 2010; Fuentes and Meja, 2013). Thus, for the working example about opticalmaterials, we can find in Dbpedia that:

Given that it is impossible for the same concept to be specific and generic of anotherconcept at the same time, conceptual navigation is rendered impossible. There existsan abundance of paradoxical examples in the work of Fuentes and Meja (2013).

A similar problem would occur if we search in Wikipedia for the URI that containsTransparent_Materials; we will be forwarded to the article titled Transparency andtranslucency. This last term is itself a composite term that implies a hierarchy, sincein optics translucency is a generic term of transparency. Additionally, it isnoticeable that the concepts are related under dc:subject, where instances aremixed with properties.

A summary of other problems identified on the semantic web are shown in Table I.

LHT31,4

642

Problems Authors Description

Vocabulary quality Bizer et al. (2009), Palacios(2010) Fuentes and Meja(2013)

Relevance of resources and trust. Manyvocabularies have mistakes in their hierarchyhampering conceptual navigation

Quality describingdocuments

Pastor-Sanchez et al. (2012),Jain et al. (2010)

Lack of conceptual description ofdatasets,RDF employment in the DataHubdocuments is frequently scarce andsometimes anecdotal. Descriptions in the LoDcloud presents shallow expressivity

Licensing and openinitiatives

Bizer et al. (2009),Strasunskas and Tomassen(2010)

The semantic web community prefer openstandards, like OWL or RDFS, thanalternatives with proprietary encoding formator results of open academic experiments

Semantic linking Jain et al. (2010), Palacios(2010)

The high number of description resourcescauses scalability problems (linkingarchitecture problems due to one-by-onemappings), overlapping in vocabularies. TheLoD Cloud datasets lack schema levelmappings between concepts of differentdatasets at the schema level

Obsolescence Milicic (2011), Bizer et al.(2009)

Updating or removing semantic resourcesfrom the web. Link maintenance is poor

Trustworthiness Morato et al. (2007),Bechhofer et al. (2010),Bizer et al. (2009)

Publishing requirements as absence ofauthoring information, quality, credit,attribution are scarcely implemented.Additional problems related withadvertisements and semantic spam invocabulary building and metadata description

Formalization Bikakis et al. (2013), Bizeret al. (2009), Lanthaler andGutl (2012), Milicic (2011)

Variety of technologies to representknowledge (e.g. RDF, XML, UML, TEI, topicmaps, TCML, microformats, KIF, commonlogic) and heterogeneity of linking. formats inthe semantic web: RDF, JSON linked data,Xlink, Hypernotation. Semantics in XMLSchema is informal and a closed worldassumption

URI problems Milicic (2011) Problems to assign URIs to b-nodes and RDFmolecules

Difficulties to queryingSPARQL

Jain et al. (2010) Users to specify the details of the structure ofthe graph and be familiar con multipledatasets

Privacy problems Bizer et al. (2009) Privacy problems caused by integrating datafrom distinct sources

Vocabulary suitabilityand adaptation

Palacios (2010), Mangold(2007)

The suitability of a vocabulary is defined onthe basis of low or tight coupling. There is alack of statistic data to help the selection ofvocabularies

Usability Morato et al. (2007), Urenet al. (2007)

Usability of current systems in the semanticweb

Table I.Problems identified on

the semantic webaccording to the literature


643

1.2 Semantic retrieval1.2.1 Semantic search. According to Wei et al. (2008), in this work semantic searchrefers to the retrieval of resources described for knowledge modeling and the usage oflogic-based knowledge representation languages for automated machine processing.

The term semantic search on the web is currently a buzzword with differentinterpretations (Batzios and Mitkas, 2012). Traditionally, it includes techniques thataddress the improvement of accuracy of searches (Fazzinga and Lukasiewicz, 2010,Girit et al., 2012, Guha et al., 2003): disambiguation and contextualization of queries,questions to semantic and semantically annotated documents, faceted search,question-answering, query formalization or searches by similarity. In general, the mainfeature of semantic search engines is to be able to solve complex queries by giving ananswer to a query than to offer us a set of documents where we could find that answer.

1.2.2. Semantic retrieval systems. In the context of the semantic web, the concept ofinformation retrieval systems is rather generic and vague. It encompasses differentcriteria. Scheir et al. (2007) propose the following classification:

. the system operates on the semantic web with machine-interpretable data;

. the systems is based on technology for the semantic web and ontology-driveninformation retrieval approaches; and

. the systems perform information retrieval and not data retrieval based on querylanguages as SPARQL.

Among the first search engines to appear were SHOE (Mangold, 2007) and On2broker.On2broker (Fensel et al., 1999) had the objective of retrieving XML and RDFdocuments, as well as vocabularies like MPEG-7 o Dublin Core. Since then, a number ofsearch engines have been presented: WebOWL (Batzios and Mitkas, 2012), Swoogle(Ding et al., 2005a), XSearch (Amer-yahia and Lalmas, 2006), SWSE (Hogan et al., 2011),Sindice, SemSearch or Watson. Most of these search engines are based on RDF or OWLdocuments, as for example Swoogle, Falcons and Watson. The semantic results areRDF documents (for example ontologies and ontology instances).

Some web retrieval systems extend even more the document typology; SWSEtransforms XML and HTML documents to RDF for subsequent indexing. Sindice, inaddition to RDF, includes microformats, RDFa and Microdata. Watson focuses on RDFand OWL, but it includes other ontology languages like DAML-OIL.

Regarding the positioning of results, it is usually based on solutions similar toGoogle (e.g. Swoogle, WebOWL and SWSE), but others are like Falcons utilizevariations of TF-IDF and popularity. XSearch is based on XML data, returning the partof the XML tree structure that coincides with the search. A review of XML basedsystems can be found in Amer-Yahia and Lalmas (2006).

Another type of information retrieval systems that search semantic web resourcesas metadata vocabularies are directories. They are considered as a simple retrievalsystem because searching for results is realized through the navigation of a treehierarchy that contains the resources. These directories are often not included in manystudies about semantic search; however we consider that it is a relevant resource onsemantic web environment.

1.2.3 Previous works in evaluation of semantic retrieval systems. Initial works onevaluation of semantic search engines were mainly focused on query performance(Tumer et al., 2009; Andago et al., 2010). Many of these studies compare general

LHT31,4

644

purpose search engines to those that extract semantic knowledge from naturallanguage texts (for instance, Hakia) by means of a knowledge organization system.These studies identified and analyzed common elements for the comparison of the twocategories of search engines (general purpose and semantic search). The results showan advantage of general purpose search engines. These results are different whenstructured and formalized documents, such as RDF, are taken into account.

The criteria to evaluate these semantic search engines are not based just inquery performance (Strasunskas and Tomassen, 2010). These authors state that arigorous comparison must take into account factors such as: query and ontologyquality, user interaction, semantic indexing criteria, query expansion, filtering,ranking methods and presentation of results. Therefore, they propose aclassification framework (Table II) that comprises seven categories. All of themare based on previous works to classify semantic search engines (Esmaili andAbolhassani, 2006; Mangold, 2007).

Mangold (2007) carries out a classification of semantic search approaches. Thisstudy analyzes ten systems according to seven criteria, as shown in Table II. Some ofMangolds criteria are dependent on each other. Although the author recognizes thatthere are other possible characteristics, they are not included in that study because thepurpose of that work was to focus on characteristics that most authors regard asrelevant. In the ontology structure, three types are analyzed: anonymous properties(the only aspect presented is a shared context); standard properties: the commonthesaurus relationships (synonym, hypernym, meronym, instance), in addition tonegation; and Domain specific properties. In the case of Uren et al. (2007), the authorsidentify four characteristics for classifying retrieval systems, none of which is relatedto ontology quality criteria.

Mangold (2007) Strasunskas and Tomassen(2010)

Uren et al. (2007)

Architecture Architecture Search environment: largescale, heterogeneity andportability

User context (user&sinformation needs)

Search goal (questionanswering, ontologies, data)

Query types

Query modification Search phase Iterative and exploratorydimensions: refinement,recommendation and reuse

Transparency (transparent/interactive)

User input (keywords, naturallanguage, graphics, formalquery or interactive)

Intrinsic problems:Understanding, result rankingand matching

Ontology structure Knowledge richness (taxonomy,thesaurus, ontology)

Ontology technology Ontology encoding (RDFS,OWL, . . .)

Coupling (ontology-documentstight/low)

Scope (Web, desktop)

Table II.Criteria for evaluating

semantic retrievalsystems


645

As it can be observed, some of the evaluation criteria, such as user context orsearch goal, take into account the type of semantic search. Different works showdifferent types of semantic search. Wei et al. (2008) classify semantic searchresearch with respect to objectives, methodologies and functionalities:document-oriented search; entity and knowledge-oriented search; multimediainformation search; relation-centered search; semantic analytics; mining-basedsearch. Fazzinga and Lukasiewicz (2010) points out that the evaluation of theaccuracy of a system must be dependent on its search capacity. The proposals ofUren et al. (2007) and Strasunskas and Tomassen (2010) reduce the typologyproposed by Wei et al. (2008). The work of Strasunskas and Tomassen (2010) statesthat standard IR metrics as recall and precision are not enough to measure usersatisfaction because of the complexity and the effort needed to use semantic searchtools. Therefore these authors suggest a holistic evaluation that includes systemquality, ontology quality, query quality, topic complexity and user interaction.Table III arrays the types of semantic search, as they are presented in the abovepublications.

Hence, there is a need to establish criteria to evaluate semantic search engines.Many of the earlier studies just describe the functionalities of these search engines, butthere is still a need to provide a mechanism to facilitate the comparison in a similarway to query performance metrics in classical retrieval.

2. Evaluation methodA summary of some problems identified on the semantic web are shown in Table I. Aswe observe, all characteristics are qualitative and therefore difficult to measure withclassical information retrieval evaluation methods. In this section, we propose amethod to deal with some criteria scarcely analyzed in previous studies. Next, we haveselected the Desmet method (Kitchenham, 1996) in order to analyze and evaluatedifferent types of semantic web retrieval systems (directories and search engines), withrespect to their ability to manage and retrieve semantic documents. The goal is toclarify if these semantic system types are implementing the requirements that arediscussed in prior studies and if they deal with the current problems found in thesemantic web.

Wei et al. (2008) Uren et al. (2007)Strasunskas andTomassen (2010)

Fazzinga andLukasiewicz (2010)

Document-orientedsearch

Entity search Information search Structured languages

Entity and knowledge-oriented search

Relation search Data search Keyword-based

Multimedia informationsearch

Parameterized (faceted)search

Question Answering Natural languages

Relation-centeredsearch

Ontology retrieval

Semantic analytics

Mining-based searchTable III.Types of semantic search

LHT31,4

646

DESMET is a comparative method for performing simple, reliable and impartialevaluations in software engineering, such as requirement analysis. This method isintended to help an evaluator in an evaluation exercise that is unbiased and reliable(e.g. maximizes the chance of identifying the best method/tool). The DESMET methodis context-dependent, which means that we do not expect a specific tool to be the best inall circumstances. Thus, in this work we do not intend to determine the best retrievalsystem but to offer a way to select one semantic system type or another according tothe context. We consider that the method is adequate because the main evaluationcriteria are functionalities difficult to measure in the same way that classical retrievalsystems do. Besides, these web retrieval systems are always evolving, so we suggestmethods capable to be adapted to functionality modifications. This method enables aqualitative evaluation of the level of support that various systems provide to theorganization and the retrieval of semantic elements.

Following the steps of the DESMET method, first we have identified the specificcircumstances for a context to retrieve ontologies and metadata vocabularies about aspecific subject. Second, we have performed a feature analysis, which essentially is anevaluation based on the identification of requirements and their correspondence to thecharacteristics that these specifications support. Finally, we have defined the retrievalsystems to be evaluated, the criteria to evaluate them and assigned the values andprioritization degree according to DESMET method.

2.1 Selecting retrieval systems of semantic documentsWe have collected 12 semantic retrieval systems. We have found that retrieval systemsare different according to kinds and functionalities. In consequence, we have classifiedretrieval systems in four types of semantic search engines, in order to provide acomparison framework where we can analyze the results by groups. We propose thefollowing classification by types of retrieval systems and types of document that theysearch:

. Ontology search engines. These applications crawl the web discovering semanticweb documents. The search engine indexes the ontologies in order to retrieve andrank the results. Examples are Swoogle (http://swoogle.umbc.edu/), Sindice(http://sindice.com/), or Watson (http://watson.kmi.open.ac.uk/WatsonWUI/).

. Search engines for metadata. A search engine aimed to retrieve metadata, as forexample the Linked Open Vocabulary (LOV) (http://lov.okfn.org/dataset/lov/index.html) and the DataHub (http://datahub.io/ http://datahub.io/).

. Ontology directories. Ontology catalogues collected by hand. Examples: DAMLOntology Library (www.daml.org/ontologies/) and Protege Ontologies (http://protegewiki.stanford.edu/wiki/Protege_Ontology_Library).

. Metadata directories. Metadata catalogues, such as UKOLN metadata resource(www.ukoln.ac.uk/metadata/resources/), Topic Maps PSIs (http://psi.mchapman.com/vl/index), RDA vocabulary (http://rdvocab.info/) and the OpenMetadata Registry (http://metadataregistry.org/vocabulary/list.html).

We have avoided some kinds of search engines such as question-answering andchatbots due to the fact that their technology is based on information extractioninstead of metadata description and their KOSs are not public. Although they interact


647

with a human user, they do not necessarily retrieve semantic documents, but insteadthey utilize semantic resources as a natural language processing technique for userinteraction purposes.

2.2 Evaluation criteria for retrieval systems of semantic documentsTables IV-VI present the set of criteria that we have defined for evaluating theresources. These characteristics have been selected and refined from the previousliterature and classified in three types of criteria associated to each characteristic:

(1) Schema management. The related criteria are: interoperability, formalization,interactivity and semantic framework (Table IV).

(2) Semantic management. Related to the meaning of concepts and theirmanagement; related criteria are: disambiguation, multilingualism, synonyms,scope, extensibility, reusability, modifiability, and language (Table V).

(3) Queries. Concerning the query process and the management of the obtainedresults. This category copes with sense specification, conceptual queries,contextual queries and document retrieval (Table VI).

Following the Desmet method, we establish two types of features: simple andcompound. The simple characteristics are those that can be present or absent and canbe assessed using a Boolean scale. The compound characteristics get the degree towhich they are supported and quantified in an ordinal scale. The characteristics areidentified and prioritized, and we establish the system for the assessment of thecharacteristics, with respect to their type and importance:

. Simple types: No (0) and Yes (5).

. Compound types: None (0), Low (1), Medium (3), High/Fundamental (5).

. Importance: Optional (3), Desirable (6) and Obligatory (10).

3. ResultsIn the evaluation process of the different methods for the retrieval of semanticdocuments, we have assigned one value to each of the characteristics. None of theretrieval systems supports the characteristic of formalization, using the schema as it isdefined by the entity that is responsible for its creation and maintenance. This aspectalso determines that none of the resources copes with the ambiguity that exists in thesyntactic and semantic representation to, disambiguate for each concept and propertyof the candidates that are included in the schema. We have neither found thecharacteristics of multilingualism nor sense specification.

Characteristic Importance/type Description

Interoperability Obligatory/simple Possibility to establish relationships between concepts ofdifferent schemas

Formalization Obligatory/simple Possibility to realize or improve the formalization of aschema, regarding the management process

Interactivity Desirable/compound Possibility that the user participates actively in the SchemaManagement, in accordance to the Web 2.0 guidelines

Table IV.Characteristics of schemamanagement for theevaluation of systems forsemantic retrieval

LHT31,4

648

In this study, we have obtained criteria to be considered in a semantic retrieval systeminstead of answering what system obtains the best results, because in this contextsystems are constantly evolving.

3.1 Results of schema managementWith respect to interoperability, we have observed that metadata directories do notsupport this characteristic. The metadata registries do incorporate one-to-one


Disambiguation Obligatory/simple Possibility ability to eliminate structural andsemantic ambiguity of concepts, in order to facilitatethe conceptual retrieval

Semantic framework Obligatory/compound The scope in which the semantic and the conceptualretrieval of concepts is managed. The possible valuesare: None (0), Local, in the schema (1), Local, withrelationships between schemas (3), Global, betweenschemas that use a shared resource (e.g. Anontology) (5)

Multilingualism Desirable/simple Possibility to support multiple languages

Synonymy Obligatory/compound Possibility to solve problems that arise fromdifferent concepts with the same meaning

Scope Obligatory/simple The domain in which the semantics of the schemasto be managed are defined. It can be eitherhomogeneous or heterogeneous

Extensibility Desirable/compound Possibility to expand the representation of theschema semantics

Reusability Desirable/compound Possibility to reuse the representation of the schemasemantics

Modifiability Desirable/compound Possibility to modify the representation of theschema semantics

Language Optional/compound Possibility to represent the language that is use inthe formalization of the semantic

Table V.Characteristics of

semantic management forthe evaluation of systems

for semantic retrieval


Sense specification Obligatory/simple Possibility to express the concrete meaning of aconcept in the query process

Conceptual query Obligatory/compound Possibility to perform queries, according to themeaning of the concepts

Contextual query Obligatory/compound Possibility to obtain results that derive from theexisting relationships between concepts

Document retrieval Optional/simple Chance to obtain semantic documents that derivefrom schemas, as well as the schemas themselves

Table VI.Characteristics of queries

for the evaluation ofsystems for semantic

retrieval


649

relationships between schemas. Some special cases of ontology engines, such asWatson, analyze relationships between concepts.

Metadata registries and ontology directories often provide extra functionality to theusers so that they can incorporate new resources to the system. Metadata directories,similar to the ontology engines, are usually closed to user interventions, except for thequery processes.

3.2 Results of semantic managementWith respect to the semantic framework, metadata directories do not use the semanticsassociated to the concept; rather they only use the description tokens. In contrast to themetadata engines, ontology engines and ontology directories utilize the semantic thatis local to the schema, including relationships with other schemas. Likewise, only thesecategories present the characteristics of language and modifiability. The schemadefinition language that they use is either XML or RDF. On the other hand, thecorrespondence between schemas and the semantic representation model is aone-to-one relationship, which implies the revision and update of all correspondences.

With respect to synonymy, we have not detected it in metadata directories.However, we consider it partially covered in the rest of typologies, because theysupport the definition of one-to-one correspondences between concepts.

The scope of retrieval systems is wide and heterogeneous. As an example in thecase of metadata engines, LOV works with 322 vocabulary spaces. This resourceincludes statistics such as LOV distribution, LOV popularity and LOD popularity. TheDataHub also includes ratings, but they are scarcely implemented.

The reusability, defined as the ability to reuse the representation of the schemasemantics, is applied only by the ontology directories through the publication ofone-to-one alignments for their possible reuse.

3.3 QueriesConcerning the query process and the management of the obtained results, we analyzefeatures such as Sense specification, conceptual and contextual queries, and documentretrieval. From the point-of-view of semantic retrieval, differentiating betweenpolysemic meanings, we have not detected in any of the categories the possibility tosearch the concrete meaning.

Regarding conceptual queries, metadata directories base the retrieval process to thesyntactic search of the labels and attributes. In contrast, metadata engines, ontologyengines and ontology directories extend the searching by including meanings andrelationships between concepts in a generic environment, at the time that they permitthe establishment of a concrete semantic for the concept to be retrieved.

The possibility to extend queries with relationship concepts is present in metadataengines, ontology engines and directories. However, metadata directories do not extendthe results of concepts through their relationships.

Finally, with respect to document retrieval, schema, metadata and ontologydirectories only permit schema retrieval, while the corresponding engines permit theretrieval of documents that are instances of these schemas.

For each characteristic, we have obtained the product of the assigned value by thefactor of importance. Once the weighted values of each system are calculated, wecalculate the aggregate percentages for each category, in order to facilitate their

LHT31,4

650

interpretation. More specifically, for each of the defined categories (Schemamanagement, Semantic management and Query), we have summed the value oftheir characteristics and we have calculated the percentage of the above-mentionedsum over the maximum possible value, which would correspond to 100 percent. InFigure 1, we present the results that correspond to the evaluation of each method inpercentage and grouped by category.

In the schema management category, the metadata search engines and the ontologysearch engines and directories obtain the highest results (43.1). The ontologydirectories obtain this result mainly because they promote the participation of the userand support the definition of relationships between schema elements. The ontologysearch engines are positioned just below them due to their lesser ability of interactivitywith the user. The rest of the methods obtain noticeably lower values, as a result of thelack of support to the management of correspondences between elements, as well as alesser degree of interactivity with the user.

For the semantic management category (Figure 1), the ontology directories and theontology search engines obtain the best results (53.4). In this case, they highlight themanagement of relationships between concepts; their application scope, heterogeneouswith respect to the knowledge domain; the modifiability of the solution and thesemantic representation language employed. The decrease of the values for metadatasearch engines (43.3) is caused by the fact that these engines deal with a more restrictedscope, as well as the use of languages with less semantic expressivity forrepresentation. The metadata directories obtain the lowest value (6.1), a fact that can beattributed mainly to the restricted nature of the application environment.

In the Query category (Figure 1), the ontology search engines and the metadata searchengines obtain the best results (57.6), mainly due to their ability to perform contextualizedconceptual queries, as well as the possibility to obtain semantic documents. The nextvalue corresponds to the ontology directories (36.4). The decrease of their score is due tothe impossibility to obtain documents that are associated to the schemas. The decrease ofthe rest of the values is caused by the absence of contextualization of the results and thelocal use of the schema semantics. As a result, the previous points cause the schemadirectories and the metadata directories to get lower values.

The overall results of the evaluation of the methods (Figure 1) show that theontology directories achieve a good score (46.8), resulting from a positive evaluation

Figure 1.Results of the evaluationof each system grouped

by category


651

regarding the schema management and semantic management categories. Proceedingin descending order, the ontology search engines (52.4) owe this result to the positiveevaluation of the query and semantic management methods. In the case of metadatasearch engines (47.0), the obtained assessment arises from the tradeoff between thesemantic management and a good schema and query management. In the last place, wefind the metadata directories (7.1), due to shortcomings in the support of all theevaluated categories.

4. DiscussionIn this work, the definition of a semantic document is extended to other schemas andcodifications that contain a semantic description of document content. Standards liketopic maps or OWL can be represented with XML Schema and without the use of RDF.Rigorous studies of this field must not be limited to retrieval, maintenance and storageof RDF documents only. Our main motivation is that this type of semantic documentsconstitutes a key issue for the semantic description of other resources. Since thesevocabularies are considered as semantic documents, they must be retrievable by asemantic search engine. Nevertheless, it is true that the semantic web communityprefers open standards, like OWL or RDFS, than alternatives with proprietaryencoding format or results of open academic experiments (Strasunskas and Tomassen,2010).

Wei et al. (2008) has stressed the need to develop a formalized semantic searchframework. We believe that a desirable characteristic of semantic search is tointegrate this framework in the semantic web evaluation procedures. As an exampleof the challenges that arise on the semantic web, we can take a closer look to thelinked data proposals and its element sets and value vocabularies. The first problemthat arises is the linking architecture approach. There are scalability problems whenconnecting resources using one-to-one mappings between vocabularies. In fact if wecompute the potential number of alignments in the whole set of vocabularies taking2 at a time, we will have n! divided by 2! *(n-2)!, where n is the number ofvocabularies. The W3C Library Linked Data Incubator Group has undertaken greateffort of collecting and classifying value vocabularies and metadata element sets todecrease these possibilities, but it is a long-term project. Another solution is aunique central resource, which would connect to the rest of vocabularies, wouldresult into n-1 mappings between all possible concepts. But updating problems willstill remain: what effects have updates in vocabularys hierarchy or corrections indescriptions due to ambiguities? Besides, we have taken into account that values forthe elements can been drawn not just from values vocabularies but even from freetext. Finally, there are difficulties to cope with identifying and adaptingvocabularies. There is a need to identify and adapt the vocabularies, selecting themost appropriate among the candidates and leaving open the possibility of adaptingit to the resource without modifying the original vocabulary, thus reducing possibleambiguities.

Improving the process of identifying the essential functionalities, such as usability,in the implementation of a semantic retrieval system is a critical point for popularizingthese resources (Morato et al., 2012). Interaction with a larger set of user is an essentialelement that will help the semantic web and linked data technologies to achieve aneven greater degree of potential.

LHT31,4

652

5. ConclusionsWe have performed an evaluation of methods for semantic documents retrieval. Theresults of the evaluation indicate that, at the moment, many of the resources ofsemantic document retrieval lack the minimum of functionality in order to popularizetheir use. Some of the difficulties can be identified in ambiguity and lack offormalization of resource descriptions, difficulties in usability and operation, isolationof datasets that impede more exhaustive searches and the difficulty to carry outconceptual searches and navigation.

As it has been shown in the analysis only the ontology directories get hardly over 50percent in the evaluation. In our judgment, the current methods need to covercharacteristics that are essential for the management of semantic documents. Suchcharacteristics may include the formalization of the documents, their disambiguation,multiple language support and the semantic coverage of queries.

There are many problems that are difficult to measure. If we observe metadatavocabularies, we realize that selecting the right vocabularies is a tough task due to thelarge number of vocabularies in the cloud. The absence of URIs, the low usability andthe lack of consensus between overlapping vocabularies, are difficulties that we haveto overcome to facilitate the access of users to semantic web resources.

In this study we have proposed a mechanism to facilitate the comparison in asimilar way to query performance metrics in classical retrieval. Previous studies haveemphasized a descriptive approach to evaluate semantic search engines. We proposean approach that gives weight to each evaluation criteria facilitating the comparison inthe future.

As a work in progress, we are studying how to identify criteria related withtrustworthiness and link quality. Although it is noticeable that some search engineshave included some statistics to guide the user in the selection of a vocabulary, there isa lack of studies showing the real importance of this data in the user behavior.

References

Allemang, D. and Hendler, J. (2011), RDF-The basis of the Semantic Web, Semantic Web for theWorking Ontologist, 2nd ed., Morgan Kaufmann, Burlington, MA, pp. 27-50.

Amer-yahia, S. and Lalmas, M. (2006), XML Search: languages, Inex and Scoring, SIGMODRec., Vol. 36 No. 7, pp. 16-23.

Andago, M.O., Phoebe, T. and Thanoun, B.A.M. (2010), Evaluation of a semantic search engineagainst a keyword search engine using first 20 precision, Intern. Journal for the Advanc.of Science & Arts, Vol. 1 No. 2, pp. 55-63.

Batzios, A. and Mitkas, P.A. (2012), WebOWL: A semantic web search engine developmentexperiment, Expert Systems with Applications, Vol. 39 No. 5, pp. 5052-5060.

Bechhofer, S., Ainsworth, J., Bhagat, J., Buchan, I., Couch, P., Cruickshank, D., Delderfield, M.,Dunlop, I., Gamble, M., Goble, C., Michaelides, D., Missier, P., Owen, S., Newman, D.,De Roure, D. and Sufi, S. (2010), Why linked data is not enough for scientists,IEEE International Conference on eScience, IEEE Sixth International Conference one-Science, pp. 300-307.

Bikakis, N., Tsinaraki, C., Gioldasis, N., Stavrakantonakis, I. and Christodoulakis, S. (2013),The XML and sematic web worlds: technologies, interoperability and integration: asurvey of the state of the art, in Anagnostopoulos, I.E. (Ed.), Sematic Hyper/MutlimediaAdaptation, Vol. 418, Springer, Berlin, pp. 319-360.


653

Bizer, C., Heath, T. and Berners-Lee, T. (2009), Linked data the story so far, InternationalJournal on Semantic Web and Information Systems, Vol. 5 No. 3, pp. 1-22.

Buscaldi, D., Guerrini, G., Mesiti, M. and Rosso, P. (2003), Tag semantics for the retrieval ofXML documents, Proceedings of the 1st International Symposium on Information andCommunication Technologies, 24-26 September, Dublin, Ireland, Trinity College Dublin,Dublin, pp. 273-278.

Ding, L., Finin, T., Peng, Y., Pinheiro da Silva, P. and McGuinness, D. (2005a), Tracking RDFGraph Provenance using RDF Molecules, report TR-CS-05-06, Computer Science andElectrical Engineering, University of Maryland, Baltimore County, April 30.

Ding, L., Pan, R., Finin, T., Joshi, A., Peng, Y. and Kolari, P. (2005), Finding and rankingknowledge on the semantic web, Proceedings of the 4th International Semantic WebConference.

Esmaili, K.S. and Abolhassani, H. (2006), A categorization scheme for semantic web searchengine, AICCSA 06 Proceedings of the IEEE International Conference on ComputerSystems and Applications, IEEE Computer Society, Washington DC, pp. 171-178.

Fazzinga, B. and Lukasiewicz, T. (2010), Semantic search on the web, Semantic Web, Vol. 1,pp. 89-96.

Fensel, D., Angele, J., Decker, S., Erdmann, M., Schnurr, H.-P., Staab, S., Studer, R. and Witt, A.(1999), On2broker: Semantic-Based Access to Information Sources at the WWW,Proc.WWW Conf., Internet (WebNet 99), Honolulu, 25-30 October, pp. 25-30.

Fuentes, D. and Meja, A. (2013), Cycle management in semantic similarity among Wikipediasconcepts, degree thesis, University Carlos III.

Girit, H., Eberhard, R., Michelberger, B. and Mutschler, B. (2012), On the precision of searchengines: results from a controlled experiment, 15th Int. Conf. on Business InformationSystems (BIS 2012), pp. 260-271.

Guha, R., McCool, R. and Miller, E. (2003), Semantic search, WWW2003, Budapest, pp. 700-709.

Hogan, A., Harth, A., Umbrich, J., Kinsella, S., Polleres, A. and Decker, S. (2011), Searching andbrowsing linked data with SWSE: the semantic web search engine, Web Semantics:Science, Services and Agents on the World Wide Web, Vol. 9 No. 4, pp. 365-401.

Howarth, L.C. (2000), Creating a metadata-enabled framework for resource, available at: www.cais-acsi.ca/proceedings/2000/howarth_2000.pdf (accesed January 2013).

Jain, P., Hitzler, P., Yeh, P.Z., Verma, K. and Sheth, A.P. (2010), Linked data is merely moredata, in Brickley, D., Chaudhri, V.K., Halpin, H. and McGuinness, D. (Eds), Linked DataMeets Artificial Intelligence. Tech. Rep. SS-10-07, AAAI Press, Menlo Park, CA, pp. 82-86.

Kitchenham, B. (1996), DESMET: a method for evaluating software engineering methods andtools, available at: www.osel.co.uk/desmet.pdf (accesed June 2012).

Kobilarov, G., Scott, T., Raimond, Y., Oliver, S., Sizemore, C., Smethurst, M. and Lee, R. (2009),Media meets semantic web - how the BBC uses DBpedia and linked data to makeconections, European Semantic Web Conf. Semantic Web in Use Track, Crete.

Lanthaler, M. and Gutl, C. (2012), On using JSON-LD to create evolvable RESTful services,Proc.3rd Internat. Workshop on RESTful Design (WS-REST 2012) at WWW2012,ACM Press, Lyon, pp. 25-32.

Mangold, C. (2007), A survey and classification of semantic search approaches, Int. J. MetadataSemant. Ontologies, Vol. 2 No. 1, pp. 23-34.

LHT31,4

654

Milicic, V. (2011), Introducing hypernotation an alternative to linked data available at: http://milicicvuk.com/blog/category/hypernotation/ (accessed Ago 2013).

Morato, J., Sanchez-Cuadrado, S., Fraga, A. and Moreno Pelayo, V. (2007), Towards a socialsemantic web environment, El profesional de la Informacion, Vol. 17 No. 1, pp. 78-85.

Morato, J., Fraga, A., Andreadakis, Y. and Sanchez-Cuadrado, S. (2012), Eight steps towards thesocialisation of the semantic web, International Journal of Social and HumanisticComputing, Vol. 1 No. 4, pp. 347-362.

Nesic, S. (2010), Semantic document architecture for desktop data integration andmanagement, PhD thesis, Univ. Svizzera Italiana.

Palacios, V. (2010), Sistema de recuperacion conceptual mediante niveles semanticos en larepresentacion de esquemas de metadatos,[Conceptual retrieval system using semanticlevels in the representation of metadata schemas] PhD Carlos III University, available at:http://hdl.handle.net/10016/9332 (accesed June 2012).

Pastor-Sanchez, J.A., Martnez-Mendez, F. and Rodrguez-Munoz, J.V. (2012), SKOS applicationfor interoperability of controlled vocabularies in the field of linked open data,El Profesional de la Informacion, Vol. 21 No. 3, pp. 245-253.

Research Group Data and Web Science (University of Mannheim) (n.d.), Datasets in the nextLOD Cloud, available at: http://wifo5-03.informatik.uni-mannheim.de/lodcloud/ (accessedJanuary 2013).

Strasunskas, D. and Tomassen, S.L. (2010), On variety of semantic search systems and theirevaluation methods, Proceedings of the International Conference on InformationManagement and Evaluation, Academic Conferences Publishing, Cape Town, pp. 380-387.

Scheir, P., Pammer, V. and Lindstaedt, S.N. (2007), Information retrieval on the semantic web does it exist?, Proceedings of Lernen-Wissen-Adaption (LWA) 2007, pp. 252-257.

Tumer, D., Shah, M.A. and Bitirim, Y. (2009), An empirical evaluation on semantic searchperformance of keyword-based and semantic search engines: Google, Yahoo, Msn andHakia, 2009, 4th International Conference on Internet Monitoring and Protection (ICIMP09), available at: http://doi.ieeecomputersociety.org/10.1109/ICIMP.2009.16 (accessedAgo. 2013).

Uren, V., Lei, Y., Lopez, V., Liu, H., Motta, E. and Giordanino, M. (2007), The usability ofsemantic search tools: a review, The Knowledge Engineering Review, Vol. 22 No. 4,pp. 361-377.

W3C (2007), Semantic Annotations for WSDL and XML Schema, Farrell, J., Lausen, H. W3CRecommendation 28 August 2007, available at: wwww.w3.org/TR/sawsdl/ (accesedJanuary 2013).

W3C (2010), XML Linking Language (XLink) Version 1.1. DeRose, S., Maler, E., Orchard,D. Walsh, N. W3C Recommendation 06 May 2010, available at: www.w3.org/TR/xlink11/(accesed January 2013).

W3C (2011), Library Linked Data Incubator Group Final Report, Baker, T. et al. W3C IncubatorGroup Report 25 October 2011, available at: www.w3.org/2005/Incubator/lld/XGR-lld-20111025/ (accesed January 2013).

W3C (2012), OWL 2 Web Ontology Language XML Serialization (second edition), Motik,B. Parsia., B., and Patel-Schneider P.F. W3C Recommendation 11 December 2012, availableat: www.w3.org/TR/owl-xml-serialization (accessed January 2013).

W3C (2013), Ontologies available at: www.w3.org/standards/semanticweb/ontology/ (accessedJanuary 2013).


655

Waitelonis, J. and Sack, H. (2009), Augmenting video search with linked open data2009),2-4 September, Graz, Proceedings of International Conference on Semantic Systems 2009(i-semantics, Verlag der TU Graz, Austria.

Wei, W., Barnaghi, P.M. and Bargiela, A. (2008), Search with meanings: an overview of semanticsearch systems, Int. J. Commun. SIWN, No. 3, pp. 76-82.

About the authorsJorge Luis Morato is currently a Professor of Information Science in the Department ofInformatics at the Carlos III University of Madrid (Spain). In 1999, he received his PhD in LibraryScience from Carlos III University. Jorge Luis Morato is the corresponding author and can becontacted at: [email protected]

Sonia Sanchez-Cuadrado works as an Assistant Professor in the Department of Informatics atCarlos III University of Madrid. In 2007, she received her PhD in Library Science and DigitalEnvironment, designing a methodology for the automatic construction of knowledgeorganization systems and NLP.

Christos Dimou is a Visiting Lecturer at the Department of Informatics, at the Carlos IIIUniversity of Madrid. In 2010, he obtained his PhD in Electrical and Computer Engineering,Aristotle University of Thessaloniki, Greece, defining a framework for the performanceevaluation of software agents. His research interests include requirements engineering, softwareagents and information retrieval.

Divakar Yadav is an Assistant Professor in the Department of Computer Science andEngineering at Jaypee Institute of Information Technology, Noida and Carlos III University forthe last 12 years. His area of interests includes information retrieval, soft-computing, andoperating systems. He has participated, reviewed and organized many international and nationalconferences. He received his PhD in Computer Sc. and Engineering in 2010.

Vicente Palacios is currently working as Systems Engineer at the Carlos III University ofMadrid, where he is also a lecturer of Software Processes and Advanced Software Design.

LHT31,4

656

To purchase reprints of this article please e-mail: [email protected] visit our web site for further details: www.emeraldinsight.com/reprints

semantic web

Documents