ariadne: final implementation report

D12.5: Final Implementation Report

Authors: Dimitris Gavrilis, ATHENA RCChristos Papatheodorou, ATHENA RCEleni Afiontzi, ATHENA RCDimitra-Nefeli Makri, ATHENA RCCostis Dallas, ATHENA RC

Ariadne is funded by the European Commission’s 7th Framework Programme.

Theviewsandopinionsexpressedinthisreportarethesoleresponsibilityoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheEuropeanCommission.

ARIADNED12.5:Finalimplementationreport(Public)

Version:1.0(final) 26thJanuary2017

Authors: DimitrisGavrilis,ATHENARC

ChristosPapatheodorou,ATHENARC

EleniAfiontzi,ATHENARC

Dimitra-NefeliMakri,ATHENARC

CostisDallas,ATHENARCContributingPartners

JohanFihn,SND

OlofOlsson,SND

CarloMeghini,CNR

FrancaDebole,CNR

NicolaAloia,CNR

DouglasTudhope,USW

CeriBinding,USW

SebastianCuy,DAI

AchilleFelicetti,PIN

ARIADNED12.5(Public)

3

Table of Contents

1 ExecutiveSummary....................................................................................................7

2 Objectives...................................................................................................................8

3 ARIADNEInfrastructureOverview..............................................................................9

4 ARIADNEServicesforDataCollectionandIntegration..............................................12

5 ARIADNEServicesforDataAccessandManagement................................................15TheSemanticTripleStore................................................................................................................15TheIndexingandSearchEngine......................................................................................................16TheARIADNERegistry......................................................................................................................20

6 ARIADNEServicesforDataValidation.......................................................................20SchemaValidation...........................................................................................................................21Normalization..................................................................................................................................21LinkChecking...................................................................................................................................21SchematronRuleValidation............................................................................................................22

7 ACDMMetadataEnrichment....................................................................................23SubjectEnrichment..........................................................................................................................23

AATMappingandConversionofSubjectTerms.........................................................................................25TerminologicalInformationfortheARIADNEInfrastructure......................................................................26

GeographicEnrichment...................................................................................................................28TemporalEnrichment......................................................................................................................29OtherServices..................................................................................................................................29

8 ARIADNEServicesforMetadataQualityChecking.....................................................30MetadataCompleteness..................................................................................................................30MetadataConsistency.....................................................................................................................31

9 TheARIADNEPortal..................................................................................................33PortalMock-ups...............................................................................................................................33DevelopmentFramework................................................................................................................37InitialPrototype...............................................................................................................................37ARIADNEPortalFinalimplementation.............................................................................................40SourceCode.....................................................................................................................................41

10 SupportPortal......................................................................................................45

11 SummaryofImprovements..................................................................................46

12 Conclusions..........................................................................................................47

13 References...........................................................................................................48


4

AnnexI–MOReAPI.........................................................................................................49GetMetadataSources.....................................................................................................................49GetaspecificMetadataSource.......................................................................................................49CreateaMetadataSource...............................................................................................................50GetPackages....................................................................................................................................51GetaspecificPackage.....................................................................................................................51Getpackagedatastreams................................................................................................................52Getpackagetasks............................................................................................................................52Getpackagenotifications................................................................................................................53HarvestaPackage............................................................................................................................54IngestaPackage..............................................................................................................................54TransformaPackage.......................................................................................................................55EnrichaPackage..............................................................................................................................55PublishaPackage.............................................................................................................................56RejectaPackage..............................................................................................................................56DeleteaPackage..............................................................................................................................57WithdrawaPackage........................................................................................................................57Getmappings...................................................................................................................................58Getrights.........................................................................................................................................58Getenrichmentplans......................................................................................................................59Getprojects.....................................................................................................................................60Getschemas....................................................................................................................................60

AnnexII–RegistryAPI.....................................................................................................62GetDataResources.........................................................................................................................62GetaDataResource........................................................................................................................63CreateanewDataResource............................................................................................................64GetDistributions..............................................................................................................................65GetaDistribution............................................................................................................................66CreateanewDistribution................................................................................................................67GetDataFormats.............................................................................................................................68GetaDataFormat...........................................................................................................................69CreateanewDataFormat...............................................................................................................70Getfoaf:Agents................................................................................................................................71Getafoaf:Agent...............................................................................................................................72Createanewfoaf:Agent..................................................................................................................73

AnnexIII–ThesauriAvailablethroughMORe...................................................................75


5

Acronymsandabbreviations

ACDM ARIADNECatalogueDataModel(seeAnnex)

API ApplicationProgrammingInterface

BC/AD BeforeChrist/AnnoDomini,i.e.datesbefore/afterJesus'sbirth

CIDOC InternationalCommitteeforDocumentationoftheInternationalCouncilofMuseums

CIDOC-CRM CIDOC-ConceptualReferenceModel;ISO21127:2006standard:Areferenceontologyfortheinterchangeofculturalheritageinformation

CRUD Create,Read,Update,DeleteoperationsconductedthroughaRESTAPI(seebelow)

CSS3 CascadingStyleSheets(level3)

DC DublinCore(metadatastandard)

DCAT DataCatalogVocabulary,anRDFvocabularydesignedtofacilitateinteroperabilitybetweendatacatalogspublishedontheWeb

GIS GeographicalInformationSystems/Services

HTML HyperTextMarkupLanguage

IEC InternationalElectrotechnicalCommission

ISO InternationalOrganizationforStandardization

ISO/IEC11179 Internationalstandardforrepresentingmetadataforanorganizationinametadataregistry

Java/Javascript Programminglanguages

JDCB JavaDatabaseConnectivity

JSON JavaScriptObjectNotation

lat/lon Latitudeandlongitudecoordinates

LDAP LightweightDirectoryAccessProtocol

LOD LinkedOpenData

Log4J AJava-basedloggingpackage

METS MetadataEncodingandTransmissionStandard

MySQL Anopensourcerelationaldatabasemanagementsystem

OAI OpenArchivesInitiative

OAI-PMH OAI-ProtocolforMetadataHarvesting

OAI-ORE OAI-ObjectReuseandExchangeSpecification


6

OAuth Authenticationprotocol

ODBC OpenDatabaseConnectivity

PHP PHPHypertextPreprocessor(server-sidescriptinglanguageformakingdynamicandinteractivewebpages)

RDF ResourceDescriptionFramework

REST RepresentationalStateTransfer

RESTAPI RESTfulApplicationProgrammingInterface

RESTservices RESTfulwebservices

SAML SecurityAssertionMarkupLanguage

SKOS SimpleKnowledgeOrganizationSystem

SKOSify Transformingathesaurus,classificationsystemorotherknowledgeorganizationsystemintotheSKOSformat

SPARQL SPARQLProtocolandRDFQueryLanguage

SQL StructuredQueryLanguage

Syslog Standardforcomputermessagelogging

UI UserInterface

URI UniformResourceIdentifier

W3C WorldWideWebConsortium

WGS84 WorldGeodeticSystem1984

XML ExtensibleMarkupLanguage

XSD XMLSchemaDefinitionLanguage


7

1 Executive Summary

This document is a deliverable (D12.5 Final Implementation Report) of the ARIADNE project(“Advanced Research Infrastructure for Archaeological Dataset Networking in Europe”), which isfundedundertheEuropeanCommunity'sSeventhFrameworkProgramme.ItpresentsresultsoftheworkcarriedoutinTasks:12.3“ImplementingIntegration”.

TheoverallarchitectureoftheARIADNEinfrastructurehasbeenlaiddownbytheuserrequirementsanalysis (D12.1), and the infrastructure specifications (D12.2). Initial implementation has alreadybeendescribedinD12.3.ThegoaloftheARIADNEinfrastructureistointegratedataandmetadatafromdifferentproviders intoonecommonschema,andalsotoprovidesemantic integrationalongdifferentaxes(e.g.subject,space,time).Thisintegrationintendstoprovideusefulanduser-friendlyinformation services for archaeology. The services are intended to be available not only toresearchersandrelatedstakeholders,butalsotoawiderrangeofpotentialusersrequiringaccesstocollectionsanddatasets.

The main goal of Task 12.3 is to accomplish the infrastructure implementation of ARIADNE. Theinfrastructure is specifiedonD12.2and includesservices suchas the registry,vocabulary services,metadataenrichmentservices,preservationservices,etc.withthefollowingcomponents:

a) atriplestorewithsemanticfeaturesb) theARIADNEportalc) theMOReaggregationinfrastructured) anindexercomponent(Elasticsearch)e) ametadataqualitymeasurementservicef) asetofenrichmentservices

TheAPIspecificationspresentedintheAnnexesareespecially important,astheyallowdeveloperstobuildonthe infrastructureanddeployservices.TheAPIspresentedareREST-basedandrequiredataformatssuchasJSON,XMLandRDF.


8

2 Objectives

ThemainobjectivesofWP12are:• ToadaptinfrastructuresprovidedtoARIADNEforintegration• To design, implement and configure the necessary mechanisms (crosswalks, mappings) and

resourcesforinteroperability• Tosetuptheinternal(APIs)andexternal(human)interfacestoaccesstheintegratedresources

WP12comprisesthefollowingtasks:

• Task12.1AssessmentofUseRequirements• Task12.2DesignandSpecifications• Task12.3ImplementingIntegration• Task12.4Testing

The main goal of Task 12.3 is to present the infrastructure implementation of ARIADNE. Theinfrastructure is specifiedonD12.2and includesservices suchas the registry,vocabulary services,metadataenrichmentservices,preservationservices,etc.(fig.1).

Figure1.TheARIADNEdataintegrationarchitecture.

ThisdeliverablereportstheresultsofTask12.3.Itconsistsofareportonthefinalimplementationofthecorecomponentsoftheinfrastructure.


9

3 ARIADNE Infrastructure Overview

Thegoalof theARIADNE infrastructure is to integratedataandmetadatafromdifferentprovidersinto one common schema, and also to provide semantic integration along different axes (e.g.subject, space, time). This integration intends to provide useful and user-friendly informationservices for archaeology. The services are intended to be available not only to researchers andrelatedstakeholders,butalsotoawiderrangeofpotentialusersrequiringaccesstocollectionsanddatasets. The overall architecture of the ARIADNE infrastructure has been laid down by the userrequirements analysis (D12.1) and the infrastructure specifications (D12.2). Initial implementationhasalreadybeendescribedinD12.3.

AtLevel1,dataiscreatedbyresearchprojectsandgroups,andsubsequentlystoredininstitutionalrepositoriesatLevel2.Then,atLevel3,thisdataisaggregatedbyhigher-leveldatamanagers,suchas data centers, portals and thematic information gates. Finally, at Level 4, the ARIADNEinfrastructureintegratesallinformationandprovidesasetofservicesthroughtheARIADNEportal.Theseservicesinclude:resourcediscovery,integrateddatasearchservices,andotherservices(suchaspreservation,quality, etc.).DeliverableD12.1 “UserRequirements” [1],whichoutlines theuserneedsfortheinteroperabilityframeworkoftheARIADNEInfrastructure,setsoutageneralviewoftheintegrationfunctionalities,organizedinfoursuccessivelevels(fig.2)[1].

Figure2.TheARIADNEdataintegrationarchitecture.

TheARIADNE infrastructureservicesaimtoprovideusefulservicesandtools tothearchaeological


10

community.Thus,asimpleyeteffectiveportaldesignisimperativeinordertoachievethesegoals.Thisportalconsistsofthreemainpillars:

a) Resourcediscoveryb) Dataintegrationservicesc) Otherservices

Theresourcediscoveryservices relydirectlyontheregistry,whereas thedata integrationservicesprovidemoreintelligentwaysofdataintegrationthroughLODandNLPtechnologiesorthroughtheARIADNECRM(conceptualreferencemodel).

Theservicesthatfacilitatetheintegration(fig.1)aredescribedindetailinDeliverableD12.2.

The implementation of the infrastructure focuses on the creation and aggregation of contentaccordingtotheACDMmodelspecification[4](thecurrentversionis2.6).Accordingtothecontentaggregationworkflow(fig.3),theMOReaggregator(seerespectivesectionbelow)isusedtoharvestexistingcontent inotherschemasandmapittoACDM,or importdirectly frombatchesofXMLorExcelfiles.ACDMencodedinformationisthenpushedtoanRDFstoreandtotheRegistryservice.AnotherserviceisusedtosynchronizecontentfromtheRDFstoretoanElasticsearchindexservice.The rest of the services within the infrastructure consume these three APIs in order toaccess/managethisinformation.TheAPIsare:

1. TheRDFstoreSPARQLendpoint2. TheRegistryRESTAPI3. TheElasticsearchAPI


11

Figure3.TheARIADNEdataaggregationworkflow.


12

4 ARIADNE Services for Data Collection and Integration

InARIADNEtheaggregationservice iscomprisedofasetof toolsandservicesthataimtoharvestcontentrelatedtoarchaeology,andtocleanandtransformittoacommonformat(inthiscasetheACDM).ThiscontentisthenstoredpersistentlyinanRDFstoreandanindexserver.Thisprocessisintegral to the data integration process of the project, which focuses on integration of datascatteredamongstdiversecollections,datasets,unpublishedfieldworkreports(greyliterature),andpublications. Thisprovidesresearcherswiththeabilitytouseheterogeneousdistributeddatasetsasanintegralcomponentofarchaeologicalresearchmethodologies.ContentaggregationisnecessaryinordertomapallcontentavailablebytheARIADNEpartnersinacommonschema(inthiscase,theACDM),soastofacilitatetheintegrationprocess.Aggregationincludesanumberofsteps,suchasvalidation,cleaningandenrichment.

Inorder toaddress theserequirements, theMetadata&ObjectRepository (MORe)aggregator [6]has been employed in ARIADNE (http://more.dcu.gr/). The MORe aggregator, developed by theDigitalCurationUnit–IMIS,AthenaR.C.,hasbeeneffectivelyusedindiverseprojects(CARARE,3D-ICONS,andLoCloud)toaggregateandenrichmillionsofrecordsanddeliverthemtotheEuropeanaplatform (http://www.europeana.eu).MORe provides a flexible architecture, and is used here toaggregatecontent invarious formats,andfromavarietyofsources, totheRDFstore, theregistryandtheElasticsearch.MORe incorporatesamicro-serviceorientedarchitecture(fig.4),supportingcore services for input, validation, transformation, eEnrichment and publication of the ingestedcontent.

Thecoreservicesaredesignedinamodularwaysotheycancombineoneormoremicro-servicestoaccomplishthevarioustasks.Thisapproachallowsthesystemtobeextendedmoreeasily,andalsore-use certainmicro-services. The stackable andmodular architectureofMORe (fig. 4) presents adataaccess layerused toaccessdataon the storageanda core services layer, encompassing thevarious core services management. The different core management services are responsible fororchestrating numerous micro-services. For example, the enrichment management services areresponsiblefororchestratingandstreamliningtheexecutionoftheenrichmentmicro-services.


13

Figure4.TheMORearchitecture.

Otherimportantarchitecturalaspectsofthesystem(fig.5)providethecapabilityofusingmultiplestoragetechnologiessimultaneously.Thedataaccesslayer,whichisinstantiatedthroughastorageAPI, provides seamless access todataobjects regardlessof the storage inwhich they reside. Twoadditional importantcomponentsofMORearethemessagingand loggingservices.Themessagingservice is responsible forprovidingmessagingacross services. It isbasedon theRabbitMQAMPQbroker, a framework that provides scalability and elasticity. The logging service provides amechanismformonitoringalltheservicesintheaggregationinfrastructure.ItisimplementedbasedontheGraylog2loggingframework,whichprovidesamechanismforstoringlogsofanyformatandfrom any service. Graylog2 provides a way of organizing these logmessages (using streams) andpresentingtheminauser-friendlyway.


14

Figure5.SignificantarchitecturalaspectsofMORe.

Themainaggregationworkflowispresentedbelow(fig.6).Aninformationpackageisharvestedbyamicro-service and is pre-validated (pre-validation includes basic validation, such as a record to bestructurally checked to see if it is well formed). The package can be either deleted or ingested.During ingest, thepackage isstored inthestorage layer (it firstgoes intotemporarystorage),andthenitcanbeeitherrejectedortransformedintoacommonschema(inthiscasetheACDM).Duringtransformation,thenewrepresentationisvalidatedandindexed.Afterithasbeentransformed,thepackagecanberejected,enrichedorpublished.Inthecaseofenrichment,anewrepresentationiscreated (enriched ACDM), which has to be validated and indexed again. During the publishingprocesstheuserselectsthepublicationtarget.

InthecaseofARIADNE,theintermediateschemaisACDM.Thepublicationtargetsthatmaybeusedare:

- PublishtoElasticsearch- PublishtoVirtuosoTripleStore- PublishtoRegistryAPI


15

Figure6.MOReAggregationworkflow.

5 ARIADNE Services for Data Access and Management

The ARIADNE services for data access andmanagement consist of a layer that provides differentways of allowing access to the available data sources. These are native access to the RDF store,access through an Elasticsearch, and access through the Registry. Although they access the sameinformation, each might contain only a subset (e.g. Elasticsearch), or provide access throughdifferentprotocolsandformats.

Furthermore, as ARIADNE enables trans-national access of researchers to data centres, tools andguidance, it is important to provide multiple access points in the data sources. This means thatthrough thedataaccessmanagementservices, researcherscanpreviewtheirdatasets indifferentformats,suchasRDF,JSONorXML.

The Semantic Triple Store

ARIADNEimplementsanRDFstoretoholdthecataloginformationrepresentedinRDFandavailablethrough a SPARQL endpoint. Taking into account the complexity of the ACDM, an RDFrepresentationof theARIADNECatalog is necessary to support the complexqueries required. Forthatreason,theACDMcanberepresentedasRDFandURIsareassignedforeachtypeofresource.


16

ThedataintheRDFstoreisupdatedbyMORedynamicallyduringcontentaggregation.TheVirtuososemantic technology (http://virtuoso.openlinksw.com/) has been used for implementing theARIADNEtriple store.Virtuosowasselectedprimarilydue itsbetterperformance,especiallywhenhandlinglargeamountsofcontent.Inaddition,Virtuosohasbeenavailablesince1998,asopposedtoSesame,whichwasreleased in2004.Virtuoso isalsoavailable inmanyprogramming languages(.Net,C,C#,C++,Java,JavaScript,Perl,PHP,Python,Ruby,VisualBasic),whereasSesameisavailableto Java,PHPandPython. Themost important characteristics thatmakeVirtuosomore favourableover Sesame are its capability to partition the data and being able to be setup in a clusteredenvironment.TheSPARQLendpointandascreenshotcanbeseenbelow(fig.7).

SPARQLConsoleforVirtuoso:

http://ariadne-registry.dcu.gr:8890/sparql

Figure7.ScreenshotoftheARIADNESPARQLendpoint.

InordertostoretheentirecataloginRDF,anRDFrepresentationofACDMmustbedefined.ThishasbeenaccomplishedthroughasimplemappingprocesshandledinternallybytheMOReaggregator.

The Indexing and Search Engine

The Elasticsearch component provides the indexing capabilities required in order to power theARIADNE portal. Elasticsearch (ES) is based on thewell-known Lucene indexing engine (the samethat SolR uses) and provides extremely low response times for performing full text search andfacetedbrowsing.ThetwomainreasonsforselectingElasticsearchoverothertraditionalinterfacesto SolRare its simpleandeffectiveRESTfulAPI (fig. 8) and its clustering capabilities. Elasticsearchprovidesveryrobustandeasytouseclusteringsupport.


17

The primary format that Elasticsearch accepts through its API is JSON. Thatmeans that all ACDMrecordsmustbefirstmappedintoJSONinordertobepushedtoES.

ElasticsearchRESTAPI:

http://ariadne-registry.dcu.gr:9200/_search

Figure8.ScreenshotshowingElasticsearchRESTinterfacethroughWebhttpclient.

SincetheprimarymissionofESistofacilitatetheResourceDiscoverysectionoftheARIADNEPortal,onlythefeaturesassociatedwiththisfunctionalityarekept.Furthermore,thesearchresultsshouldbespecifiedinawaythatminimisessubsequentAPIcalls.Anexampleofsucharecordcanbeseeninthetable1.

Table1.ExampleofaJSONrepresentationofanACDMrecord{ _index: "dataresources", _type: "dataset", _id: "46949", _score: 1, _source: { accessRights: "Unrestricted access for all registered EASY users.", modified: "2010-01-11", distribution: { id: 46, title: "EASY OAI-PMH", modified: "2014-01-01", issued: "2014-01-01", description: "EASY OAI-PMH Endpoint", OAI-PMHServerURI: "http://easy.dans.knaw.nl/oai/", }, type: "dataset",


18

description: "Onderzoeks- en adviesbureau voor Bouwhistorie, Archeologie, Architectuur- en Cultuurhistorie (BAAC bv) heeft een archeologisch bureauonderzoek en inventariserend veldonderzoek met behulp van boringen (karterende fase) uitgevoerd op een terrein in de bebouwde kom van Tilburg. Tijdens het veldonderzoek is aangetoond dat de bodem is verstoord tot in de Chorizont van de dekzandafzettingen. Archeologische indicatoren werden daarbij niet aangetroffen. De verwachte duinvaaggronden zijn niet aangetroffen. Hieruit kan worden geconcludeerd dat aan het plangebied een lage archeologische verwachting voor archeologische resten uit alle perioden kan worden toegekend.", rights: "BAAC bv", id: 46949, language: ["nl"], subject: "Event/intervention databases", title: "Tilburg Bredaseweg 421", originalId: "DMO_ID easy-dataset:22059", keyword: ["50F", "Bredaseweg 421", "Tilburg", "Noord-Brabant"], creator: [{ id: 480, name: "Boshoven, E.H.", type: "person" }], spatial: [{ id: 34449, coordinate_system: "http://www.opengis.net/def/crs/EPSG/0/4326", lon: 5.04075, label: "51.55941102,5.04075138", lat: 51.5594 }], landingPage: "http://www.persistent-identifier.nl/urn:nbn:nl:ui:13-dox-y9f", issued: "2008-01" } }

In order to provide accurate and robust thematic based results, an indexwith the complete AATterms was created. This index contains the complete AAT hierarchy plus the content providermappingsfromtheirnativesubjectstoAAT.AnexampleofanAATispresentedinfig.9.

{"_index":"aat","_type":"terms","_id":"300201790","_score":1,"_source":{"prefLabels":[{"label":"onderarmstukken","lang":"nl"},{"label":"avambrazos","lang":"es"},{


19

"label":"vambraces","lang":"en"},{"label":"arambrazo","lang":"es"},{"label":"armguards","lang":"en"},{"label":"arm-guards","lang":"en"},{"label":"avambrazo","lang":"es"},{"label":"vambrace","lang":"en"},{"label":"guards,arm","lang":"en"},{"label":"onderarmstuk","lang":"nl"}],"prefLabel":"vambraces","providerMappings":[{"sourceURI":"http://purl.org/heritagedata/schemes/mda_obj/concepts/97103","matchURI":"http://www.w3.org/2004/02/skos/core#broadMatch","providerId":1104,"sourceLabel":"ARMGUARD"}],"id":"300201790","broader":[{}],"uri":"http://vocab.getty.edu/aat/300201790","altLabels":[],"scopeNote":"Termappliedtotheensembleofplatearmorpieces forthearmbelowtheshoulder,consisting of two cannons linked by a cowter, in use in Europe from the 14th to the 17th century. Alsosometimesusedtomeancannonswornontheforearmonly,includingsuchancientGreekpieces."}}

Figure9.TherecordofanAATterm.


20

The ARIADNE Registry

TheregistryserviceisasinglepointwhereallARIADNEresourcesarestoredandmadeavailable.Itprovides access to all ACDM records, and allows users to create/edit them both through a RESTinterface and through a Web UI. The registry consists of an SQL database and a REST API thatprovidesaccesstothedata.TheuserscanregisteranAPIkeyandusetheRESTAPItoperformCRUDoperationsonallentities.AnexampleofanAPIcallisasfollows:

ThefullAPIdocumentationisprovidedinAnnexII.TheWebUIhasbeenpresentedanddocumentedintheACDMmodelspecificationreport.Userswithoutanycontentinanexistingformatandavailablethrougharepositorycanusetheregistryservicetocataloguecontentfromscratch,manuallyorthroughtheRESTAPI.

6 ARIADNE Services for Data Validation

This sectiondocuments the listof services thatcouldbeused toperformvalidationon theACDMXSD.Validationisperformedduringcontentingestandcreation,andisfacilitatedautomaticallybyMORe.AllvalidationservicesprovideareportingmechanismthroughtheUItoalerttheuser.


21

Figure10.Screenshotofthethematicandspatialvalidationservice.

Apart from the standard validation reporting mechanism, when a certain type of information isdetected (spatial, temporal, thematic), respective blocks of information appear alerting the uservisually.Ascreenshotpresentinganexampleforthematicandspatialinformationcanbeseeninfig.10.

Schema Validation

This service provides schema validation for every XML record ingested. This functionality isautomaticallyprovidedbyMORe.SchemavalidationrequiresanXSD(inthiscaseACDMXSD).

Normalization

This service performs normalization of the ingested datasets. This process ensures the properidentitymanagementofeachresourcebymatchingproviderandcollectioninformation,alongwithnativeidentifiers,etc(fig.11).

Link Checking

ThelinkcheckingservicerequiresalistofelementsthatcontainURLs,forwhichitattemptstocheckifthelinksareavailable(or,else,itreturnsanHTTPerrorcode).ThisserviceisalreadyprovidedbyMORe, but requires a list of elements to check. Elements thatmay contain links and need to becheckedhavetobemarkedbeforehand.


22

Figure11.Theaggregationworkflownormalizationperspective.

Schematron Rule Validation

ThisserviceperformsmorecomplexchecksonanXMLrecord.Itprovidesamechanismtoperformcomplex conditional checks, and also provide feedback back to the user in a human-readablemechanism. The mechanism is provided by MORe but requires a Schematron rule for ACDM. AscreenshotofanindicativeSchematronrulevalidationreportcanbeseeninfig.12.

MOReStorage

Registry GUI

OAI-PMH

EXCEL

…..

Harvest

ACDM Schema

Validation

Normalisation

Registry API

Schematron Rule

Validation

Mapping

Link Checking

Enrichment

Purge Records

Publish

Elastic Search

Virtuoso RDF Store


23

Figure12.ScreenshotofasampleSchematronrulevalidationreport.

7 ACDM Metadata Enrichment

The systemprovidesa service formetadataenrichment that consistsof a seriesofmicro-servicesthatcanbeusedeffectivelytoaugmentACDMcontent.Thesemicro-servicescanbeeitherusedasenrichmentservicesinMORe,ordirectlybytheARIADNEpartnersandservices.Inthissection,alistofavailablemicro-servicesispresented.

Once the metadata enrichment services are registered in MORe, users can utilize them byincorporatingthemintoenrichmentplans.Enrichmentplansareawayofstreamliningandexecutingenrichmentservicestogether.Theyapplytooneschemaandcanbere-usedindifferentpackages.Thismeansusers candefine themonce,andapply them forall incomingpackageswith the samecharacteristics.

Although enrichment services can perform a wide variety of operations, only the main ones arepresented in this section, and in particular those concerning subject, special and temporalenrichment.

Subject Enrichment

Thethematicservices focusonenrichingsubject-related information.Thisenrichmentprocesscaneither be automatic (e.g. by automatically matching concepts from vocabularies) or manual (byallowing users to create collections of concepts and attach them to all items in a package). Allthematic enrichment services have access to the large number of vocabularies listed inAnnex III.Thesevocabulariesarestandardised,publishedandSKOSified,andcontainhundredsof thousandsofterms.


24

ThebrowsingandsearchingforthesevocabulariescanbeaccomplishedeitherthroughaRESTAPIorthroughtheMOReUI.Anindicativescreenshotofthelattercasecanbeseeninfig.13.

Other subject-related services have access to other resources, such as Wikipedia and DBpedialemmas.

Figure13.ScreenshotofabrowsingforvocabulariesintheMOReUI.

Inthefollowingtables,theimplementedsubjectrelatedenrichmentservicesarepresented.

Service: Subjectcollections

Status: ProvidedasaservicethroughMORe

Description: Thisserviceallowstheusertocreatecollectionsofsubjecttermsfromstandardizedthesauri.Thesesubjectcollectionsarethenusedtoenrichtheitemsofapackage(byaddingtheconceptsofthecollectiontotheitemsofthepackage).


25

Service: Automaticmetadataenrichment


Description: A string (e.g. dc:subject or dc:description) is submitted to the service and a list ofrelevant concept terms are retrieved (basedon a list of 29 SKOS vocabularies). Theelementdc:subjectisupdated.

Service: DBPediaandWikipediaautomaticenrichment


Description: Astring(e.g.dc:title,dc:subjectordc:description)issubmittedtotheserviceandalistofrelevantWikipediaandDBPediaURLsareretrieved.Anopen-annotationsbasedlist(RDFencoded)isreturnedthatannotatesthetext.

Service: Subjectmappings


Description: This service allows the user to upload or create a list ofmappings from their ownnativesubject terms(found intheirprovidedpackages) toAAT.ThesemappingsareusedtoenrichtheincomingpackageswithAATterms.

AAT Mapping and Conversion of Subject Terms

In order to ensure interoperability and facilitate integration, all ACMD recordsmust contain AATterms(AATstandforArtsandArchitectureThesaurusandreferstothefacetedthesaurusdevelopedbytheGettyFoundation,andintendedfor“vocabularycontrolofmuseum,culturalheritageandartcollections”). Thus, a translation service, developed by the Hypermedia Research Group at theUniversity of South Wales, is provided to allow partners to upload their native subjectterms/vocabularies,mapthemtotheAAT,andexportamappingfile.Thismappingserviceproducesthemappingrulesinvariousformats(suchasJSONandCSV).Thetoolisavailableat:

http://heritagedata.org/vocabularyMatchingTool/

The service comes as a web application (thus only a web browser is needed) and provides anintuitiveUIinterface(fig.14)consistingmainlyoftwocolumns(leftforthesourcevocabulary,rightforthetargetvocabulary).


26

Figure14.ScreenshotoftheVocabularyMatchingTool.

Userscandefinetheconceptmappings,thensaveandexporttheminvariousformats.AnexampleofaJSONencodedmappingcanbeseen inTable2.ThismappingcouldbealsouploadedintotheMORe subject translation enrichment service and used to automatically transform native subjecttermstoAATconcepts.

Terminological Information for the ARIADNE Infrastructure

The use of thesauri, and especially the required use/existence of at least 1 AAT termper record,addsacertaincomplexitythatrequiresspecialhandlingbytheinfrastructure.Furthermore,hands-ontestingoftheportal(e.g.throughrunningdifferentqueries)broughtouttheneedforsemanticexpansionanduseofmultilingualsearch.Theinformationflow(regaringsubjects)canbeseeninfig.15,inwhichcontentprovidersprovidetwokindsofsubjectterms:a)nativesubjects(couldbeanything,SKOSified,simpleterms,etc)b)providedsubjects(havetobeAAT)ProvidersthatdonotprovideAATtermshavetomaptheirnativesubjecttermstoAATusingthevocabularymappingtool.ThistoolinturnfeedsallthemappinginformationtoMOReandassociatesitwiththeprovider.Thus,whenapackagefromaproviderisreceivedthatcontainsnativesubjectsandamappingexists,thederivedsubjectisautomaticallycomputedandusedtoenrichtherecord.

Table2.ExampleofaJSONrepresentationofanmappingrd[{


27

"created":"2015-04-14T13:57:18.206Z","sourceURI":"http://purl.org/heritagedata/schemes/mda_obj/concepts/96909","sourceLabel":"ADZE","targetURI":"http://vocab.getty.edu/aat/300023553","targetLabel":"adzes","matchURI":"http://www.w3.org/2004/02/skos/core#closeMatch","matchLabel":"closematch","notes":"","creator":"anonymous"},{"created":"2015-05-11T14:25:15.011Z","sourceURI":"http://purl.org/heritagedata/schemes/mda_obj/concepts/95253","sourceLabel":"COMPASS","targetURI":"http://vocab.getty.edu/aat/300196707","targetLabel":"compasses(directionindicators)","matchURI":"http://www.w3.org/2004/02/skos/core#closeMatch","matchLabel":"closematch","notes":"","creator":"anonymous"}]

Figure15.HandlingofsubjecttermsthroughtheARIADNEinfrastructure.

DuringtheenrichmentandpublicationprocesstheAATthesaurusisusedtoperformthefollowingforeachACDMrecord:-expandtheAATtermsinordertoincludethebroaderGenericterm-expandtheAATtermsinordertoincludethealtLabels-expandtheAATtermsinordertoincludethemulti-linguallabels(prefLabelsandaltLabels)inordertofacilitatemulti-lingualsearch.


28

Asmentioned, in the elastic search a separate AAT index is published that includes the providermappings(undereachAATtermthenativesubjectsareassociatedaccordingthemappings).

Geographic Enrichment

Thespacerelatedenrichmentservicesfocusonenrichingspatialcontentandcurrentlytheyperformfourmaintasks:

• Geo-coding• Reversegeo-coding• Coordinatetranslation• Geo-normalization

Thevariousavailablemicro-servicescanbeseeninthefollowingtables.

Service: Geonames


Description: 1. A string (e.g. place Label) is submitted for geo-coding and theLatitude/Longitude coordinates are returned. The lat/lon elements areupdated.

2. Asetoflat/loncoordinatesaresubmittedandaplacenameisreturned.Theplacenameisupdated.

APIavailableat:http://api.geonames.org/

Service: DAIGazetteer


Description: Astring(e.g.placeLabel) issubmittedforgeo-codingandthelat/loncoordinatesarereturned.Thelat/lonelementsareupdated.

APIavailableat:http://gazetteer.dainst.org/search

Service: PleiadesPlus


Description: ThePleiadesPlusancientplacenamesserviceallowsresolvingancientplacenamestomodernnames.

APIavailableat:https://github.com/ryanfb/pleiades-plus


29

Service: BritishNationalGrid→WGS84Transformation


Description: A string containing the grid coordinates is submitted for transformation and theWGS84lat/loncoordinatesarereturned.Thelat/lonelementsareupdated.

Service: TransformationbetweenEPSGcodes


Description: The lat/lon coordinatesare transformed fromanEPSGcode toanother. The lat/lonelementsareupdated.

Service: Geo-normalization


Description: A string that contains a concatenated string of coordinates is submitted for geo-normalization. The user indicates the delimiter and the lat/lon coordinates arereturned.Thelat/lonelementsareupdated.

Temporal Enrichment

ARIADNEpartnershavecontributedtothePeriodOgazetteerofperiodassertions,andPeriodOURIsareavailableforintegrationwithMORe.Moreinformationcanbefoundathttp://perio.do/.

Service: PeriodO


Description: ThePeriodOserviceprovidesaSKOSifiedcollectionofperiodnames.Thismakesitpossibletoresolveperiodnamestoabsolutedates.Periodnamesareencodeddifferentlydependingparameterssuchasthegeographicalcoverage.

Other Services

This section currently includes a language identification service to automatically detect languageinformation based on text. It is based on Apache Tika and contains trainedmodels for all majorlanguages.


30

Service: LanguageIdentification


Description: Thisservicedetectsthelanguageinwhichapieceoftextiswritten.Atextelementissubmittedandthelanguageisreturned.Thexml:langattributeisupdated.

8 ARIADNE Services for Metadata Quality Checking

This sectionaimsatdocumenting the listof services thatcouldbeused toprovidequality relatedinformationforACDMrecords.Thesemetadataqualityservicesmainlyfocusoncompletenessandconsistency.

Metadata Completeness

Completenessreferstometadatacompleteness,estimatedforanentirepackage(atitemlevel)andfallsundertwoseparatecases:

1. Completenessofimportantelements2. Completenessofmandatory/recommendedelementsets

The first case aims to visually provide the completeness of a list of the 10-15 most importantelementsoftheACDM(fig.16).

Thesecondcaseaimstoprovideinformationonthecompletenessofallelements,usingtwofacets:mandatoryandrecommendedsets(fig.17).


31

Figure16.Metadataqualityinformationprovidedtotheuser.

Metadata Consistency

This service allows to check certain elements for consistency issues. Consistency refers to thefollowingcases:

1. URLformatchecking(forexample:elementslikerdf:aboutshouldhaveaURLasvalue)2. Dateformat(basedonISOdatesformat)3. Person names (regex patterns that could detect patterns such as: Name Surname, Title

NameSurname,etc.)


32

Figure17.Metadataqualityinformationprovidedtotheuser.


33

9 The ARIADNE Portal

ThissectionpresentsthefinalimplementationoftheARIADNEportal.

Portal Mock-ups

A series ormock-ups were created to start a discussion on the design and functionalities of theARIADNEportal,and toactasaguideline for implementation.Thesemockupspresentedadesignwiththreedifferentsections:Browse,SearchandServices.TheBrowsesectionwasintendedtoactas a startingpoint for exploring theARIADNE Info Space in different geo, time and subject basedvisualizations.TheSearchsectionpresentsaclassic informationretrieval interfaceforthedifferenttypesofresourcesintheARIADNECatalog,andisintendedtobecomplementedusingfacet-basedsearch(fig.18–fig.22).TheServicessectionprovidesaccesstothedifferentservicesdevelopedintheprojectaspartofWP13.

Figure18:Mockupoftheportal’shomepage.


34

Figure19:Asearchresultspagewithfacetedbrowsing/filtering.


35

Figure20:Ademoofcombiningsubject,placeandtime.


36

Figure21.Apageshowingmap-basedbrowsing.


37

Figure22:Apageshowingbrowsingthroughatimeline.

Development Framework

Although the ARIADNE portal is designed to be modular, the main framework selected for itsimplementation is Laravel (http://laravel.com/). Laravel is a PHP based framework designed tofacilitatethedevelopmentofmodel-view-controller(MVC)applications.Laravelisopensourceandalreadyprovidesagoodnumberofservices,ascanbeseenfromthetable3.

Initial Prototype

Aninitial implementationoftheARIADNEportalusingthementionedtechnologiesisshowninthefollowing screenshots (fig. 23 – fig. 25). They present the initial implementation,which has beenusedbypartners to inspect their contentandevolvedaccording to themockupspresented in theprevioussection.TheprototypewasbasedonaBootstrap3.0responsivethemeanddesign,whichallows use of the portal from virtually any kind of device, including workstations, tablets andsmartphones. The prototypewas designed using amodular architecturewhere eachmodulewasusedtoprovideaspecific functionality, includingsearching,browsingandviewingofrecords,mapandthematicviews,contentproviders’facets.


38

Table3.LaravelServices

- Authentication- Billing- Cache- Collections- Command Bus- Core Extension- Elixir- Encryption- Envoy- Errors & Logging- Events- Filesystem/Cloud Storage

- Hashing- Helpers- Localization- Mail- Package Development- Pagination- Queues- Session- Templates- Unit Testing- Validation

Figure23:Homescreenoftheportal.


39

Figure24:Listofexistingdatasets.

Figure25:Searchresultsforterm‘archeological’.


40

ARIADNE Portal Final implementation

The final implementation of the ARIADNE portal (http://portal.ariadne-infrastructure.eu/) wascarriedoutaccording to themockupsand,as canbe seen in fig.26, itpresents theuserwith thetoolstoperformasimplesearch(wherehecanselectoneoranyfield),anAATbasedautocompletesearch based on a subject, and a search based on space (using a map), time (using a timelinehistogram)andsubject(usingathematiccloud).

Figure26:Ariadneportalhomepage.

Afterenteringaquery,theuser ispresentedwithasearchresultspagewherethematching itemsare listed on the right (initialy sorted by score, allowing the user to also change the sortingparameter)anda listof facetson the left. The facets includeaquery refine textbox,aheatmapwhereallthesearchresultsaredepictedonthemap,atimelinehistogramandalistofboxeswithlistedattributesthatcorrespondtothoseofthematchingrecords.

The user can utilize any (or a combination) of these facets to refine his/her search, include thetimelineandheatmap.


41

Figure27:Ariadneportalsearchresults.

Figure28:Ariadneportalrecordview.

Source Code

ThesourcecodeoftheAriadnePortal(latestversion1.2.0)ishostedinGitHubrepository(fig.29),with open access. It can be found at https://github.com/dainst/ariadne-portal, together with theinstallationinstructions(fig.30).


42

Figure29:AriadnePortalsourcecode


43

Figure30:InstallationinstructionsGitHubincludesanissuetrackingsystem(https://github.com/dainst/ariadne-portal/issues)thatmanagesandmaintainslistsofreportedissues,tocreateandupdatefeaturesortoresolvebugs.Eachusercanreportanissue,whichisthenassignedtothepersonresponsibleforresolvingit(fig.31).


44

Figure31:Issuetrackingsystem


45

10 Support Portal

InordertoprovidesupporttotheARIADNEcommunity,asupportportalhasbeencreatedto:

• informtheusersandcontentprovidersoftheinfrastructureregardingtheACDMmodel,• offerthemappingguidelines• providehelpdesk• provideaticketingsupportsystem

Anindicativescreenshotispresentedinfig.32.

Figure32:Ariadnesupportportalhomepage


46

11 Summary of Improvements

The improvements that were made in the period from the initial implementation until the finalimplementationinclude:

• ChangesintheJSONschemainordertobetteraccomodatethedifferentfacets,nativeandprovidedsubjects.

• Improvementsonthesearch/browsequeriesinElasticSearch.• ImprovementsinthemappingoftheElasticSearchindextoimprovefacets.• ImprovementsinthePortalsearchandbrowsefunctionality.• Buttonineachresourceandgeneralcontactforminportalforreportingdataissues.• Creation of a date normalization service in order to better handle dates and improve the

facetedfunctionality.• Creationof a spatial normalization service that calculates themidpointof aboundingbox

andenrichestheresources,thusprovidingthemageographicalrepresentation.• RDFmicro-mappingandexportoftheCatalog.• Creation of new andmodification of existing import services that import data from excel

filesbecausemanypartners chose theirownexcel templates inwhich theyencoded theirdata.

• Modificationofimportservicethatimportsdatafromasinglexmlfiletofacilitateproviderswithcustomschemas.


47

12 Conclusions

This document presented the final implementation of the ARIADNE infrastructure project. Thetechnicaldesignchoices,aswellastheprimarycomponentsthatarerequiredtosetupandmakethe infrastructure operational, have been presented. The infrastructure has aggregated over 2millionrecords,cleaned,enrichedandpublishedthemtotheARIADNEportal(http://portal.ariadne-infrastructure.eu/),whichisavailableandusedglobally.

Thediversityofthecomponentsrequiredtoruntheinfrastructure,alongwiththelargenumberofservices, are indicative of the complexity of the infrastructure. During the testing of theinfrastructure,thevariousissueswereaddressedandimplemented.

ThisARIADNEportalservesasthepublicfaceoftheinfrastructure,andwaslaunchedduringtheCAA2016 conference onMarch 29th, 2016. Since then, and up to theNovember 17th, 2016 (almost 8months),ithasbeenvisitedbyover8,700users,withover45.000viewsfrom39countries(referredtocountrieswithmorethan10visits).


48

13 References

1. H.Wright,ARIADNED12.1UseRequirements,July2014,http://www.ariadne-

infrastructure.eu/Resources/D12.1-Use-Requirements.2. H.Hollander,M.Hoogerwerf,ARIADNED.13.1ServiceDesign,July2014,

http://www.ariadne-infrastructure.eu/Resources/D13.1-Service-Design.3. P.Ronzino,K.Fernie,C.Papatheodorou,H.Wright,J.Richards,ARIADNED3.2–Reporton

projectstandards,November2013,http://www.ariadne-infrastructure.eu/Resources/D3.2-Report-on-project-standards.

4. C.Papatheodorou,D.Gavrilis,K.Fernie,H.Wright,J.Richards,P.Ronzino,C.Meghini,ARIADNED3.1Initialreportonstandardsandontheprojectregistry,November2013.

5. N.Aloia,C.Papatheodorou,D.Gavrilis,F.Debole,C.Meghini,DescribingResearchData:ACaseStudyforArchaeology,13thInternationalConferenceonOntologies,DataBases,andApplicationsofSemantics(ODBASE2014),Amantea,Italy,October2014,inR.Meersmanetal.(Eds.):"OntheMovetoMeaningfulInternetSystems:OTM2014Conferences",LectureNotesinComputerScience(LNCS)No.8841:Springer-Verlag,2014,pp.768-775.

6. D.Gavrilis,S.Angelis,C.Dallas,ACuration-OrientedThematicAggregator,Proceedingsofthe17thInternationalConferenceonTheoryandPracticeofDigitalLibraries,TPDL2013,Valetta,Malta,September2013,pp.132-137

7. CIDOC-CRM(ConceptualReferenceModel),http://www.cidoc-crm.org8. M.A.Gonçalves,B.L.Moreira,E.A.Fox,L.T.Watson,"Whatisagooddigitallibrary?"-A

qualitymodelfordigitallibraries”,Inf.Process.Manage.43(5):1416-1437(2007)9. Geonames,http://www.geonames.org10. OAI-PMH-OpenArchivesInitiativeProtocolforMetadataHarvesting,

http://www.openarchives.org/pmh/11. OAI-ORE-OpenArchivesInitiative,ObjectReuseandExchangeSpecification,

http://www.openarchives.org/ore/12. METS-MetadataEncodingandTransmissionStandard,

http://www.loc.gov/standards/mets/13. DCAT-DataCatalogVocabulary,http://www.w3.org/TR/vocab-dcat/14. EuropeanUnionOpenDataPortal,http://open-data.europa.eu15. RabbitMQ,https://www.rabbitmq.com/16. GrayLog2,https://www.graylog.org/17. Elasticsearch,https://www.elastic.co/products/elasticsearch18. Lucene,http://lucene.apache.org/19. SolR,http://lucene.apache.org/solr/20. JSON,http://json.org/


49

Annex I – MORe API

Touse theMOReAPI, anAPI keymust be first registered through theDevelopers’ sectionwithinMORe.

Get Metadata Sources

GetalltheMetadataSourcesoftheMOREaggregatorusingsomefiltersifitisneeded.

Method GEThttp://more.locloud.eu:8080/MoRe_API/api/v1/metadataSources

ContentType application/son

CustomHeaders

X-API-Key

Request

Response [ {"id": 38, "name": "Metadata source 3 -KAMRA", "projectId": 3, "type": "OAI_PMH"}, {"id": 134, "name": "Test - duplications", "projectId": 3, "type": "OAI_PMH"} ]

Status 200:OK500:INTERNAL_SERVER_ERROR

Get a specific Metadata Source

GetaspedificMetadataSourceoftheMOREaggregatorgiventheidofthesource.

Method GEThttp://more.locloud.eu:8080/more_api/api/v1/metadataSources/{id}

ContentType

application/json

CustomHeaders

X-API-Key


50

Request

Response {"id":134,"format":"EDM","name":"Test_duplications","projectId":3,

"rootElement":"record","schema":"OAI_DC","set":"ARC3", "type":"OAI_PMH", "uri":"http://more.locloud.eu:8080/locloud/","webUrl":"" }


Create a Metadata Source

CreateaMetadataSourceintheMOREaggregator.

Method POSThttp://more.locloud.eu:8080/MoRe_API/api/v1/metadataSources

ContentType

application/json

CustomHeaders

X-API-Key

Request {"format:"EDM", "name":"Test API","projectId":3,"rootElement":"record","schema":"OAI_DC", "set":"ARC1", "type":"OAI_PMH", "uri":"http://api.test.eu","webUrl":""}

Response {"id":172,"format:"EDM","name":"Sample API MetadataSource","projectId":3,"rootElement":"record","schema":OAI_DC, "set":"ARC3","type":"OAI_PMH", "uri":"http://sample.locloud.eu:8080/locloud/","webUrl":""}

Status 201:CREATED500:INTERNAL_SERVER_ERROR


51

Get Packages Get all the packages of the MORE aggregator.

Method GEThttp://more.locloud.eu:8080/MoRe_API/api/v1/packages

ContentType application/json

CustomHeaders

X-API-Key

Request

Response [ {"id": 1144, "schema": "OAI_DC", "sourceId": 133, "status": "Rejected" }, {"id": 1146, "schema": "OAI_DC", "sourceId": 134, "status": "Published"}, {"id": 1149, "schema": "OAI_DC", "sourceId": 134, "status": "Enriched"} ]


Get a specific Package

GetdetailsforaspedificpackageoftheMOREaggregator,givenanid.

Method GEThttp://more.locloud.eu:8080/MoRe_API/api/v1/packages/{packageId}


CustomHeaders

X-MoRe-API-Key

Request

Response {"id": 1146,"itemsCount":11,"itemsEnriched":0,"itemsIngected":11,"itemsPublished":0,"itemsTransformed":0,"pubDatastreamId":0, "rightsId":0, "schema":"OAI_DC", "sourceId": 134,


52

"status": "Published" }


Get package datastreams Get datastreams for a specific package of the MORE aggregator, given an id.

Method GEThttp://more.locloud.eu:8080/MoRe_API/api/v1/packages/{packageId}/datasreams/{schema}


CustomHeaders

X-API-Key

Request

Response [ {"itemId":5322222, "uri":"http://more.locloud.eu/rest_get_item.php?package=1146&schema=OAI_DC&file=5322222.xml"}, {"itemId":5322223, "uri":"http://more.locloud.eu/rest_get_item.php?package=1146&schema=OAI_DC&file=5322223.xml"} ]


Get package tasks

GetthetaskscompletedforaspecificpackageoftheMOREaggregator,givenanid.

Method GEThttp://more.locloud.eu:8080/MoRe_API/api/v1/packages/{packageId}/tasks

ContentType

application/json


53

CustomHeaders

X-API-Key

Request

Response [ {"desc":"Package 498 is ready for ingestion.","timestamp":"2014-10-14 12:49:34.0"} ]


Get package notifications

GetthenotificationscreatedforaspecificpackageoftheMOREaggregator,givenanid.

Method GEThttp://more.locloud.eu:8080/MoRe_API/api/v1/packages/{packageId}/notifications

ContentType

application/json

CustomHeaders

X-API-Key

Request

Response [ {"desc":"A new package [498] has been created.","timestamp":"2014-10-14 12:49:14.0"}, {"desc":"Harvest complete for package 498","timestamp":"2014-10-14 12:49:24.0"}, {"desc":"Validation complete for package 498","timestamp":"2014-10-14 12:49:34.0"} ]



54

Harvest a Package

HarvestapackagetotheMOREaggregator.

Method POSThttp://more.locloud.eu:8080/MoRe_API/api/v1/metadataSources/{id}/harvest

ContentType

application/json

CustomHeaders

X-API-Key

Request

Response { "packageId":1299,"schema:"OAI_DC","sourceId":134}


Ingest a Package

IngestapackagetotheMOREaggregator.

Method POSThttp://more.locloud.eu:8080/MoRe_API/api/v1/packages/{packageId}/ingest

ContentType

application/json

CustomHeaders

X-API-Key

Request

Response {"msg":"Ingest request received.","packageId": 1299,"schema":"OAI_DC","sourceId":134 }



55

Transform a Package

TransformapackageoftheMOREaggregator.

Method POSThttp://more.locloud.eu:8080/MoRe_API/api/v1/packages/{packageId}/transform

ContentType

application/json

CustomHeaders

X-API-Key

Request {"mappingId":5,"rightsId":1}

Response {"msg":"Transform request received.","packageId":1299,"mappingId":5,"rightsId":1}


Enrich a Package

EnrichapackageoftheMOREaggregator.

Method POSThttp://more.locloud.eu:8080/MoRe_API/api/v1/packages/{packageId}/enrich

ContentType

application/json

CustomHeaders

X-API-Key

Request {"enrichPlanId":22}

Response {"msg":"Enrich request received.","packageId":1299,"enrichPlanId":22 }



56

Publish a Package

PublishapackageoftheMOREaggregator.

Method POSThttp://more.locloud.eu:8080/MoRe_API/api/v1/packages/{packageId}/publish

ContentType

application/json

CustomHeaders

X-API-Key

Request {"schema":"EDM"}

Response {"msg":"Publish request received.","packageId":1299,"schema":"EDM" }


Reject a Package

RejectapackageoftheMOREaggregator.

Method DELETEhttp://more.locloud.eu:8080/MoRe_API/api/v1/packages/{packageId}/reject

ContentType

application/json

CustomHeaders

X-API-Key

Request {"message": "Reject Message"}

Response {"msg":"Reject request received.","packageId":1299 }



57

Delete a Package

DeleteapackagefromtheMOREaggregator.

Method POSThttp://more.locloud.eu:8080/MoRe_API/api/v1/packages/{packageId}/delete


CustomHeaders

X-API-Key

Request

Response {"msg":"Delete request received.","packageId":1301 }


Withdraw a Package

WithdrawapackageoftheMOREaggregator.

Method POSThttp://more.locloud.eu:8080/MoRe_API/api/v1/packages/{packageId}/withdraw

ContentType

application/json

CustomHeaders

X-API-Key

Request key

Response {"msg":"Withdraw request received.","packageId":1299 }



58

Get mappings

Getthemappingsthatcorrespondtoaspesificprovider.

Method GEThttp://more.locloud.eu:8080/MoRe_API/api/v1/packages/{packageId}/mappings

ContentType

application/json

CustomHeaders

X-API-Key

Request

Response [ {"id":5,"title":"OAI_DC to EDM"} ]


Get rights

Gettherightsappearedinallpackagesofeachprovider.

Method GEThttp://more.locloud.eu:8080/MoRe_API/api/v1/rights


CustomHeaders

X-API-Key

Request

Response [ {"id":1,"name":"The Public Domain Mark (PDM)"}, {"id":2,"name":"Out of copyright - non commercial re-use (OOC-NC)"}, {"id":3,"name":"The Creative Commons CC0 1.0 Universal Public Domain Dedication (CC0)"}, {"id":4,"name":"Creative Commons - Attribution (BY)"},


59

{"id":5,"name":"Creative Commons - Attribution, ShareAlike (BY-SA)"}, {"id":6,"name":"Creative Commons - Attribution, No Derivatives (BY-ND)"}, {"id":7,"name":"Creative Commons - Attribution, Non-Commercial (BY-NC)"}, {"id":8,"name":"Creative Commons - Attribution, Non-Commercial, ShareAlike (BY-NC-SA)"}, {"id":9,"name":"Creative Commons - Attribution, Non-Commercial, No Derivatives (BY-NC-ND)"}, {"id":10,"name":"Free access - no re-use"}, {"id":11,"name":"Paid access - no re-use"}, {"id":12,"name":"Orphan work"}, ]


Get enrichment plans

Gettheenrichmentplansoftheprovider.

Method GEThttp://more.locloud.eu:8080/MoRe_API/api/v1/enrichmentPlans


CustomHeaders

X-API-Key

Request

Response [ {"id":1,"schema":"EDM", "title":"Standard EDM Enrichment Plan"} ]

Status 200 : OK500:INTERNAL_SERVER_ERROR


60

Get projects

Gettheprojectsthataproviderbelongsto.

Method GEThttp://more.locloud.eu:8080/MoRe_API/api/v1/projects


CustomHeaders

X-API-Key

Request

Response [ {"acronym":"LoCloud", "projectId":3} ]


Get schemas

Gettheavailableschemas.

Method GEThttp://more.locloud.eu:8080/MoRe_API/api/v1/schemas


CustomHeaders

X-API-Key

Request

Response [ {"schema":"CARARE", "schemaId":1,} {"schema":"CARARE20", "schemaId":2,} {"schema":"OAI_DC", "schemaId":3,} {"schema":"EDM", "schemaId":4,} {"schema":"eEDM", "schemaId":5,} {"schema":"ESE", "schemaId":6,} {"schema":"LIDO", "schemaId":7,}


61

]



62

Annex II – Registry API

The first thing the user who wants to use the API must do, is to find his own API key fromtheSettings.Thiswayhecanaccessonlyhisownrecordsandmanipulatethem.HewillneedthekeyfortheAPIcalls.

Get Data Resources

GetalltheDataResourcesoftheregistryusingsomefiltersifitisneeded.

Method GEThttp://ariadne-registry.dcu.gr:8080/ariadneWeb/api/v1/dataResources


Request

Parameter DataType Description

key String TheAPIkeyoftheuser

typeId Integer The type of the Data Resources(0:Collections1:Datasets2:Databases3:GIS)

Response


id Integer TheidofeachDataResource

name String ThenameofeachDataResource

Status 200–SUCCESS500–INTERNAL_SERVER_ERROR

Example


63

Request:{"key":"tUzQwU2T1RqBRuHC1ENJmtaopTMfWF6sHE7OrjRPzr98DiB","typeId":"1"}Response:[{"id":20,"name":"CulturaItalia - Archaeology"},{"id":33,"name":"Archaeologicalmaterials"}]

Get a Data Resource

GetallthepropertiesofaspecificDataResource,givenitsid.

Method GEThttp://ariadne-registry.dcu.gr:8080/ariadneWeb/api/v1/dataResources/{dataResourceId}

Contenttype application/json

Request



id Integer TheidofthespecificDataResource

Response


id Integer TheidofthespecificDataResource

name String ThenameofthespecificDataResource

properties List ThelistofthepropertiesforthespecificDataResource


Example


64

Request:{"key":"tUzQwU2P2T1RqBRHC1ENJmtaopTMfWF63sHE7OrjRPzr98DiBG"}Response:{"id":41, "name":"dFMROe","properties":{ "dct_issued":"1990s","dct:accessRights":"access only to 75565 records", ":scientificResponsible":"282","dcat:keyword":"Roman coins", "ARIADNEDistributionId":"44", "dct_identifier":"dat:41","dct:subject":"Roman coins", "dct:extent":"140000 records", "dct:audience":"researchers","ariadne_subject":"Artefacts", "dct_temporal":"116", "dct:creator":"281", "dct_language":"en","dct:description":"information on coins of the Preroman (Celtic) period and of Roman ImperialTimes found in Austria and in Romania", "dct:landingPage":"http://www.oeaw.ac.at/numismatik/fmroe/content/suche.de.php","dct_spatial":"108",":dbms":"MySQL"},"type":2}

Create a new Data Resource

CreateanewDataResourcewithall itsproperties.Allmandatoryelementsmusthaveavalue forthesuccessfulcreation.

Method POSThttp://ariadne-registry.dcu.gr:8080/ariadneWeb/api/v1/dataResources


Request



name String ThenameofthenewDataResource

properties List ThelistofthepropertiesforthenewDataResource

type Integer The type of the Data Resource(0:Collections1:Datasets2:Databases3:GIS)

Response Parameter DataType Description

id Integer TheidofthenewDataResource


65

name String ThenameofthenewDataResource

properties List ThelistofthepropertiesforthenewDataResource

type Integer The type of the Data Resource(0:Collections1:Datasets2:Databases3:GIS)

Status 201–SUCCESS400-BAD_REQUEST500–INTERNAL_SERVER_ERROR

Example

Request:{"name":"APIExampleDatabase","properties":{"ariadne_subject":"APIExampleDatabase", "dct_language":"en", ":dbms":"MySQL" }, "type":2 }Response:{ "id":50820, "name":"API Example Database", "properties":{ "dct_language":"en","ariadne_subject":"APIExampleDatabase",":dbms":"MySQL"},"type":2}

Get Distributions

GetalltheDistributionsoftheregistry.

Method GEThttp://ariadne-registry.dcu.gr:8080/ariadneWeb/api/v1/distributions


Request Parameter DataType Description




66

id Integer TheidofeachDistribution

name String ThenameofeachDistribution


Example

Request:{"key":"tUzQwU2P2T1RqBRHC1ENJmtaopTMfWF63sHE7OrjRPzr98DiBG"}Response:[{"id":6,"name":"dati.culturaitalia"},{"id":17,"name":" SITAR - Sistema InformativoTerritorialeArcheologicodiRoma"}]

Get a Distribution

GetallthethepropertiesofaspecificDistribution,givenitsid.

Method GEThttp://ariadne-registry.dcu.gr:8080/ariadneWeb/api/v1/distributions/{distributionId}

ContentType

application/json

Request



id Integer TheidofthespecificDistribution



67

id Integer TheidofthespecificDistribution

name String ThenameofthespecificDistribution

properties List ThelistofthepropertiesforthespecificDistribution


Example

Request:{"key":"tUzQwU2P2T1RqBRHC1ENJmtaopTMfWF63sHE7OrjRPzr98DiBG"}Response:{ "id":45, "name":"DANS Web Portal", "properties":{ "dct_modified":"2014-01-01","dcat:accessURL":"https://easy.dans.knaw.nl/", "dct_issued":"2014-01-01", "dct:description":"EASYonlinearchivingsystem",}}

Create a new Distribution

CreateanewDistributionwithallitsproperties.

Method POSThttp://ariadne-registry.dcu.gr:8080/ariadneWeb/api/v1/distributions


Request



name String Thenameofthenewdistribution

properties List Thelistofthepropertiesforthenewdistribution


68

Response


id Integer Theidofthenewdistribution

name String Thenameofthenewdistribution

properties List Thelistofthepropertiesforthenewdistribution


Example

Request:{"key":"tUzQwU2P2T1RqBRHC1ENJmtaopTMfWF63sHE7OrjRPzr98DiBG", "name":"DANSWeb Portal", "properties":{ "dct_modified":"2014-01-01","dcat:accessURL":"https://easy.dans.knaw.nl/", "dct_issued":"2014-01-01", "dct:description":"EASYonline archiving system", "dct:publisher":"5", "dct:licence":"8" } }Response:{ "id":45, "name":"DANS Web Portal", "properties":{ "dct_modified":"2014-01-01","dcat:accessURL":"https://easy.dans.knaw.nl/", "dct_issued":"2014-01-01", "dct:description":"EASYonlinearchivingsystem","dct:publisher":"5","dct:licence":"8"}}

Get Data Formats

GetalltheDataFormatsoftheregistry.

Method GEThttp://ariadne-registry.dcu.gr:8080/ariadneWeb/api/v1/dataFormats





69

Response


id Integer TheidofeachDataFormat

name String ThenameofeachDataFormat


Example

Request:{"key":"tUzQwU2P2T1RqBRHC1ENJmtaopTMfWF63sHE7OrjRPzr98DiBG"}Response:[{"id":149,"name":"aerialimage"},{"id":146,"name":"InformationSourceDataFormat"}]

Get a Data Format

GetallthethepropertiesofaspecificDataFormat,givenitsid.

Method GEThttp://ariadne-registry.dcu.gr:8080/ariadneWeb/api/v1/dataFormats/{dataFormatId}

ContentType

application/json




70

id Integer TheidofthespecificDataFormat

Response


id Integer TheidofthespecificDataFormat

name String ThenameofthespecificDataFormat

properties List ThelistofthepropertiesforthespecificDataFormat


Example

Request:{"key":"tUzQwU2P2T1RqBRHC1ENJmtaopTMfWF63sHE7OrjRPzr98DiBG"}Response:{"id":149,"name":"aerialimage","properties":{"dct:description":"JPGimage",}}

Create a new Data Format

CreateanewDataFormatwithallitsproperties.

Method POSThttp://ariadne-registry.dcu.gr:8080/ariadneWeb/api/v1/dataFormats





71

name String ThenameofthenewdataFormat

properties List ThelistofthepropertiesforthenewdataFormat

Response


id Integer TheidofthenewdataFormat

name String ThenameofthenewdataFormat

properties List ThelistofthepropertiesforthenewdataFormat


Example

Request:{"key":"tUzQwU2P2T1RqBRHC1ENJmtaopTMfWF63sHE7OrjRPzr98DiBG", "name":"aerialimage", "properties":{ "expressedIn":"xml", "dct:description":"JPG image", ":characterSet":"utf-8" }}Response:{ "id":149, "name":"aerial image", "properties":{ "expressedIn":"xml","dct:description":"JPGimage",":characterSet":"utf-8"}}

Get foaf:Agents

Getallthefoaf:Agentoftheregistry.

Method GEThttp://ariadne-registry.dcu.gr:8080/ariadneWeb/api/v1/foafAgents


72


Request



typeId Integer Thetypeofthefoaf:Agent(0:Agent1:Organization)

Response


id Integer Theidofeachfoaf:Agent

name String Thenameofeachfoaf:Agent


Example

Request:{"key":"tUzQwU2P2T1RqBRHC1ENJmtaopTMfWF63sHE7OrjRPzr98DiBG","typeId":"1"}Response:[{"id":270,"name":"EmerenzianaUsai"},{"id":259,"name":"MatejaBelak"}]

Get a foaf:Agent

GetallthethepropertiesofaspecificAgent,givenitsid.

Method GEThttp://ariadne-registry.dcu.gr:8080/ariadneWeb/api/v1/foafAgents/{foafAgentId}



73

Request



id Integer Theidofthespecificfoaf:Agent

Response


id Integer Theidofthespecificfoaf:Agent

name String Thenameofthespecificfoaf:Agent

properties List Thelistofthepropertiesforthespecificfoaf:Agent


Example

Request:{"key":"tUzQwU2P2T1RqBRHC1ENJmtaopTMfWF63sHE7OrjRPzr98DiBG"}Response:{ "id":110, "name":"Data Archiving and Networked Services (DANS)", "properties":{"foaf_mbox":"[email protected]",},"type":1}

Create a new foaf:Agent

CreateanewAgentwithallitsproperties.

Method POSThttp://ariadne-registry.dcu.gr:8080/ariadneWeb/api/v1/foafAgents



74

Request



name String Thenameofthenewfoaf:Agent

properties List Thelistofthepropertiesforthenewfoaf:Agent

type Integer The type of the foaf:Agent(0:Agent1:Organization)

Response


id Integer Theidofthenewfoaf:Agent

name String Thenameofthenewfoaf:Agent

properties List Thelistofthepropertiesforthenewfoaf:Agent

type Integer The type of the foaf:Agens(0:Agent1:Organization)


Example

Request:{"key":"tUzQwU2P2T1RqBRHC1ENJmtaopTMfWF63sHE7OrjRPzr98DiBG", "name":"DataArchiving and Networked Services (DANS)", "properties":{ "foaf_mbox":"[email protected]","foaf:phone":"888999214", "foaf:homepage":"www.example.com", "foaf:skypeID":"dans" },"type":1 }Response:{ "id":110, "name":"Data Archiving and Networked Services (DANS)", "properties":{"foaf_mbox":"[email protected]", "foaf:phone":"888999214","foaf:homepage":"www.example.com","foaf:skypeID":"dans"},"type":1}


75

Annex III –Thesauri Available through MORe

Thistablepresentssome(ofthemostrelevant)thesaurithatMOReisabletoaccessthroughtheTemaTresAPI.Thesethesauriareaccessibletotheenduserthroughthesubjectcollectionsenrichmentmicro-services.

Author Name Subject Language

UniversityofCalifornia,SantaBarbara

AlexandriaDigitalLibraryFeatureTypeThesaurus

Science&Technology:places,geographica

English

RoyalCommissionontheAncientandHistoricalMonumentsofScotland(RCAHMS)

ArcheologicalObjectsThesaurusScotland

Objectsmadebyhumanactivity

English

EnglishHeritage ArcheologicalSciencesThesaurus

Techniques,recoverymethodsandmaterialsassociatedwitharchaeologicalsciences

English

EnglishHeritage BuildingMaterialsThesaurus

Mainconstructionalmaterialtypes(eg.thewalls)forindexingofmonuments

English

EnglishHeritage ComponentsThesaurus Terminologycoveringdivisionsandstructuralelementsofabuildingormonument

English

AmericanFolkloreSociety

EthnographicThesaurus Socialsciences:ethnographic,folklore,unitedstates,ethnomusicology

English

EnglishHeritage EventTypeThesaurus Terminologyusedforrecordingarchaeologicalandarchitecturalinvestigative,datacollectionexercises;fromintrusiveinterventionstonondamagingsurveys

English

EnglishHeritage EvidenceThesaurus Terminologycoveringtheexistingphysicalremainsofa

English


76

Author Name Subject Languagemonument,orthemeansbywhichamonumenthasbeenidentifiedwherenophysicalremainsexist

EnglishHeritage FISHArcheologicalObjectsThesaurus

RecordingofarchaeologicalobjectsinBritainandIrelandcoveringallhistoricalperiods

English

GEMETThesaurus GeneralMultilingualEnvironmentalThesaurusGEMET

Environment,policies Dutch,English,French,German,Italian,Portuguese,Spanish

FederationInternationaledesArchivesduFilm(FIAF)

GeneralSubjectheadingsforFilmArchives

Arts&Humanities:films,filmsarchives,cinema

English

TheDiscoveryProgramme

IrishMonuments Irishmonuments English

TheDiscoveryProgramme

IrishPeriods Irishperiods English


MaritimeCraftThesaurusScotland

Typesofcraftthatsurviveaswrecks,oraredocumentedaslosses,inScottishmaritimewaters

English

EnglishHeritage MaritimeCraftTypeThesaurus

CrafttypeswhichsurviveaswrecksinEnglishHeritage’smaritimerecord

English

EnglishHeritageandRoyalCommissionontheHistoricalMonumentsofEngland

MDAArchaeologicalObjectsThesaurus

Socialsciences:archaeologicalobjects;museums;vocabulary

English

RoyalCommissionontheAncientandHistoricalMonumentsofWales(RCAHMW)

MonumentThesaurusWales

ClassificationofmonumenttypesinWalesbyfunction

English


77

Author Name Subject Language


MonumentTypeThesaurus

MonumenttypesrelatingtothearchaeologicalandbuiltheritageofScotland

English,ScottishGaelicforsometerms

EnglishHeritage PeriodThesaurus EnglishHeritagePeriodsList English

RoyalCommissionontheAncientandHistoricalMonumentsofWales(RCAHMW)

PeriodThesaurusWales AlistofperiodsforuseinWales

English

BibliographicStandardsCommitteeoftheRareBooksandManuscriptsSection(ACRL/ALA)

RelatorTermsforUseinRareBookandSpecialCollectionsCataloguing

Arts&Humanities:RareBook;SpecialCollection

English

UniversidaddeLeón

TesaurodeCienciasdelaDocumentación

Socialsciences:librarysciences,informationsciences,documentation,librarianship

Spanish

LibraryofCongress.PrintsandPhotographsDivision

ThesaurusforGraphicMaterials1:SubjectTerms

Arts&Humanities:graphicmaterials,subjectterms

English

LibraryofCongress.PrintsandPhotographsDivision

ThesaurusforGraphicMaterials2:GenreandPhysicalCharacteristicTerms

Arts&Humanities:graphicmaterials

English

MinisteroperiBenieleAttivitàCulturali

ThesaurusPICO4.1Arts&Humanities:italianculture,heritage

English,Italien

UKAT UKArchivalThesaurus(UKAT)

Generalreference:archives,ukat,unitedkingdom

English

UNESCO UNESCOthesaurus Generalreference:education,culture,socialandhumansciences,informationandcommunication,politics,law,economicscience

English,French,Spanish


78

ariadne: final implementation report

Data & Analytics