geometric and semantic matching for cultural...

Copyright University of Southampton IT Innovation Centre 2017 1 www.gravitate-project.eu

Geometric and Semantic Matching for Cultural

Heritage Artefacts

Paul Walland

IT Innovation

ResearchSpace Symposium

28-07-2017

Initial Ambition of GRAVITATE Project

Re-Assembly

Re-Unification

Re-Association

------------------------

Similarity Search

British Museum, London Cyprus Museum, Cyprus

Ashmolean Museum, Oxford Fitzwilliam Museum, Cambridge

Re-Unification

Head in the British Museum

Statue in the Cyprus Museum

Re-unify 3D models of the statue

3D print copies for re-unification

Re-unify statue in a virtual museum

Re-Association

Artefact with semantic description Material, decoration, size, shape, period, style, glaze, …

Data in another museum catalogue

Re-Association

Find similarities leading to new insights about past cultures

Data on Artifacts is in many different forms:

Physical Descriptors: Image (2D and 3D) Geometrical Material Pattern Colour … etc

Abstract (non-physical, semantic) Age Condition Style Location Part (e.g. foot or ear) Specialised curatorial terms … etc

And in many different places!

Fragments of Terracotta Statues from Salamis

Problem Statement British Museum, London Cyprus Museum, Cyprus

Ashmolean Museum, Oxford Fitzwilliam Museum, Cambridge

Problem Statement

Jigsaws

most pieces missing

all the edges worn off

many puzzles jumbled together

pieces spread across many countries

Problem Statement

Front Back

Cracked

Eroded

Problem Statement Re-Assembly

You have some (eroded) pieces.

The computer fits them together.

This is hard in 2 dimensions, never mind in 3D!

Localised geometry

3D descriptors

Localised detail

Colour descriptors

We need to Gather all the Knowledge

Catalogue data with text

descriptions

Archaeological papers

Excavation notes

Chemical analysis

X-ray fluorescence

3D scanning models

descriptions

Excavation notes

Chemical analysis

X-ray fluorescence

3D scanning models

descriptions

Excavation notes

The British Museum has 2.5M objects described in CIDOC CRM Catalogue data from Ashmolean, Fitzwilliam and Cyprus museums is now mapped to the same data model

Object hasNote “…lots of free-form text added by the curator”

Using Natural Language Processing to extract and encode meaning from this text: • References to papers, to catalogue entries • Parts and features • Conservation condition • Measurements • … CIDOC CRM / CRMarchaeo / etc

Chemical analysis

X-ray fluorescence

3D scanning models

Volume, area thickness,

curvature, …

Colour, texture, distance from convex hull, …

Feature detection

Faceting: front / back

/ fracture

Part annotation

CRMdig CH Artefact Partonomy

All fragments (211) are scanned, in a

variety of resolutions and with a variety of

scanners

What is our raw data?

Semantic data

From catalogues

From curators

Measurements of the fragments

3D scanning 3D models

Chemical analysis, pigment composition

X-ray fluorescence

…etc

Fundamental Geometric Properties and Descriptors

Library of tools for the analysis of the geometric and colorimetric properties of a fragment’s shape Curvature-based

Distance-based

Volume, area, thickness

Library of descriptors and signatures to evaluate similarity geometry

textures

Mean Shape index

Distance from convex hull

Average geodesic distance

Faceting Faceting partitions a fragment into

distinct facets by sharpness and roughness

Each facet will be classified into: Skin (original) boundary surface of

the unbroken object

Fracture created when the object broke

Re-Assembly modules use the facets differently: Skin facets local continuity

in color and shape

Fracture facets complementarity of abraded local shape

(SotA faceting from predecessor PRESIOUS)

NLP Example Head from Toumba, Ashmolean Museum, Oxford, Inv. no. 1909.837=1891.468 (Tubbs and Munro 1891, 149, fig. 7; SCE IV:2, pl. IX bottom; Hermary 1991, pl. XXXIX:a). The lower part of the beard is restored. Restored height of the head: 39 cm. Upper part of (conical?) helmet missing; upturned cheek-pieces; preserved part of helmet is covered with stamped circles. The face is rectangular; almond-shaped eyes, ridged eyelids, broad 'feathered' eyebrows. The beard is neatly separated from the rest of the face by a 'rope' line as on Cat. no. 73. The restored lower part of the beard must originally have been square with rounded corners, like that of Cat. no. 73. It is grooved vertically with each groove terminating in a curl (cf. a beard fragment from Samos, no. T324, Schmidt 1968, pl. 60). A fringe of curly hair appears on the forehead below the lower edge of the helmet and at the temples, as on Cat. no. 73. Trapezoidal moustache with grooved strokes. The ears are rendered in the same way as on Cat. no. 73. Black paint on beard, hair, moustache, eyebrows and irises; red paint on lips. This head was assigned by Gjerstad (SCE IV: 2, 108) to the Western Neo-Cypriote style. It has several stylistic similarities with Tamassos heads Cat. nos 68-69, though the expression of Cat. no. 74 is more smiling (Karageorghis 1993, 32, Cat. no. 74).

Ashmolean reference AN1909.837

The discourse styles for Salamis artefacts in BM, Ashmolean, Fitzwilliam and Cyprus museums are similar, but different

We attempt to automate annotation of the artefact semantics from such sources

ResearchSpace

• British Museum’s platform impementing CIDOC CRM as framework

• Components: Search, Clipboard, Data Annotation, Data Argumentation, Image Annotation/ Argumentation, Workflow, User Defined Pages/ Publication • Shows faceted relationships

• Combines results from multiple datasets

• Supports contextual search

Similarity Search

Similarity matching of artefacts Start with search criteria or set of example objects

Geometric descriptors » size, part, colour, texture, edge/surface » Geometric matching algorithms

Semantic metadata » type, style, location, size, part, colour, texture, damage, hypothesis/conjecture » Graph matching

Fusion of match results Aggregation of the result space + Uncertainty analysis

Not a black box approach

Iterative expansion / refinement of match results with expert feedback

Demonstration of GRAVITATE Platform

Desktop app

GRAVITATE-PROJECT.EU

Thank you for your attention!

geometric and semantic matching for cultural...

Documents

researchspace platform in use

comparison methodologies. evaluating the matching...

toxic compounds. in cycads - researchspace home

fusing matching and biometric similarity measures for face...

myplan - similarity metrics for matching lifelong learner...

view/open - researchspace

1 best-buddies similarity - robust template matching using...

unravelling in two-sided matching markets and similarity ......

similarity, typicality, and category-level matching of...

researchspace overview

template matching with deformable diversity...

electrochemical characterization of mixed self -...

researchspace collaborative features

flavel cover sheets - researchspace - researchspace

graph similarity and matching laura zager

topology matching for fully automatic similarity matching of...

graph matching networks for learning the similarity of...

sansp 088.pdf - researchspace - csir

convolutionalset matching for graph similarity ·...

fallacies in the interpretation of text- matching software...