![Page 1: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/1.jpg)
1
SAMT’08
Semantic-driven multimedia retrieval with the MPEG Query Format
Ruben Tous and Jaime Delgado
Distributed Multimedia Applications Group (DMAG)Universitat Politècnica de Catalunya (UPC)
Dept. Computer ArchitectureBarcelona (Spain)
![Page 2: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/2.jpg)
Presentation outline
The MPEG Query Format (MPQF) MPQF evaluation model How to extend MPQF to allow SPARQL-like queries MPQF and metadata interoperability
2
![Page 3: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/3.jpg)
MPEG Query Format (MPQF)
![Page 4: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/4.jpg)
4
MPEG Query Format (MPQF): Introduction
ISO/IEC 15938-12 Query Format (MPQF) Recently reached the International Standard level
MPQF: “Provides a standardized interface for multimedia content retrieval systems (e.g.: MPEG-7 databases)”
Allows combining Data Retrieval like conditions (XPath, XQuery) with Information Retrieval like conditions (Query-by-Example, etc.)
Is an XML application: One XML schema describing messages: Requester -> responder = Input Query Format Responder -> requester = Output Query Format Management messages (service properties, etc.)
Part 12 of ISO/IEC 15938 (MPEG-7), but METADATA-NEUTRAL
![Page 5: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/5.jpg)
5
Service Provider
Requester
MPQF
MPQF
MPQF
MPQF Service N
Service 2
Service 1
Responder
Input Query FormatOutput Query Format
Query Management InputQuery Management Output
Client
Query Splitter
Resultlist Aggregator
![Page 6: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/6.jpg)
MPEG Query Format (MPQF)
Schema overview of the uppermost elements of MPQF
6
MpegQuery
Query
Management
FetchResult
Output
Input
Output
Input
![Page 7: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/7.jpg)
MPEG Query Format (MPQF)
Input Query Format (IQF)
7
QFDeclaration
ServiceSelection
Input
OutputDescription
QueryCondition
Path
TargetMediaType
Join
Condition
![Page 8: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/8.jpg)
8
MPEG Query Format (MPQF)
Condition types
IR-like criteria
QueryByFreeText
QueryByDescription
QueryByMedia
TemporalQuery
SpatialQuery
Embedded XQuery expressions(metadata-neutral)
XML query algebra(metadata-neutral)
DR-like criteria
QueryByROI
DR-like criteriaBoolean connectors
![Page 9: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/9.jpg)
MPEG Query Format (MPQF)
Example input query condition tree (graphical view)
9
AND
<InlineMedia type=“image/jpeg"> <MediaData64>R0lGODlhDwAPAKECAAAAzMzM///// wAAACwAAAAADwAPAAACIISPeQHsrZ5ModrLlN 48CXF8m2iQ3YmmKqVlRtW4MLwWACH+H09 …. </MediaData64></InlineMedia> 1000//FileSize
<QueryByMedia
![Page 10: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/10.jpg)
10
Input Query Format (Example)
![Page 11: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/11.jpg)
MPEG Query Format (MPQF)
Output Query Format (OQF)
11
GlobalComment
ResultItemOutput
SystemMessage
Comment
TextResult
Thumbnail
MediaResource
Description
AggregationResult
![Page 12: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/12.jpg)
12
Output Query Format (Example)
![Page 13: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/13.jpg)
MPQF evaluation model
![Page 14: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/14.jpg)
14
<metadata>
image2
image1
<metadata>
<metadata>image3
Multimedia Database: Simplistic view
![Page 15: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/15.jpg)
15
<metadata>
Metadata DB
image2
i3.1 i3.2
image1
Content DB
<metadata>
<metadata>
Multimedia Database: Complex view (the one in the Query Format)
image4 <metadata>
<metadata>image1
![Page 16: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/16.jpg)
16
<metadata> =
XML
The Query Format assumes XML metadata
![Page 17: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/17.jpg)
MPQF in depth
The EvaluationPath element
17
<MpegQuery> <Query> <Input> <QueryCondition> <EvaluationPath>//VideoSegment</EvaluationPath> <Condition xsi:type="QueryByFreeText"> <FreeText>Lausanne</FreeText> </Condition> </QueryCondition> </Input> </Query></MpegQuery>
![Page 18: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/18.jpg)
18
Metadata DB
image2
i3.1 i3.2
image1
Content DB
Multimedia Database: Complex view (the one in the Query Format)
image1
Image/region locators//IDs
![Page 19: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/19.jpg)
image2
i.1 i3.2
image1
Content DB Metadata DB
Image region locators
Independentmetadata
Image Database: Practical view (ONE BIG XML TREE)
![Page 20: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/20.jpg)
MPQF in depth
Two evaluation styles in MPQF: Boolean and fuzzy-logic
20
//FileSize < 1000
1= TRUE
1= TRUE1= TRUE
0.4
0.56
0.7
QueryByMedia
inputsequence
outputsequence
![Page 21: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/21.jpg)
How to extend MPQF to allow SPARQL-like queries
![Page 22: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/22.jpg)
How to extend MPQF to allow SPARQL-like queries
Knowledge base (KB) = Tbox (terminological component) +Abox (assertion component)
Some relevant initiatives are choosing the RDF language for modeling metadata metadata
Typical Semantic Web based multimedia KB (e.g. PhotoRDF): Tbox = RDFS/OWL ontologies Abox = RDF statements about the media objects
22
ABox
Knowledge DB
ps
ops
ps o
p oop
sp
s o
ops
ps o
TBox (ontologies)
o
s
![Page 23: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/23.jpg)
How to extend MPQF to allow SPARQL-like queries
SPARQL. W3C Recommendation on 15th January 2008 An SPARQL query consists of a basic graph pattern, expressed as a
list of triple patterns
23
PREFIX dc: <http://purl.org/dc/elements/1.1/>PREFIX ns: <http://example.org/ns#>SELECT ?title ?priceWHERE { ?x ns:price ?price .FILTER (?price < 30.5)?x dc:title ?title . }
ps
ops
ps o
o
ops
ps o
![Page 24: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/24.jpg)
How to extend MPQF to allow SPARQL-like queries
Proposal 1/3: Extend the MPQF data model
24
video2
v3.1 v3.2
video1
Content DB Metadata DB
ABox
Knowledge DB
ps
ops
ps o
ps o
ops
ps o
ops
ps o
TBox (ontologies)
o
![Page 25: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/25.jpg)
How to extend MPQF to allow SPARQL-like queries
Proposal 2/3: Include a new query type: QueryBySPARQL
25
<MpegQuery mpqfID="someID"> <Query> <Input> <QueryCondition> <Condition xsi:type="QueryBySPARQL"> <SPARQL> <![CDATA[ PREFIX dc: <http:///> ASK { ?resource dc:title "Barcelona . } ]]> </SPARQL> </Condition> </QueryCondition> </Input> </Query></MpegQuery>
![Page 26: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/26.jpg)
How to extend MPQF to allow SPARQL-like queries
Visual representation of the execution of the new QueryBySPARQL query type
26
video2
v3.1 v3.2
video1
Content DB Metadata DB Knowledge DB
ps
ops
ps o
ps o
o
p
sp
p
o
ps o
matching RDF triples
Evaluation-Item (EI)
![Page 27: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/27.jpg)
AND
PREFIX dc: <http://purl.org/dc/elements/1.1/> ASK { ?resource dc:title "Barcelona . }
1000//FileSize
<QueryBySPARQL
<InlineMedia type=“image/jpeg"> <MediaData64>R0lGODlhDwAPAKECAAAAzMzM///// wAAACwAAAAADwAPAAACIISPeQHsrZ5ModrLlN 48CXF8m2iQ3YmmKqVlRtW4MLwWACH+H09 …. </MediaData64></InlineMedia>
QueryByMedia
![Page 28: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/28.jpg)
But, what about metadata interoperability with MPQF?
![Page 29: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/29.jpg)
MPQF and metadata interoperability
Currently MPQF is metadata-neutral Queries refer to paths related to different metadata formats
Several metadata formats exist: Dublin Core, EXIF, ID3, MPEG-7, etc.
Semantic mappings can help circumventing the problem The mappings are (supposed to be) easiest to specify with
ontologies (with our proposal) we can use RDF and SPARQL in MPQF but… Can I use “ontologies” with MPQF?
(currently) no, at least not in a “standard” way
29
![Page 30: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/30.jpg)
MPQF and metadata interoperability
Proposal 3/3: Extend MPQF: Allow registering RDFS/OWL ontologies Be aware of the registered ontologies when querying XML Be aware of the registered ontologies when querying RDF Be aware of the registered ontologies when querying free text?
30
![Page 31: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/31.jpg)
MPQF and metadata interoperability
Two choices to achieve interoperability with N formats: Specify NxN mappings Specify one reference metadata format and N mappings (better!)
Which kind of reference metadata format? An XML Schema (e.g. XMP) But mappings are simpler if the reference metadata model is
also an ontology This issue is now being discussed at JPEG (JPSearch initiative) OPEN ISSUE
31
![Page 32: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/32.jpg)
Conclusions
The MPEG Query Format is finished! But now is time for amendments and corrigenda
The implementation race has begun MPQF lacks the ability of handling RDF and SPARQL
Proposal 1/3: Extend the logical model Proposal 2/3: New query type: QueryBySPARQL Proposal 3/3: Allow registering ontologies
A relevant piece is still missing: A pivot metadata format
32
![Page 33: 1 SAMT’08 Semantic-driven multimedia retrieval with the MPEG Query Format Ruben Tous and Jaime Delgado Distributed Multimedia Applications Group (DMAG)](https://reader033.vdocuments.us/reader033/viewer/2022051416/56649e765503460f94b77335/html5/thumbnails/33.jpg)
Acknowledgments
33
Thank you