direct mapping of x3d scenes to mpeg-7 descriptions
DESCRIPTION
Direct Mapping of X3D Scenes to MPEG-7 Descriptions. Markos Zampoglou , Patti Spala , Konstantinos Kontakis , Athanasios G . Malamos , J Andrew Ware. What is our motivation. - PowerPoint PPT PresentationTRANSCRIPT
TEI of Crete Multimedia Lab
Direct Mapping of X3D Scenes to MPEG-7 Descriptions
Markos Zampoglou, Patti Spala, Konstantinos Kontakis, Athanasios G. Malamos, J Andrew Ware
TEI of Crete Multimedia Lab
What is our motivationi-Promotion: we work on a platform for VR web advertising. The goal of this project is to join cloud stored X3D objects to more complicated scenes and to publish them in the way that we used to publish GIF Banners!
The first questions we had to answer was how can we search and retrieve the resources according to technical as well as functional and contextual characteristics. The resources might be the 3D geometry as well as Routes, Textures (images or videos), audios and Sounds embedded on the 3D.
What is missing is an efficient annotation of X3D in MPEG7.
MPEG-7 defines several descriptors to facilitate the description of 3D entities involving geometries, textures, animations and content metadata. However, the majority of 3D descriptors are centered on vertices representation of 3D objects, lacking in the ability to provide optimized semantic descriptions based on XML encoded 3D objects. However the existence of the 3D descriptors helped us to extend them to fit in our case allowing us to still use the older and the newer descriptors at the same time.
I-Promotion is a project granted by the European Union and the Hellenic General Secretary of Research and Technology under the ”COOPERATION 2009 /09SYN-72-956” Framework.
TEI of Crete Multimedia Lab
The i-Promotion platform
HTTP
SOAPCROSS3D Service Protocols
HTTPSOAP
I Promotion Service Protocols (MPEG7, MPEG21, etc)
Virtual machinesCloud
I-promotion coreCROSS 3D Hosts
3D Model Repositories Rendering enginesRendering engines
Device Interfaces
Mobile WEB Digital TV
Client Devices
Mobile WEB
3D Model Repositories Rendering enginesRendering engines
Device Interfaces
Mobile WEB Digital TV
Device Interfaces
WEB/HTML5
Digital Media
Client Devices
HTTPHTTP
Mobile WEB Digital Media
RTSPWeb-socketsHTTPHTTP/SOAP
WEB/HTML5
3D worlds Authoring
tool
3D banners Promotion
Manager
Media framework
TEI of Crete Multimedia Lab
The i-Promotion platform architecture
• A cloud-based repository for 3D scenes• A remote rendering server
– supplementing a potential local rendering approach• Service-oriented content distribution• HTML5-based clients
– PCs, mobile devices, wall-to-wall projections…• Supplementary modules for content adaptation
– Device Interfaces, Media Framework• Additional tools
– 3D Worlds authoring tool, 3D banners promotion manager
TEI of Crete Multimedia Lab
The i-Promotion platform:Standards conformance
• MPEG-7: Content description for 3D scenes– Content-based search
• MPEG-21: Multimedia Item declaration & adaptation– Scene organization (thumbnails, descriptors, rights)– Content adaptation for extreme device flexibility
• ???? : Scene structure composition• X3D: 3D scene format
– Offers a choice between server-based rendering or local, HTML5 compatible rendering (via X3DOM)
TEI of Crete Multimedia Lab
MPEG-7 for 3D scenes
• MPEG-7 – Part 3: Visual includes descriptors for 3D model shape.
• Efforts towards full 3D scene description (shape, color, structure, light etc.)– 3DSEAM: X3D scene structure and object location (Bilasco et al.
2006)– Multimedia Interaction Model (Chmielewski 2012)– MPEG-7 Shape and semantics for MPEG-4 3D scenes
(Bonhomme et al. 2007)– Various RDF and OWL Semantic annotations
TEI of Crete Multimedia Lab
What MPEG 7 does support in 3DGeometry DescriptorsMPEG-7 Part 3, Visual, defines two shape descriptors for 3D objects: the Shape3D Spectrum descriptor and the MultipleView Descriptor Container.
Motion DescriptorsMPEG-7 Part 3, Visual, defines three descriptors, The Camera Motion descriptor, The Motion Trajectory descriptor and the Motion Activity descriptor.
Texture and Color DescriptorsIn order to describe textures, MPEG-7 Part 3, Visual, provides three texture descriptors: the TextureBrowsing Descriptor, the HomogeneousTexture Descriptor and EgdeHistogram Descriptor. The color description tools presented in, MPEG-7 Part 3, Visual, include four distinct color descriptors to annotate different color features, the DominantColor Descriptor, the ScalableColor Descriptor, and the ColorLayout and ColorStructure Descriptors. Meanwhile, two additional supporting descriptors are included, the ColorSpace and ColorQuantization Descriptors, which are used in conjunction with the DominantColor Descriptor, providing supplementary details to representative colors.
TEI of Crete Multimedia Lab
MPEG-7 Extension DescriptorsMPEG-7 Part Name Usage
MPEG-7Part 3 – Visual
BoundingBox3D Position and size of a 3D object in the scene
Geometry3D Types of primitive or complex geometries and their curvature.
Metadata3D Additional metadata information (such as MatadataFloat, MetadataInteger, MetadataSet).
Interactivity3D Interactions with other objects in the scene or with the end user, (based on X3D ROUTE elements).
MotionTrajectory Animation characteristics of a moving object: type of animation, through the Interpolator or Follower nodes bound to that object, along with the motion path keypoint values.
Viewpoint3D Viewpoint nodes’ position, orientation, animation, and coordinates.
Lighting3D The type of X3D Light nodes used within an X3D Scene, their illumination specifications and their scope (global or not).
MPEG-7Part 5 – MDS
Profile3D The X3D profile.
Script3DIf the file contains Script nodes, specifies the script class location (internal or external) and the scripting language used.
TEI of Crete Multimedia Lab
Automatic extraction of MPEG-7 descriptions
• X3D and MPEG-7 are both XML-based standards• We use XSLT for the direct extraction of (extended-)
MPEG-7 compliant descriptions from X3D scenes– Both scenes are kept: the original X3D scene is retained for
rendering and display, while the new MPEG-7 description is used for search and retrieval
TEI of Crete Multimedia Lab
XSLT Mapping
X3D Scene Document
(in XML format)
XSLT Processor
XSLT Style Sheet
MPEG-7 XML Document
X3D Scene and Model Annotation
Repository
3D Media Graphics Repository
TEI of Crete Multimedia Lab
Extensions in MPEG-7 Part 3
Visual
TEI of Crete Multimedia Lab
BoundingBox3D Elements
Name DefinitionBoundingBox3DType Descriptor
Volumetric coordinates as obtained from the X3D BoundingBox element
BoundingBox3DSizeWidth, Height and Length coordinates of the BoundingBox, passed into the BoxWidth, BoxHeight and BoxDepth attributes. Default values are (-1, -1, -1) as in the X3D bounding box specification.
BoundingBox3DCenter
Center coordinates of the BoundingBox. Each of the three values contained in bboxCenter is passed into the BoxCenterW, BoxCenterH and BoxCenterD attributes. Default values are (0, 0, 0) as in the X3D bounding box specification.
TEI of Crete Multimedia Lab
BoundingBox3D
TEI of Crete Multimedia Lab
BoundingBox3DType FieldsMPEG-7 Description Fields Value Type Input from X3D Fields
BoundingBox3DType Descriptor - Created if an X3D Grouping Node contains valid
bboxCenter and bboxSize attributes
BoundingBox3DSize - Created if an X3D Grouping Node contains a valid bboxSize attribute
BoxWidth Float Position[1] of the value of the X3D bboxSize attribute String
BoxHeight Float Position[2] of the value of the X3D bboxSize attribute String
BoxDepth Float Position[3] of the value of the X3D bboxSize attribute String
BoundingBox3DCenter - Created if an X3D Grouping Node contains a valid bboxCenter attribute
BoxCenterW Float Position[1] of the value of the X3D bboxCenter attribute String
BoxCenterH Float Position[2] of the value of the X3D bboxCenter attribute String
BoxCenterD Float Position[3] of the value of the X3D bboxCenter attribute String
TEI of Crete Multimedia Lab
BoundingBox Mapping
TEI of Crete Multimedia Lab
Geometry3D Elements
Name Definition
Geometry3DType Describes the types of primitive or complex shapes contained in an X3D file.
Geometry3D Describes one shape from the X3D file.
Metadata3D Specifies the additional metadata values, such as MetadataDouble, MetadataFloat, and MetadataSet.
TEI of Crete Multimedia Lab
Geometry3D
TEI of Crete Multimedia Lab
Geometry3DType Fields
MPEG-7 Description Fields Value Type Input from X3D Fields
Geometry3D Descriptor - Created if an X3D Shape element
contains an X3D Geometry nodecreaseAngle Float Value of the X3D creaseAngle
attributeconvex Boolean Value of the X3D convex attributeObjectType String Value of the X3D Geometry node type
(node name)DEF String Value of the X3D DEF attributeMetadata3D - Created if an X3DMetadataObject
exists within the X3D Geometry node.
TEI of Crete Multimedia Lab
Geometry3D Mapping
TEI of Crete Multimedia Lab
Metadata3DType ElementsName Definition
Metadata3DType Additional metadata values contained within an X3D node, such as MatadataFloat, MetadataInteger, MetadataSet.
name The name of the metadata type being described, from the Name attribute of the X3DMetadataObject.
type Specifies the type of metadata being described, such as MatadataFloat, MetadataInteger, MetadataSet.
referenceThe value of the reference field from the X3DMetadataObject. If the field is not provided or is empty, the meaning of the name field is considered implicit to the characters in the string.
valueAll metadata values as derived from the corresponding X3DMetadataObject. In the case of a MetadataSet, the value element contains all internal elements and values.
Ref The XPath location from the X3D XML graph of the metadata element being described.
TEI of Crete Multimedia Lab
Metadata
TEI of Crete Multimedia Lab
MetadataSet
TEI of Crete Multimedia Lab
Metadata3DType Fields
MPEG-7 Description Fields Value Type Input from X3D Fields
Metadata3D Descriptor -
If an X3D Metadata element such as MatadataFloat, MetadataInteger, MetadataSet, exists.
name String Value of the X3DMetadataObject name attributetype String Value of the X3DMetadataObject type attributereference String Value of the X3DMetadataObject reference
attribute
valueInteger/ Float/ String/ Double/
nested Metadata3D element
Value of the X3DMetadataObject attributes’ value. Depends on the X3DMetadataObject type (Float, Integer, String). In the case of X3D MetadataSet, the value attribute holds the X3D MetadataSet children
Ref XPath XPath route to the specific X3DMetadataObject.
TEI of Crete Multimedia Lab
MetadataSet Mapping
TEI of Crete Multimedia Lab
MotionTrajectory ElementsName DefinitionMotionTrajectoryType
Specifies the motion trajectory of a moving object, as defined in the MPEG-7 ISO Standard. A MotionTrajectory Descriptor is formed for each X3D Interpolator node.
CoordRef Allows the definition of a CoordRef or CoorfDef element, depending on the coordinate system. The CoordRef element is used to reference a two-dimensional coordinate system. It is therefore not used in the case of X3D descriptions.
CoordDef If the attribute value is set to meter, then the trajectory is considered three-dimensional and the coordinate system is local to the specified trajectory
Params the spatio-temporal key-points and interpolations used to express the trajectory data, as per official MPEG-7 ISO. The X3D Interpolator node key and keyValue attributes are provided in the Params element to describe the trajectory of the objects’ motion. In the case of dynamic components such as the X3DFollowerNode, annotation of the trajectory can only be described by their initial estimation through their initialValue and initialDestination attributes, as a smooth motion trajectory between dynamically produced vertices created within the scene during the rendering process.
TEI of Crete Multimedia Lab
MotionTrajectory
TEI of Crete Multimedia Lab
MotionTrajectory FieldsMPEG-7 Description Fields Value Type Input from X3D Fields
MotionTrajectory Descriptor - Created if when an X3D Interpolator exists.motionType String The X3D Interpolator motion activity type.motionComponent String The X3D Interpolator node name.
CoordDef - Child element of MPEG-7 MotionTrajectory Descriptor
Units String Value of “meter”, to specify 3D content held in Params element
Params - Child element of MPEG-7 MotionTrajectory Descriptor
KeyTimePoint - Child element of the Params element in the MPEG-7 MotionTrajectory Descriptor
MediaRelIncrTimePoint IntegerValue of key attribute of the X3D Interpolator node. One element for each X3D Intepolator node key attribute.
InterpolationFunctions - Child element of the Params element in the MPEG-7 MotionTrajectory Descriptor
KeyValue FloatValue of the keyValue attribute of the X3D Interpolator node. One element for each X3D Intepolator keyValue attribute.
TEI of Crete Multimedia Lab
The MotionType attributeX3D Interpolator Node MPEG-7 Motion Type
ColorInterpolator LinearCoordinateInterpolator LinearCoordinateInterpolator2D LinearNormalInterpolator LinearOrientationInterpolator LinearPositionInterpolator LinearPositionInterpolator2D LinearScalarInterpolator LinearSplinePositionInterpolator Non-Linear SplineSplinePositionInterpolator2D Non-Linear SplineSplineScalarInterpolator Non-Linear SplineSquadOrientationInterpolator Non-Linear OrientationEase-In-Ease-Out Ease-In-Ease-OutX3DFollowerNode Smooth
TEI of Crete Multimedia Lab
MotionTrajectory Mapping
TEI of Crete Multimedia Lab
Interactivity3D Elements
Name DefinitionInteractivity3DType
Describes the interactions that occur within an X3D file, specifically from the X3D ROUTE elements.
TriggerSourceThe source that triggers a specific animation. If the node is triggered internally depending on other nodes or script functions, the TriggerSource is defined as an InternalTrigger.
RouteThe corresponding X3D ROUTE node. Specifies the nodes it connects together and their equivalent node types.
TEI of Crete Multimedia Lab
Interactivity3D
TEI of Crete Multimedia Lab
Interactivity3D FieldsMPEG-7 Description Fields Value Type Input from X3D Nodes
Interactivity3DType -Specifies interactions contained within an X3D file, as defined in the X3D ROUTE node.
TriggerSource String“UserTrigger” or “InternalTrigger”, depending on the X3D fromNode attribute
Route XPath XPath route to the specific X3D ROUTE node.
fromNode String Value of the X3D fromNode attributefromNodeType String The X3D Node Type of the X3D
fromNode attributetoNode String Value of the X3D toNode attributetoNodeType String The X3D Node Type of the X3D toNode
attribute
TEI of Crete Multimedia Lab
Interactivity3D
TEI of Crete Multimedia Lab
MetadataHeader Fields
MPEG-7 Description Fields Value Type Input from X3D Fields
MultimediaContent Descriptor -
Created when the MPEG-7 DescriptionMetadata Descriptor is formed.
id String String value of the X3D document file name.
Multimedia-
Created when the MPEG-7 MultimediaContent element is formed.MediaLocator
MediaUri String String value of the X3D document file path.
TEI of Crete Multimedia Lab
MetadataHeader MultimediaContent Descriptor Mapping
TEI of Crete Multimedia Lab
Viewpoint3D Elements
Name DefinitionViewpoint3DType Position, orientation, animation properties and
coordinate system of each X3D Viewpoint node.Viewpoint3D Contains the description, fieldOfView, isAnimated and
animatedBy attributes.Viewpoint3DPosition Consists of three sub-elements, Orientation, Position
and CenterOfRotation, providing coordinates of the X3D Viewpoint.
Orientation Orientation coordinates of the Viewpoint node.
Position Positional coordinates of the Viewpoint node.
CenterOfRotation If the X3D Viewpoint node contains a centerOfRotation attribute, the value is passed into the CenterOfRotation.
TEI of Crete Multimedia Lab
Viewpoint3D
TEI of Crete Multimedia Lab
Viewpoint3D fieldsMPEG-7 Description
Nodes Value Type Input from X3D NodesViewpoint3DType -Viewpoint3D -description String String value of the description provided in the
corresponding X3D Viewpoint nodefieldOfView Float Float value of the X3D Viewpoint fieldOfView
attributeisAnimated Boolean Determines whether the X3D Viewpoint is
bound to an X3D animation nodeanimatedBy String The animation node type bound to the
Viewpoint nodeViewpoint3DPosition - The specific positional characteristics of the
X3D Viewpoint node within the X3D file.Orientation Float Vector The X3D Viewpoint orientation attributePosition Float Vector The X3D Viewpoint position attributeCenterOfRotation Float Vector The X3D Viewpoint CenterOfRotation attribute
TEI of Crete Multimedia Lab
Viewpoint3D Mapping
TEI of Crete Multimedia Lab
Lighting3D Elements
Name DefinitionLighting3DType Describes each X3D Lighting node contained in a scene.Lighting3D One Lighting node from the corresponding X3D file. Lighting3DCharacteristics Defines specific characteristics on the X3D Lighting node.Lighting3DSpecs The exact specifications on the X3D Lighting node
definition within the scene. It consists of four sub-elements, namely Attenuation, Colour, Direction, and Location.
Attenuation The attenuation properties of the X3D Lighting node. DominantColor Tcolour properties of the X3D Lighting node. Direction The directional lighting properties of the lighting node. Location The location properties of the illuminated geometry.
TEI of Crete Multimedia Lab
Lighting3D
TEI of Crete Multimedia Lab
Lighting3D FieldsMPEG-7 Description Nodes Value Type Input from X3D NodesLighting3DType -Lighting3D - One Light node from the X3D documentname String The DEF name provided in a Light nodelightingType String The Light type: Directional, Point or SpotLight.isOn Boolean The X3D Light node “on” attributeusedGlobally Boolean The X3D Light node usedGlobally attributeLighting3DCharacteristics - Specific characteristics of the Light nodeambientIntensity Float The X3D Light node ambientIntensity attributeintensity Float The X3D Light node intensity attributebeamWidth Float The X3D Light node beamWidth attributecutOffAngle Float The X3D Light node cutOffAngle attributeradius Float The X3D Light node radius attributeLighting3DSpecs - Specifications of the X3D Light nodeAttenuation Float Vector The X3D Light node attenuation attributeDirection Integer Vector The X3D Light node direction attributeLocation Integer Vector The X3D Light node location attribute
DominantColorMPEG-7:
DominantColorType
MPEG-7 DominantColor descriptor formed by the X3D Light colour attribute
TEI of Crete Multimedia Lab
Lighting3D Mapping
TEI of Crete Multimedia Lab
Extensions in MPEG-7 Part 5
Multimedia Description Schemes
TEI of Crete Multimedia Lab
Profile3D & Script3D
TEI of Crete Multimedia Lab
Profile3D Elements
Name DefinitionProfile3DType Indicates the profile type as defined in the
corresponding X3D file. (Core, Interactive, MPEG_interactive, Immersive)
TEI of Crete Multimedia Lab
Profile3D Fields
MPEG-7 Description
FieldsValue Type Input from X3D Fields
Profile3D Descriptor String The X3D root node profile attributeVersion Descriptor Float The X3D root node version attribute
TEI of Crete Multimedia Lab
Profile3D Version Mapping
TEI of Crete Multimedia Lab
Script3D Elements
Name DefinitionScript3DType The scripting type of the X3D Script node. externalScript The external scripting class that performs the
corresponding script functionalities on the X3D file described.
internalScript Specifies that the Script node contains internal script functions and does not contain an outer scripting class reference. The internal script type is Javascript by default, as specified by the X3D standard.
TEI of Crete Multimedia Lab
Script3D Fields
MPEG-7 Description Fields Value Type Input from X3D Fields
Script3D Descriptor - One for each Script nodeinternalScript String “JavaScript”, if no url attribute is
present in the X3D Script node
externalScript String“Java” or “Jscript”, if a url attribute is present and has a url with a .class or .js extension respectively
TEI of Crete Multimedia Lab
Script3D Mapping
TEI of Crete Multimedia Lab
Use of the ContentCollection Element
• An MPEG-7 DescriptionCollection element is created for each top-level X3D/Scene/X3D Grouping Node element that contains at least one Shape element.
• Subsequently, an MPEG-7 Content element is formed as a child element of the ContentCollection element previously defined, for each internal (child) X3D Grouping or Transform Node containing a Shape Node.
• The process for generating the Content elements is repeated recursively for all internal Grouping nodes at any level and a Content element is also created once for the top-level ContentCollection element.
TEI of Crete Multimedia Lab
Use of the ContentCollection Element
• The XSLT texture retrieval template utilizes the same algorithm as the Grouping retrieval.
• The main difference is that each MPEG-7 ContentCollection Descriptor is created for every X3D Grouping node that contains an X3D Shape node with a child X3D Appearance object.
• If the Appearance node contains any type of X3D Texture Objects (e.g. ImageTexture or MovieTexture), then an MPEG-7 Content element is created.
TEI of Crete Multimedia Lab
ContentCollection FieldsMPEG-7 Description Fields Value Type Input from X3D FieldsCollection Descriptor - Created if a top-level X3D Grouping
node existsContentCollection Descriptor -
name StringThe X3D DEF attribute in the X3D Grouping node. If no DEF attribute exists, a unique name is generated by the XSLT process.
id StringA unique id, generated by the XSLT process depending on the position of the X3D Grouping node
Content -One for each X3D Grouping Node at any sub-level. A content element is also created for the top-level Grouping node.
Multimedia - Created when the MPEG-7 Content element is formed.MediaLocator -
MediaUri XPath XPath route to the specific X3D Grouping node.
TEI of Crete Multimedia Lab
ContentCollection mapping
TEI of Crete Multimedia Lab
Content Fields for X3D Appearance nodes
MPEG-7 Description
FieldsValue Type Input from X3D Fields
Content - Created for each X3D Appearance node that contains an X3D Texture object.
Multimedia - Created when the MPEG-7 Content element is created.MediaLocator -
MediaUri URL String String value from the URL attribute of the X3D Texture object.
TEI of Crete Multimedia Lab
Content Mapping forX3D Appearance nodes
TEI of Crete Multimedia Lab
http://www.medialab.teicrete.gr:8080/X3DtoMPEG7Tool/mainPage.html