mark logic user-group-2012

Future Media BBC MMXII

BBC Dynamic Semantic Publishing [DSP]MarkLogic User Group July 2012

• Jem Rayfield : Lead Technical Architect• BBC Future Media


Outline

BBC News Online

BBC World Cup 2010

BBC Sport 2012 + Olympics

BBC News Mobile


Radio since 1922 TV Since 1930 Web since 1994


http://bbc.co.uk/news

online


BBC News [Static Publishing]


Static News Architecture


BBC CPS/CMS

AssetAuthoring


BBC CPS/CMS

IndexAuthoring


Static NewsThe Good

1) Simple

2) Scales cheaply

3) Difficult to break [bad rendering logic etc..]

4) Handles high load


Static NewsThe BAD 1) Relational taxonomic

meta model

2) Static! Inflexible! SSI!

3) Document publishing

4) Content non re-usable

5) Content non repurpose-able

6) Difficult to personalize

7) Publication per output


BBC World Cup 2010

http://bbc.co.uk/worldcup


1. 32 teams, 8 groups, 736 players 776 pages

2. Fixtures & Results, Groups & Teams pages

3. To many web pages for too few journalists

4. Improve the publishing system to help achieve all of this

World Cup 2010


Page Per Playerhttp://news.bbc.co.uk/sport/football/world_cup_2010/groups_and_teams/team/england/wayne_rooney


Page PerTeam


Page PerGroup


BBC Sport: http://www.bbc.co.uk/ontologies/sportOpen Sport Ontology


Semantic publishing

USER EXPERIENCE

ONTOLOGY

TRIPLE STORE


Graffiti: Suggest -> Tag [Player]


Graffiti: Suggest -> Tag [Location] (Geonames)

Journalism BBC MMIX

Graffiti Demo… (maybe video, depending on wifi… )


Rationale• Automated content publishing

• Huge increase in content breadth (number of manageable pages)

• Content re-use and re-purposing, increasing reach

• Simplified content management

• Journalist headcount reduction

• Multi-dimensional entry points and semantic navigation

• Improved user experience with high levels of user engagement

• Dynamic, state (time|event) and semantic driven page layout

• Personalized content aggregations

• Open data and API’s


World Cup DSP Architecture


API Stack

Journalism BBC MMIX

Highly Scalable Clustered BigOWLIM

Journalism BBC MMIX

ExtendableDomain DrivenAssetTagging

Journalism BBC MMIX

Open Ontology/Dataset reuse Event | Geonames | Foaf | Etc.

Journalism BBC MMIX

Infer… player->team->competition

Journalism BBC MMIX

REST APIContent negotiation: json rdf, xml rdf, turtle

Publically accessible (with SSL cert)

GET /sport/football/teams/<TEAM>Accept: application/rdf+json

GET /sport/football/<COMPETITION>Accept: application/rdf+xml

GET /assets/<ASSET>Accept: text/rdf+n3

Etc….

Journalism BBC MMIX

<http://www.chelseafc.com/> domain:documentType <http://www.bbc.co.uk/things/document-types/homepage> , <http://www.bbc.co.uk/things/document-types/external> .

<http://www.bbc.co.uk/sport/football/teams/chelsea> domain:documentType <http://www.bbc.co.uk/things/document-types/bbc-document> , <http://www.bbc.co.uk/things/document-types/homepage> .

<http://www.bbc.co.uk/things/2acacd19-6609-1840-9c2b-b0820c50d281#id> a sport:CompetitiveSportingOrganisation ; domain:canonicalName "Chelsea"^^<xsd:string> ; domain:document <http://www.chelseafc.com/> , <http://www.bbc.co.uk/sport/football/teams/chelsea> ; domain:externalId <http://dbpedia.org/resource/Chelsea_F.C.> , <urn:sports-stats:137316635> ; domain:name "Chelsea" ; domain:shortName "Chelsea"^^<xsd:string> ; sport:competesIn <http://www.bbc.co.uk/things/5cd4682a-7643-f445-8b1f-bcbaf450bc89#id> .

<http://dbpedia.org/resource/Chelsea_F.C.> domain:externalIdType <http://www.bbc.co.uk/things/external-id-types/dbpedia> .

<urn:sports-stats:137316635> domain:externalIdType <http://www.bbc.co.uk/things/external-id-types/bbc-sport-stats> .

<http://www.bbc.co.uk/things/5cd4682a-7643-f445-8b1f-bcbaf450bc89#id> domain:canonicalName "Premier League"^^<xsd:string> ; domain:externalId <urn:sports-stats:118996114> ; sport:competitionType <http://www.bbc.co.uk/things/competition-types/domestic-league> .

GET Accept text/rdf+n3https://api.live.bbc.co.uk/dsp/sport/football/teams/chelsea

Journalism BBC MMIX

PHP->EasyRDF->APIPHP Render layer consumes RDF from REST API via EasyRDF (http://www.aelius.com/njh/easyrdf/)

EasyRDF open PHP library (Primary committer Nicholas Humfrey BBC)

protected function getOptions() { return array( "config" => array("usecert" => true), "headers" => array( "Accept" => "application/rdf+json", "X-Expect" => "http://www.bbc.co.uk/things/platforms/hiweb" )

);

$options = $this->getOptions()$response = $this->get("https://api.test.bbc.co.uk/dsp/sport/football/teams/chelsea", $options)$this->data = new EasyRdf_Graph("http://www.bbc.co.uk", $response->getBody());$teams = $this->data->allofType("sport:CompetitiveSportingOrganisation”)


• 750+ Dynamic aggregations/pages (Player, Squad, Group, etc..)

• Average unique page requests a day : 2 million +

• Average OWLIM SPARQL queries a day : 1 million

• 100s RDF statement updates/inserts per minute with full OWL reasoning and associated inference.

• Multi data center fully resilient, clustered 6 node triple store

• RDF graph model ideally suited to model domain representations such as sport

World Cup statistics the GOOD


• Sports stories and indices static

• Sport content not responsive or personalized

• RDF Store unable to handle thousands of statistic updates a second

• RDF Store forward-chained closures expensive increase write latency

• RDF graph model and SPARQL not ideally suited to the BBC’s News and Sport document publication model

World Cup statistics the BAD


BBC Sport 2012;Online Refresh

http://bbc.co.uk/sport


Sport Refresh 2012• Page per Athlete [10,000+], Page per country [200+], Page per

Discipline [400-500], Page per venue, Page per team A lot of output…

• Almost real time statistics and live event pages

• Time coded, metadata annotated, on demand video, 58,000 hours of content

• Far too many web pages for far too few journalists

• DSP annotation architecture to automate content aggregation


Sport Refresh 2012; Red -> Bright Yellow


10000+ Dynamic Aggregations


Lots of Dynamic (Live) sports stats


Dynamic Navigation


Static Stories + Dynamic includes


Olympics - 27 Live Video SteamsLive Stats overlaysStats -> Ontology driven aggregations


XML powered video chapter points

Page URLhttp://www.test.bbc.co.uk/sport/olympics/2012/live-video/p00g2lqp

Xml api to get video log eventshttps://api.test.bbc.co.uk/olympicdata/bdf-log/pid/p00g2lqp


Augment architecture with a Content Store

1. Atomic content assets stored in MarkLogic XML store

2. XML content queryable via Xquery

3. Content Assets searchable

4. Sports statistics searchable/queryable via XQuery

5. Ontological SPARQL via BigOWLIM, assets Xquery via MarkLogic


Static Sports stats (OLD)


Dynamic Sports stats


Olympics XML etc..


OntologyAwareNLP

• Information Workbench• OWLIM• (Spice) GATE+Ontotext


Extraction Highlights

OWLIM

Generic Analysis

…KB Gazetteer

…………

Disambiguation………

Relevance Ranking

Generic Analysis

…KB Gazetteer

…………

Disambiguation………

Relevance Ranking

Ex-England boss Sven-Goran Eriksson says a "smear campaign" has been aimed at Roy Hodgson for omitting Rio Ferdinand.

? Roy Hodgson: coach? Roy Hodgson: hockey player? ……….

V Roy Hodgson: coach-Roy Hodgson: hockey player- ……….

V Rio Ferdinand-…….- ……….

V Sven-Goran Eriksson-…….- ……….

CE

S A

PP

1. Eriksson (78%)

2. Roy Hodgson (69%)3. Rio Ferdinand (58%)4. …

Curate

Update

Retrain & Adapt


Disambiguation of Locations

•Geospatial distance - a feature of OWLIM•Super region – GeoNames hierarchy and containment relations, e.g. parentFeature•RDF Rank •Human approval score (on the basis of curated documents)•Class/code based priority – fine grained ontology may allow a rule or machine learning prioritization of classes and entities based on learning we already have.•Asset geo association - some entities could be disambiguated by using the asset domain association. BBC UK local sports is more likely to talk about national entities.


Entity Relevance: Objective

• Rank entities by their relatedness to the article• Accuracy 75%• We consider various frequencies of entity mentions in the article

and in the entire set of articles• Positions in the article fields or in the first paragraphs of the

body boost the relevance


DSPArchitecture


API Stack

MarkLogic!!


“You could run Nasa on that”Lee Pollington


Xquery master<->master rep


MarkLogic 5 & XA Tx (just… thanks John Snelson)


Plenty ofCaching


Sport Stats REST API


Sport Stats REST API Demo….


Olympics API (RDF)/tripod2012

/athletes /{uid}

/countries /{country}

/countries-iso /{iso}

/sports /stories /{discipline} /{discipline}/events /{discipline}/events/stories /{discipline}/events/{event}

/metadata /disciplines

/onestowatch /countries/{countryUrlName} /london2012 /sports/{disciplineUrlName} /sports/{disciplineUrlName}/events/{eventUrlName}

/podium/events /{rscCode}

/record/events/venues

/{urlName}


@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .@prefix owl: <http://www.w3.org/2002/07/owl#> .@prefix foaf: <http://xmlns.com/foaf/0.1/> .@prefix event: <http://purl.org/NET/c4dm/event.owl#> .@prefix tl: <http://purl.org/NET/c4dm/timeline.owl#> .@prefix geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> .@prefix geo-pol: <http://www.bbc.co.uk/ontologies/geopolitical/> .@prefix geoname: <http://www.geonames.org/ontology#> .@prefix dcterms: <http://purl.org/dc/terms/> .@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .@prefix exif: <http://www.w3.org/2003/12/exif/ns#> .@prefix sport: <http://www.bbc.co.uk/ontologies/sport/> .@prefix asset: <http://www.bbc.co.uk/ontologies/asset/> .@prefix tag: <http://www.bbc.co.uk/ontologies/tag/> .@prefix domain: <http://www.bbc.co.uk/ontologies/domain/> .@prefix sesame: <http://www.openrdf.org/schema/sesame#> .@prefix owlim: <http://www.ontotext.com/> .@prefix oly: <http://www.bbc.co.uk/ontologies/2012olympics/> .@prefix par: <http://purl.org/vocab/participation/schema#> .@prefix dc: <http://purl.org/dc/elements/1.1/> .

<http://www.bbc.co.uk/things/82f5db84-0591-49ee-b6f4-a1d26e9381fb#id> a sport:Person ;rdfs:label "Usain Bolt"^^xsd:string , "Bolt Usain-athletics-jam-1986-08-21"^^xsd:string ;foaf:name "Usain Bolt"^^xsd:string , "Bolt Usain-athletics-jam-1986-08-21"^^xsd:string ;domain:canonicalName "Bolt Usain-athletics-jam-1986-08-21"^^xsd:string ;foaf:givenName "Usain"^^xsd:string ;foaf:familyName "Bolt"^^xsd:string ;domain:name "Usain Bolt"^^xsd:string ;oly:dateOfBirth "1986-08-21"^^xsd:date ;oly:gender "M"^^xsd:string ;oly:height "195.0"^^xsd:float ;oly:weight "94.0"^^xsd:float ;oly:worldOlympicDream "true"^^xsd:boolean ;sport:discipline <http://www.bbc.co.uk/things/b3a086df-ab42-2b44-be8b-76b600bfcdce#id> ;sport:competesIn <http://www.bbc.co.uk/things/1b499a08-4f02-4196-aa6c-c43ea353138b#id> .

<http://www.bbc.co.uk/things/b3a086df-ab42-2b44-be8b-76b600bfcdce#id> a sport:SportsDiscipline ;domain:name "Athletics"^^xsd:string ;domain:document <http://www.bbc.co.uk/sport/olympics/2012/sports/athletics> .

<http://www.bbc.co.uk/things/1b499a08-4f02-4196-aa6c-c43ea353138b#id> a sport:MedalCompetition ;domain:name "Men's 100m"^^xsd:string ;domain:shortName "Men's 100m"^^xsd:string ;domain:document <http://www.bbc.co.uk/sport/olympics/2012/sports/athletics/events/mens-100m> ;oly:measurementType <http://www.bbc.co.uk/things/measurement-types/time> ;domain:externalId <urn:ioc2012:ATM001000> .

<http://www.bbc.co.uk/things/903ef380-bdae-4a45-9a8b-5e5a270a7d6c#id> oly:oneToWatch <http://www.bbc.co.uk/things/82f5db84-0591-49ee-b6f4-a1d26e9381fb#id> .

<http://news.bbc.co.uk/sport1/hi/athletics/16554814.stm#asset> tag:tag <http://www.bbc.co.uk/things/a50dc8ba-947e-4856-8eb0-1cdbbf208ef7#thing> ;dc:title "Event Guide: ATHLETICS"^^xsd:string ;asset:storyType <http://www.bbc.co.uk/things/story-types/profile> ;domain:document <http://news.bbc.co.uk/sport1/mobile/athletics/16554814.stm> .

<http://www.bbc.co.uk/things/a50dc8ba-947e-4856-8eb0-1cdbbf208ef7#thing> tag:taggedWithTag <http://www.bbc.co.uk/things/b3a086df-ab42-2b44-be8b-76b600bfcdce#id> .

<http://news.bbc.co.uk/sport1/mobile/athletics/16554814.stm> domain:platform <http://www.bbc.co.uk/things/platforms/mobile> .

Olympics API (RDF)


Olympics API (XML)/olympicdata/

/athletes /simulator/{scenarioName} /{guid}

/bdf-log /pid/{pid} /pid/{pid}/chapter-points /{logId}

/chapter-points /pid/{pid} /{logId}

/days-to-go/live-text

/assets /assets/{id}

/medallists /athletes/{guid} /{medalGroup}

/medals /athletes/{athleteGuid} /countries/{country}

/medaltable /countries/{country} /disciplines/medals /disciplines/{rsc} /overall

/sportcontent/obsvideosessions

/full /update

/podium /countries/{country} /disciplines/{rsccode} /events/{rsccode} /latest/pulse /beat /beats/records /athletes/{guid} /events/{rsc}/results /athletes/{guid}/schedule /detail/days/{date} /detail/disciplines-code/{rsccode} /detail/disciplines-code/{rsccode}/events/{eventrsccode} /detail/disciplines-code/{rsc}/days/{date} /detail/disciplines/{rsc}/days/{date} /detail/disciplines/{urlname} /detail/disciplines/{urlname}/days/{date} /detail/disciplines/{urlname}/events/{eventrsccode} /overview/days /overview/disciplines /overview/disciplines/{rsc}/stats /count/{directory:.+} /sessionCode/{sessionCode} /simulator/{scenarioName} /{documentSerial}/team /{odfid}/unit-status /{sessionCode}/video /days/{date} /videosessionid/{videoSessionId} /{pid}


Olympics API (XML)/olympicdata/

/pulse /beat /beats

/records /athletes/{guid} /events/{rsc}

/results /athletes/{guid}

/schedule /detail/days/{date} /detail/disciplines-code/{rsccode} /detail/disciplines-code/{rsccode}/events/{eventrsccode} /detail/disciplines-code/{rsc}/days/{date} /detail/disciplines/{rsc}/days/{date} /detail/disciplines/{urlname} /detail/disciplines/{urlname}/days/{date} /detail/disciplines/{urlname}/events/{eventrsccode} /overview/days /overview/disciplines /overview/disciplines/{rsc}

/stats /count/{directory:.+} /sessionCode/{sessionCode} /simulator/{scenarioName} /{documentSerial}

/team /{odfid}

/unit-status /{sessionCode}

/video /days/{date} /videosessionid/{videoSessionId} /{pid}

Olympics API (XML - Medallist)

https://api.int.bbc.co.uk/olympicdata/public/medallists/men

<?xml version="1.0" encoding="UTF-8"?><document> <header> <documentGroup>general</documentGroup> <documentType>medalistCompact</documentType> <rsc code="GEM000000"/> <documentSerial>b1df2feb-1bcd-43b4-aa61-8ae9c82c3bb9</documentSerial> <timeStamp>2012-03-30T15:42:13.843Z</timeStamp> </header> <medalist>

<m o="1" r="1" i="82f5db84-0591-49ee-b6f4-a1d26e9381fb" g=“2" s="1" b="0" t="2" c=”JAM">Usain Bolt</m> <m o="2" r="1" i="56bdea69-dce5-4cab-91ff-802be5077350" g="1" s="0" b="0" t="1" c="NED">VOGELS Guus</m> <m o="3" r="1" i="1f199d68-3a7d-44fa-adae-a5cde084395b" g="1" s="0" b="0" t="1" c="NED">DERIKX Rob</m> <!- blah blah and blah -> </medalist></document>

XML and RDF data inter-linked via GUIDS


Olympics API (XML – Video catch example)https://api.stage.live.co.uk/olympicdata/public/videos/catchup

<?xml version="1.0" encoding="UTF-8"?><catchup poll-interval-in-seconds="60"> <slot index="1" type="session"> <pid>b017t8b4</pid> <videoSessionId>OBS-1284552321</videoSessionId> <sessionCode>FB001</sessionCode> <discipline RSC="FB0000000" name="Football" url="http://www.bbc.co.uk/sport/olympics/2012/sports/football" urlName="football" icon="http://static.live.bbci.co.uk/ivp2012/images/icons/sports/football.png"/> <scheduleStart>2001-12-17T09:30:47Z</scheduleStart> <scheduleEnd>2001-12-17T09:30:47Z</scheduleEnd> <actualStart>2001-12-17T09:30:47Z</actualStart> <actualEnd>2001-12-17T09:30:47Z</actualEnd> <available>true</available> <editorsPick>true</editorsPick> <live>false</live> <title>28/11/2011</title> <shortSynopsis>A desperate Roxy makes the unwise decision to dig for dirt on Derek.</shortSynopsis> <myImageBaseUrl>http://node2.bbcimg.co.uk/iplayer/images/episode/</myImageBaseUrl> </slot> <slot index="2" type="session"> <pid>hbtest1</pid> <videoSessionId>OBS-64835275893</videoSessionId> <sessionCode>HB001</sessionCode> <discipline RSC="HB0000001"/> <scheduleStart>2001-12-17T09:30:47Z</scheduleStart> <scheduleEnd>2001-12-17T09:30:47Z</scheduleEnd> <actualStart>2001-12-17T09:30:47Z</actualStart> <actualEnd>2001-12-17T09:30:47Z</actualEnd> <available>true</available> <editorsPick>true</editorsPick> <live>false</live> </slot> </catchup>


Olympics API (Video log/chapter points)GET https://api.test.bbc.co.uk/olympicdata/bdf-log/pid/p00g2lqp

<document> <logEvent> <header> <documentGroup>general</documentGroup> <documentType>videoLogging</documentType> <rsc code="FE0000000"/> <documentSerial>fdf38714-45d3-4d89-5100-8883be340700</documentSerial> <timeStamp>2012-07-10T10:42:38+01:00</timeStamp> <video> <pid>p00g2lqp</pid> <timeCode>2012-07-10T08:56:38Z</timeCode> </video> <taggingtool>Version 0.1.10</taggingtool> </header> <Log> <LogId>BBC-TT-1341913358.6783</LogId> <Action>UPDATE</Action> <Date>2012-07-10</Date> <TimeCode>09:56:38</TimeCode> <RSC>FE0000000</RSC> <Keywords> <Keyword>BBC free text</Keyword> </Keywords> <bbcLabel>Test One</bbcLabel> </Log> </logEvent> <logEvent> <header> <documentGroup>general</documentGroup> <documentType>videoLogging</documentType> <rsc code=""/> <documentSerial>20dc4019-d3be-4bb1-6a75-256ba7622b5d</documentSerial> <timeStamp>2012-07-10T10:42:51+01:00</timeStamp> <video> <pid>p00g2lqp</pid> <timeCode>2012-07-10T09:16:38Z</timeCode> </video> <taggingtool>Version 0.1.10</taggingtool> </header> <Log> <LogId>BBC-TT-1341913371.5765</LogId> <Action>UPDATE</Action> <Date>2012-07-10</Date> <TimeCode>10:16:38</TimeCode> <RSC/> <Keywords> <Keyword>BBC free text</Keyword> </Keywords> <bbcLabel>Test Two</bbcLabel> </Log> </logEvent></document>


Large A1 printout…. WayToo big for a slide..

Olympics Architecture


2012 http://m.bbc.co.uk/news

online


Dynamic News mobile

• Multi device capability

• Responsive Web design

• Built on a dynamic service API

• New re-usable content model

• Dynamic assets


Responsive Dynamic News mobile (iPad)


Responsive Dynamic News mobile (iPhone)


Recap Static News Architecture


Dynamic News Architecture


New Content Model (Re-usable XML/RDF )

MarkLogics handy Xinclude resolution

Including story data on news index XML

<item> <xi:include href="http://www.bbc.co.uk/asset/13447877" xpointer="xmlns(bbc=http://www.bbc.co.uk/content/asset) xpointer(/bbc:story/bbc:itemMeta)"> <xi:fallback>  </xi:fallback> </xi:include> ...


News Index API


HTTP GEThttps://api.live.bbc.co.uk/content/asset/news/technology/

HTTP HeadersX-Candy-Audience: DomesticX-Candy-Platform: EnhancedMobileAccept: application/json

Or

HTTP HeadersX-Candy-Audience: DomesticX-Candy-Platform: EnhancedMobileAccept: application/xml

Contextualised output•Audience•Platform•Response type


News Story API


HTTP GEThttps://api.live.bbc.co.uk/content/asset/news/uk-17829360

HTTP HeadersX-Candy-Audience: DomesticX-Candy-Platform: EnhancedMobileAccept: application/json

Or

HTTP HeadersX-Candy-Audience: DomesticX-Candy-Platform: EnhancedMobileAccept: application/xml


BBC sport site re-engineered to use fully dynamic approach (News Mobile style)

BBC news high web site re-engineered to use fully dynamic approach (News Mobile style)

MarkLogic as CMS repository (iSite)

MarkLogic Binary storage R&D

Etc….etc..

Platform future…..


Questions?

Slides:

[email protected]: @jemrayfield

mark logic user-group-2012

Technology

apisfuture media bbc

uksportfuture media

outputfuture media bbc

thisfuture media bbc

uknewsfuture media bbc

apistackfuture media

assport future media

bbc news online bbc