lily at hug uk

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org

LilySmart data at scale

big data,big problems

MOORE vs data

» coping with volume + need for timeliness = parallel processing

» data becomes business-critical = resilience through distributed architectures

» Hadoop, MapReduce, HBase:the future data platform

the CHALLENGES

» process ALL data» process data in REAL-TIME» derive INSIGHTS» provide INSTANT FEEDBACK

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 5

data STOREdata

warehouse analytics

batched, off-line, overnight

current thinking

1. store and manage all YOUR data

USERBehavior

2. store user behaviour, nearby

DATA data processing

USERBehavior

3. analyze usage patterns

data processing

domain knowledge

patternsruleskeywordslists...

USERBehavior

4. add domain knowledge

domain knowledge

USERBehavior

recommendationssemantic augmentationAnalytics

5. process, in real-time

domain knowledge

USERBehavior

6. augment data

SMARTER DATA data processing

domain knowledge

data insights

relations

SMARTER DATA data processing

domain knowledge

relations

SMART DATA, at SCALE... and in real time

data insights

stories

organisationsnameslocationsbrands

HYPER-PERSONALrecommendations

TOGETHERNESSinterestingness

news aggregator

productCATALOG

product familiesrelated activitiessocial graph

up-sellingCROSS-SELLING

recommendednessrelatedness

e-retail

real-time

patents

companiespeoplematerialsprocesses

competitiveinnovation

(dis)SIMILARITY

IP research

insights

Outerthought

» software product company» scalable content applications» open source product portfolio» Java, REST, internet

THIS NOTEBOOK BELONGS TO:

Noteblock_03.indd 1Noteblock_03.indd 1 23/05/10 14:4223/05/10 14:42

“The world is moving from content as a cost to data as an opportunity.”

data STORE datawarehouse

analytics+ +

real time

Lily 2.0

Lily 1.0 (CR)

Lily (now)

» Large-scale content storage, indexing and search» Current pilots

» up-to now: 4 man-years investment (since Sept/2009)

e-retail mobile media isp e-gov ip research

roadmap

» now: Lily 0.3» april 2011 : Lily 1.0» Q3 2011» real-time statistics + analytics

» Q2 2012 : Lily 2.0» real-time data processing engine» Data Insights

» Along the road:Lily SaaS edition

open source

»www.lilyproject.org» docs.outerthought.org/lily-docs-current/

Lily Core Concepts

» storage» HBase» repository model» versioning, varianting, mixins» indexing»mapping» search» SOLR

falling in love with Hbase : phase 1

» automatic scaling to large data sets» fault-tolerance» flexible datamodel with sparse data» commodity hardware» efficient random access» community-based open source» Java if possible

» need for consistency» atomic single-row updates»M/R for index regeneration

HBase» datamodel with column families and cell versioning» ordered tables with range scans» HDFS for blob storage» Apache

Lily Repository Model

Lily Datatypes

Mixins

Sample Lily Schema (excerpt)

namespaces:{/*Declarationofnamespaceprefixes.*/"org.lilyproject.bookssample":"b","org.lilyproject.vtag":"vtag"},fieldTypes:[{name:"b$title",valueType:{primitive:"STRING"},scope:"versioned"},{name:"b$pages",valueType:{primitive:"INTEGER"},scope:"versioned"},{name:"b$language",valueType:{primitive:"STRING"},scope:"versioned"},{name:"b$authors",valueType:{primitive:"LINK",multiValue:true},scope:"versioned"},

{name:"b$name",valueType:{primitive:"STRING"},scope:"versioned"},{name:"b$bio",valueType:{primitive:"STRING"},scope:"versioned"},{name:"vtag$last",valueType:{primitive:"LONG"},scope:"non_versioned"}],recordTypes:[{name:"b$Book",fields:[{name:"b$title",mandatory:true},{name:"b$pages",mandatory:false},{name:"b$language",mandatory:false},{name:"b$authors",mandatory:false},{name:"vtag$last",mandatory:false}]},

Lily Versioning

Flexible content model» generic enough to accomodate many popular content

schemas» HTML5, CMIS, RDF, NewsML, Dublin Core, ...» academically verified» not limited to ‘content applications’ only» developer convenience» higher level constructs» schema reuse» versioning, linking, ...

Lily Architecture(deployment)

HBase RowLog Library

» need for sync/async operations» updating of secondary indexes (i.e. tables)» feeding of Indexer (= bridge to SOLR index maintenance)» not: transactions» need for distribution and durability

HBase RowLog Library

» WAL» guaranteed execution of synchronous

actions

» call doesn’t return before secondary action finishes

» e.g. update secondary index tables

» if all goes well, size = #concurrent ops

» useful outside of Lily context as well!

» Queue» triggering of async actions

» e.g. (re)index (updated) record with SOLR back-end

» size depends on speed of back-end process

denormalization indexing of multiple versions of a record

incremental index updating

batch index building blob content extraction

sharding towards multiple SOLR

instances

The Lily Indexer

Indexing configuration (SOLR)

<types> [snipped: see SOLR example schema] </types>

<field name="title" type="text" indexed="true" stored="true" required="false"/> <field name="authors" type="text" indexed="true" stored="true" required="false" multiValued="true"/> </fields>

<defaultSearchField>title</defaultSearchField>

</schema>

Indexer configuration (Lily)

<?xml version="1.0"?><indexer xmlns:b="org.lilyproject.bookssample"> <cases> <case recordType="b:Book" variant="*" vtags="last" indexVersionless="true"/> </cases>

</indexer>

(opt.) Sharding configuration

{ shardingKey: { value: { source: "variantProperty", property: "language" }, type: "string" },

mapping: { type: "list", entries: [ { shard: "shard1", values: ["en", "it"] }, { shard: "shard2", values: ["nl", "de", "es"] } ] }}

Lily API

» Java (using Avro)» http://docs.outerthought.org/lily-docs-current/g3/g1/390-lily.html

» REST (HTTP + JSON)» http://docs.outerthought.org/lily-docs-current/g3/g2/427-lily.html

» All docs» http://docs.outerthought.org/lily-docs-current/ext/toc/

Demo» http://outerthought.blip.tv/file/4245615/

Lily and HBase

» adds high-level content model» data types» versioning» blob storage on HDFS» focus on sparse (efficient) storage» RowLog for synchronous cross-table updates and async

message queues

Lily and SOLR

» provides flexible mapping between HBase content model and SOLR index fields» interactive and batch (M/R) index maintenance» sharding» use(s) SOLR as-is: loose, flexible, extensible coupling» search access via SOLR (HTTP) API

Lily and CDH

»we intend to rely on CDH-‘blessed’ versions of HBase/HDFS/ZK» 700 patches and testing» next: adopting similar distribution lay-out» since we contribute patches to ASF HBase trunk, we would

expect CDH to track closely (until HBase 1.0)» some Lily users could be interested in ‘CDH-level’ services

goodbye

» It’s open source !» Content Repository: available now

(Lily model + HBase + SOLR + RowLog)» Lily 1.0 soon, will mainly focus on differentiating open

source and enterprise edition» “HBase is wa de max maat.”

Thank you !for your attentionfor your questions

» stevenn@outerthought.org

» @stevenn

lily at hug uk

zwijnaarde gent

data process data

big data

relationssmart data

lilysmart data

nearbyiic technologiepark

storiesiic technologiepark

parallel processing

Technology

hug bigtop

hug a startup

london hug-samza

hug geoconsult

“hug” size

tutorial fol-hug

the departing hug

hug ny meeting: presenting lily

hug pharmacy & sterilisation

46 the hug

the lily group summer visit to wales group newsletter autumn...

white lily residency 9350193692 white lily sonepatt

si-hug 2020

hug songbook essentials - halifax ukulele gang (hug)

artistic research joa hug joa hug no solutions: the

a modern hug

hug the pug

london hug

aug 2012 hug: hug bigtop

boston hug