lily @ work webinar

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org

Lily @ Workwebinar – july 7th 2011


Overview

» Big Data @ Work: opportunities & challenges» Lily: a smart Big Data solution» Roadmap & status» Use cases» The Lily Adoption Program» Conclusions

2


Your host

» Steven Noelsfounder and CEO of [email protected]@stevenn

3

ask questions!


Big data @ Work

» a practical view on opportunities» store now, act later» live archives» scaling for data + audience» challenges» schemas with a vengeance» index management & maintenance

4


Store Now, act later

» no Big Design Upfront» smart & scalable storage» at-leisure indexing & processing» batch, map-reduce-based» flexible and versionable schema to cope with

increasing subject-matter understanding

5


Live archives» A lot of incumbent EDM/RM solutions focus on scale at

the cost of functionality» “you’ll find your data if you know where to look”» Library systems are often under-engineered for Big Data» “we index 10.000 documents per hour ... at night”» General developer’s need for combo

of storage + put|get|post|delete(CRUD) + index + search

6

» classic architecture :» different server node types

» complex application deployment management

» upgrade nightmare

»multiple bottleneck possibilities (web/app/db)


Scaling for data + audience

7

front-end tier

application logic tier

data tier

interwebs

front-end tier services

application logic tier services

data tier services

interwebs» ‘webscale’ setup» (mostly) identical nodes

» load spread evenly across CPUs and spindles

»move app services between nodes (rather than node reconfigs)


Scaling for data + audience

8


Schemas with a vengeance

» ‘schema-free’ is not the long-term Big Data selling point» you want APIs/contracted contact surfaces in your app

architecture» Big Data is now moving beyond log file analytics → more complex (domain-inspired) data models

» ongoing insights → evolving schemas» ... schema flexibility is.

9


Index maintenance & management

» NOSQL → NO QUERY ?» all index-based access must be defined up-front» increasing need for specialized, real-time indexed data access» structured + full-text (idf) searching» increasing data corpus size = increasing index

maintenance cost» need for both interactive index updating + batch index

regeneration

10


Enter Lily, a solution for big data

» smart : copes with large-scale data- and audience sets through a simple yet flexible API and model» scale : a fully distributed architecture» easy : empowering enterprise devs with reality-inspired

tooling(batteries included: schema, indexing, search)

11×


LILY

The Real-time Platform built for the Age of Data.

We manage, track and measure your data and users,and do the mat(c)hmaking in-between:» provide you with business intelligence and analytics» harvest user profiles and learn their interests» dynamically engage your users using quality recommendations

12


where would you use lily?

» large collections of data» content repositories

» library catalogs

» (media) asset management

» product catalogs

» ‘live’ archives

» large groups of users» e-commerce / retail

» news / media

» ... if you want to use big data, but you need easy.

13

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 14

+this i

s where th

e magic happens


Roadmap & status» Out now: Lily 1.0.1» Out real soon:» Lily embed - developer-friendly lightweight Lily launching

options for use in (unit) tests» Cloudera whirr-based (EC2) cluster deployment tools» Next (Q4 2011):» Even more flexible data model

(complex attributes, e.g. tags, keywords, substructures)» Secondary Indexes (pure HBase-based indexes)

15


Next up: usage stats» sits in CRUD-path» tracks users ops against

records» from both perspectives» arbitrary K/V properties: time,

location, ...

» automatically builds user profiles (as records)» tied to records ops» indexed access» time dimension: trending

16

interactions

indexes

time

recommendationsrecord user


from usage stats to recommendations ‘light’

» grouping of users based on» shared properties» shared record access» grouping of records based on» shared properties» shared user operations

17

record user

{connections

recommendations


Use cases

» Adeo» Fatminds»Media

18


Groupe adeo

» ADEO Groupe = Nr. 4 DIY retailer worldwide» Project “Wikeo”: data- and services platform to back

business-unit-operated e-commerce websites» product data, profiles, pricing info, product manuals ...» Lily as a data aggregation, interactivity & scaling

platform

19


Fatminds

» US startup: “Kayak meets Yelp for continuing education”» originally based on Alfresco, started using Lily as their

go-to-web scalability platform»www.fatminds.com

20


media

» USA magazine publisher» newsfeed server

» feed aggregation

» personalization through Solr searches

» loading via custom Ruby app

» Swedish newspaper publisher

external (editorial) systems

core services : user profiles / entitlement content / analytics

extended services

webmobile

archive

...


Lily Adoption program

» facilitated multi-step process» qualify challenges» explore opportunities» learn + change

23

explore

discover

adopt

deploy

support


Lily adoption program

» open source:www.lilyproject.orgbusiness-friendly Apache license

24

explore

discover

adopt

deploy

support



» 2-day workshop-facilitated exploration of opportunities» creating a true business case»make benefits tangible & measurable» architecture study» prepare proof of concept

25

explore

discover

adopt

deploy

support



» classroom developer training» PoC implementation and validation» project guidance and assistance» professional services» work with integrators» architecture and implementation review

26

explore

discover

adopt

deploy

support



» going live» deployment tools» Lily cluster install tools» cluster mgmt» Lily admin application

27

explore

discover

adopt

deploy

support



» enterprise support» bug fixes, hot fixes» intermediate releases» access to roadmap

28

explore

discover

adopt

deploy

support

» Go and discover !» Exploration workshop -20%» This Summer (until of end of September ’11)» Excl. travel, incl. extensive workshop report

✘IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org

How to get started

29

US$ 9,000 ➙ US$ 7,250

€ 6.250 ➙ € 5.000


conclusions

Lily allows youto collect, process and manage your data,

all of it,in a developer-friendly packaging,

to grow knowledge about your data,and how it is being used.

30

lily @ work webinar

Technology

zwijnaarde gent

largescale data

big data smart

age of data

longterm big data

overview big data

thiiic technologiepark

trendingiic technologiepark