ELASTIC @ SHOPBACK

Alberto Resco PerezEngineering Manager

Elastic Meetup Singapore – Dec 15, 2016

Agenda

1. Introduction to ShopBack2. Product Catalog in ShopBack3. Logs/Metrics Collection4. Future development5. QA

Our Founding Journey

Who we are

• We are the smarter way to shop • Over 1.5M shoppers in 6 countries have adapted the

shop-and-save lifestyle• Southeast Asia’s No. 1 Online Cashback portal• Bringing happiness to users via:

• Cashback Rewards• New Store Discoveries• Enhanced Shopping Experiences

What we do

Where we are

SG: SEP 2014MY: FEB 2015PH: JUN 2015IN: JAN 2016ID: MAR 2016TW: SEP 2016

Product CatalogChallenges

Thousands of categories & stores, Millions of Products

Fast, Reliable & Concise information

Multi country – Multi Language

Presentation in small screens

Product CatalogChallenges

9

Gateway

API Product Service

MySQLMongoDB

Web IOS Android Extension

Product CatalogMoving towards micro-services

• Moved to our Elasticsearch 2.4 cluster -> Already in 5.0!• Latest version with new features available

• Schema for multi-country, multi-language• Indices

• “Product” micro-service• Performance is great

Product CatalogMoving towards micro-services

Crawler

Product CatalogProduct Updates

CrawlerSynchronizer

MapperImage Uploader

Indexer

No Cluster Metrics

Slow browsing

Limited by language

Fast browsing based on ES

Large amount of SKUs

Language is not a issue

Cluster Metrics collected

Product CatalogPost-elastic implementation

Initial Setup

Cluster v1

Clusterv2

• Spin up cluster• Basic setup

• Single index, Single type by country:

• Products x 5• Stores x 5

• Better configuration of analyzers (different languages)

• Removed Boolean queries

• Single index, 2 types:• Products• Stores

• Types complete different, need different analyzers…

• Multiple Boolean queries• Country, domain

Product CatalogOur journey to current status

Clusterv3

• Single index, Single type by country:

• Products x 6• Stores x 6• Categories x 6• Campaigns x 6

• Auto-suggestion

Before Now

Documents Indexed• 10M products• 2k stores• 15k categories• 100+ custom pages• Updated when we find a change

Performance• Product search: 1ms• Store listing: 4s• Auto-suggestion: Live. 100ms

Product CatalogSome data

Documents Indexed• 60k products• Updated every week

Performance• Product search: 2s• Store listing: 30s• Auto-suggestion: 30Kb preload

• Papertrail‒ Cloud provided $$$‒ 7 days Retention‒ No dashboards

• AWS Cloud Watch‒ 2 days Retention‒ 5 Minutes interval or 1 Minute interval $$$‒ Not all metrics

• New Relic‒ Cloud provided -> $$$$$$$$‒ 3 months Retention‒ 5 Minutes interval or 1 Minute interval -> $$$‒ Not all metrics

Logs / Metrics CollectionFormer/Legacy System

Logs / Metrics CollectionFormer/Legacy System

Logs / Metrics CollectionELK based

• Implement country multi-language -> Using different indexes

• Implement fuzzy matching

• Recommendation engine

• Machine Learning Fraud detection

• Collect all services metrics with Metricbeat Mongodb, MySQL, Memcached

• Move Logstash to Filebeat lightweight shipper

• Setup Alerting -> X-Pack Alerting

Future Implementations

• Previously:‒ Papertrail‒ New Relic‒ AWS CloudWatch Metrics

• Moving towards Beats + Kibana‒ All covered except APM (NewRelic)‒ First Micro-Service (Product Catalog) migrated

Log management and metrics collection