C L O U D Y W I T H A C H A N C E O F S C A L E

M I T C H P I R T L E E L A S T I C S E A R C H M E E T U P C A P I TA L O N E L A B S , A R L I N G T O N VA D E C E M B E R 2 0 1 6

A B O U T M E

• FOSS founder/contributor

• Startupper

• Technology Fellow, Capital One

• Skate punk

• Musician

• Football coach

A B O U T Y O U

• Just a user

• In production by hand

• Fully automated in production

• I should be giving this talk

A B O U T T H I S S E S S I O N

• ES from an Ops perspective

• The good, bad and truncated

• What to expect

H O W I T A L L S TA R T E D

– M E , T H E E A R LY- O N C L U E L E S S V E R S I O N

“All I want is a web crawler, this can’t be too hard to setup.”

– M E , T H E E A R LY- O N C L U E L E S S V E R S I O N

“All I want is a web crawler, this can’t be too hard to setup.”

F E S S U P !

• Fess: http://fess.codelibs.org/

• FOSS

• Simple web UI for setup

• Multiple sources

• Multiple types

• Supports threads, throttling

• ES as persistence store

S C E N A R I O S .

L O C A L S E T U P

• Fess comes as a self-contained package

• All you need is a java runtime and you’re good

I ’ M T H I N K I N G B I G G E R T H A N T H AT.S C R E W L O C A L H O S T,

W H Y P U B L I C C L O U D

• Horizontal scale

• Access to integrated services - storage, load balancing, etc

• Opportunity to automate. All. The. Things.

S I N G L E I N S TA N C E .

• Quick to setup, uses embedded Elasticsearch + plugins

• Quick to duplicate

• Easy to maintain

S I N G L E I N S TA N C E : I S S U E S

• Single point of failure

• Zero scale opportunity

• Fully manual effort

• Good golly filesystem access is SLOW

C L U S T E R E D S E R V I C E .

• Horizontal scale

• Easier to expand

C L U S T E R E D S E R V I C E : I S S U E S

• Needs separate instance of Elasticsearch, requires plugin installation (version specific)

• Even harder to setup: Shards or replicas? Master or data or both, or neither?

• Even harder to automate

• Requires additional tooling for operations (logs, events)

H O W D O I D O T H I S ?

• ElasticSearch Cluster: Configuration & Best Practices (http://www.xmsxmx.com/elasticsearch-cluster-configuration-best-practices/)

A M A Z O N E L A S T I C S E R V I C E

• No setup

• Simple scale

• Fully automated

• Advanced configuration by default

A M A Z O N E L A S T I C S E R V I C E : I S S U E S

• No custom plugins (no Fess!)

• Service limits (number of nodes per cluster, etc)

• Lack of customization options

T H E L I N K S

• https://aws.amazon.com/elasticsearch-service/

• http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html

S T U M P T H E S P E A K E RM Y FA V O R I T E G A M E

T H A N K Y O U V E R Y G R A Z I E .

• @mitchitized

• mitch.pirtle@capitalone.com

• github.com/spacemonkey

• about.me/mitchitized

• www.slideshare.net/spacemonkeylabs