ElasticSearch

Introduction and quick startupmedcl 9-29

introduction

• ElasticSearch,a distributed search solution ，– domain driven– schema free– anything pluggable– open source, distributed, RESTful

• Author:shay.banon (expert in search and analytics)– Compass– GigaSpaces

• Current Version 0.11.0

Features

• Reliable, Asynchronous Write Behind for long term persistency.

• (Near) Real Time Search.• Built on top of Lucene.– shard is a fully functional Lucene index.– All the power of Lucene easily exposed through simple

configuration / plugins.• Per operation consistency– Single document level operations are atomic, consistent,

isolated and durable.• Open Source under Apache 2 License.

Distributed and Highly Available

• Each index is fully sharded with a configurable number of shards.

• Each shard can have zero or more replicas.• Read / Search operations performed on either

replica shard.

Multi Tenant with Multi Types.

• Support for more than one index.• Support for more than one type per index.• Index level configuration (number of shards,

index storage, ...).

Document oriented

• No need for upfront schema definition.• Schema can be defined per type for

customization of the indexing process.

Various set of APIs.

• HTTP RESTful API.• Native Java API.• 3rd Clients– perl、 python、 php、 ruby、 groovy、 erlan

g、 .NET• All APIs perform automatic node operation

rerouting.

UP AND RUN

install

• Zero Conf

index• $ curl -XPUT http://localhost:9200/twitter/user/kimchy -d '{ "name" : "Shay

Banon" }'

$ curl -XPUT http://localhost:9200/twitter/tweet/1 -d '{ "user": "kimchy", "post_date": "2009-11-15T13:12:00", "message": "Trying out Elastic Search, so far so good?" }'

$ curl -XPUT http://localhost:9200/twitter/tweet/2 -d '{ "user": "kimchy", "post_date": "2009-11-15T14:12:12", "message": "You know, for Search" }'

Schema mapping

• $ curl -XPUT http://localhost:9200/twitter

$ curl -XPUT http://localhost:9200/twitter/user/_mapping -d '{ "properties" : { "name" : { "type" : "string" } }}'

GET

• $ curl -XPUT http://localhost:9200/twitter/tweet/2 -d '{ "user": "kimchy", "postDate": "2009-11-15T14:12:12", "message": "You know, for Search" }'

$ curl -XGET http://localhost:9200/twitter/tweet/2

Search• $ curl -XPUT http://localhost:9200/twitter/tweet/2 -d '

{ "user": "kimchy", "postDate": "2009-11-15T14:12:12", "message": "You know, for Search" }'

$ curl -XGET http://localhost:9200/twitter/tweet/_search?q=user:kimchy

$ curl -XGET http://localhost:9200/twitter/tweet/_search -d '{ "query" : { "term" : { "user": "kimchy" } } }'

$ curl -XGET http://localhost:9200/twitter/_search?pretty=true -d '{ "query" : { "range" : { "post_date" : { "from" : "2009-11-15T13:00:00", "to" : "2009-11-15T14:30:00" } } } }'

multenancy• $ curl -XPUT http://localhost:9200/kimchy

$ curl -XPUT http://localhost:9200/elasticsearch

$ curl -XPUT http://localhost:9200/elasticsearch/tweet/1 -d \'{ "post_date": "2009-11-15T14:12:12", "message": "Zug Zug", "tag": "warcraft" }'

$ curl -XPUT http://localhost:9200/kimchy/tweet/1 -d \'{ "post_date": "2009-11-15T14:12:12", "message": "Whatyouwant?", "tag": "warcraft" }'

$ curl -XGET http://localhost:9200/kimchy,elasticsearch/tweet/_search?q=tag:warcraft

$ curl -XGET http://localhost:9200/_all/tweet/_search?q=tag:warcraft

Setting• $ curl -XPUT http://localhost:9200/kimchy/ -d \

'index : store: type: memory'

$ curl -XPUT http://localhost:9200/elasticsearch/ -d \' { "index" : { "number_of_shards" : 2, "number_of_replicas" : 3 }}'

BEHIND ELASTICSEARCH

Modules

scalability

• nodes that can hold data, and nodes that do not.

• There is no need for a load balancer in elasticsearch, each node can receive a request, and if it can’t handle it, it will automatically delegate it to the appropriate node(s).

• If you want to scale out search, you can simply have more shard replicas per shard.

automatic shard allocation

From:http://www.slideshare.net/elasticsearch/elasticsearch-at-berlinbuzzwords-2010#

BASE support

• Each document you index is there once the index operation is done.

• No need to commit or something similar to get everything persisted.

• A shard can have 1 or more replicas for HA. • Gateway persistency is done in the

background in an async manner.

The River

• A river is a pluggable service running within elasticsearch cluster pulling data (or being pushed with data) that is then indexed into the cluster.

Geo Location and Search

• 1. make your data geo enabled• {

"pin" : { "location" : { "lat" : 40.12, "lon" : -71.34 }, "tag" : ["food", "family"], "text" : "my favorite family restaurant" }}

• Find By Location• Sorting• Faceting … …

• More details in http://www.elasticsearch.com/docs/

COMPARISON

Compare with solr

• Though support dynamic schema,but it sucks– *i ,name_i,age_i,….

• Distribute ,just do many replica,Master-Slave,and with a dirty query like this:– http://localhost:9080/solr/select/?q=xxx:xxx&shar

ds=localhost:8080/solr,localhost:9080/solr WTF!

• Does it really RESTful?anyway, doesn’t matter

Compare with katta• Featrures

– Makes serving large or high load indices easy– Serves very large Lucene or Hadoop Mapfile indices as index shards on many servers– Replicate shards on different servers for performance and fault-tolerance– Supports pluggable network topologies– Master fail-over– Fast, lightweight, easy to integrate– Plays well with Hadoop clusters

• May heavy to us(may be not)• Master-Node,complex and ops will killed us？ can’t be a little easy？• Lack of Client and documents• Inactivity Community• Lake of Some Search Features

RESOURCES

• Link：– http://www.elasticsearch.com– http://www.elasticsearch.com/blog– http://www.elasticsearch.com/docs/– http://www.elasticsearch.com/community/mailinglist/user/– http://github.com/elasticsearch

• References:– http://highscalability.com/blog/2010/2/10/elasticsearch-open-s

ource-distributed-restful-search-engine.html– http://blog.sematext.com/2010/05/03/elastic-search-distribute

d-lucene/– http://mail-archives.apache.org/mod_mbox/hbase-user/201006

.mbox/%3C149150.78881.qm@web50304.mail.re2.yahoo.com%3E

– http://www.slideshare.net/elasticsearch/elasticsearch-at-berlinbuzzwords-2010#

• Thanks/