riak search 2: yokozuna

Eric Redmond@coderoshi

Tuesday, November 12, 13

[Slide Intentionally Left Blank]

A million values isn’t cool.You know what’s cool?...

A trillion values spread acrosslots of nodes that auto-index and you canalso search in cool ways like geographicallyor maybe you could extract from a pdf on the server side if you used the extractor API

┬─┬ ( ^_^ノ)

## The enabled Yokozuna set this 'on'.yokozuna = off

## The port number which Solr binds to.yokozuna.solr_port = 10014

## The port number which Solr JMX binds to.yokozuna.solr_jmx_port = 10013

## The arguments to pass to the Solr JVM. Non-standard## arguments, i.e. -XX, may not be portable across JVM## implementations. E.g. -XX:+UseCompressedStrings.yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX:+UseStringCache -XX:+UseCompressedOops

## The data under which to store all Yokozuna related data.## Including the Solr index data.yokozuna.data_dir = ./data/yz

riak.conf

## The enabled Yokozuna set this 'on'.yokozuna = on

## The port number which Solr binds to.yokozuna.solr_port = 10014

## The port number which Solr JMX binds to.yokozuna.solr_jmx_port = 10013

## The arguments to pass to the Solr JVM. Non-standard## arguments, i.e. -XX, may not be portable across JVM## implementations. E.g. -XX:+UseCompressedStrings.yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX:+UseStringCache -XX:+UseCompressedOops

## The data under which to store all Yokozuna related data.## Including the Solr index data.yokozuna.data_dir = ./data/yz

riak.conf

riak-admin cluster join riak@10.0.0.2... and so on...riak-admin cluster planriak-admin cluster commit

riak-admin security add-user eric 12345riak-admin security add-user admin 123456riak-admin security grant yokozuna.search ON index simple TO ericriak-admin security grant yokozuna.admin ON index TO admin

require 'riak'client = Riak::Client.new

# create indexclient.create_search_index('simple')

# tie the index to bucket 'cats'bucket = Riak::Bucket.new(client, 'cats')bucket.props = { yz_index: 'simple' }

Indexing DatatypesTuesday, November 12, 13

UTF8中搜索UTF8חיפוש בΑναζήτηση σε UTF8

Искать в UTF8

Search in UTF8

Advanced Searchaka. Next Level Search

solr = RSolr.connect(url: 'http://yokozuna01.bos1:8098/solr/docs')

resp = solr.get('select', params: {q: '*:*'})

Facets, Stats, and stuff

eDisMax

defType=edismax

Highlighting

Languages

Map/ReduceTuesday, November 12, 13

{"inputs": { "module":"yokozuna", "function":"mapred_search", "arg":["docs","title_s:Key* AND language_s:en"] }, "query":[ {"map":{ "language":"javascript", "keep":false, "source":"function(v) { return [1]; }"}}, {"reduce"{ "language":"javascript", "keep":true, "name":"Riak.reduceSum" }} ]}

client.create_search_index('simple')

bucket = Riak::Bucket.new(client, 'people')bucket.props = { yz_index: 'simple' }

bucket = Riak::Bucket.new(client, 'cats')bucket.props = { yz_index: 'simple' }

bucket = Riak::Bucket.new(client, 'mountains')bucket.props = { yz_index: 'simple' }

Schemas

Fallen Nodes

ThermoclineTuesday, November 12, 13

Thermocline of ReplicationTuesday, November 12, 13

GoalsTuesday, November 12, 13

But... why?Tuesday, November 12, 13

Yokozuna = GlueTuesday, November 12, 13

Extractorsyz_extractor.erlyz_json_extractor.erlyz_noop_extractor.erlyz_text_extractor.erlyz_xml_extractor.erlyz_doc.erl

Interfaces

yz_pb_admin.erlyz_pb_search.erlyz_schema.erlyz_wm_extract.erlyz_wm_index.erlyz_wm_schema.erlyz_wm_search.erl

SecurityTuesday, November 12, 13

Permissions

riak-admin security grant yokozuna.admin ON index TO admin

riak-admin security grant yokozuna.search ON index TO user

riak-admin security grant yokozuna.admin ON schema TO admin

riak-admin security grant yokozuna.search ON index wiki TO user

Administrationyz_pb_admin.erlyz_schema.erlyz_wm_index.erlyz_wm_schema.erl

Searchesyz_pb_search.erlyz_wm_search.erl

Backend Magicyokozuna.erlyz_app.erlyz_general_sup.erlyz_kv.erlyz_misc.erlyz_sup.erl

Distribution

yz_cover.erlyz_events.erl

AAEyz_entropy.erlyz_entropy_mgr.erlyz_exchange_fsm.erlyz_index.erlyz_index_hashtree.erlyz_index_hashtree_sup.erl

Solr code

EntropyData.javaMonitor.javayz_solr.erlyz_solr_proc.erlyz_solr_sup.erl

yz_stat.erlyz_stat_worker.erl

“Testing only showsthe presence, not

the absence,of bugs”

aae_test.erlyokozuna_essential.erlyz_errors.erlyz_fallback.erlyz_flag_transitions.erlyz_index_admin.erlyz_languages.erlyz_mapreduce.erlyz_monitor_solr.erlyz_pb.erlyz_rs_migration.erlyz_rt.erlyz_schema_admin.erlyz_siblings.erlyz_solr_start_timeout.erlyz_stat_test.erlyz_wm_extract_test.erlyz_component_tests.erlyz_json_extractor_tests.erlyz_kv_tests.erlyz_misc_tests.erlyz_text_extractor_tests.erlyz_xml_extractor_tests.erlyz_driver.erlyz_file_terms.erl

Dijkstra

We Want More

• CRDT Support

• Simplify custom extractors

• Bucket Types (?)

• Users

Thanks

Eric Redmond@coderoshi

riak search 2: yokozuna

Technology

riak and ruby

riak handbookjohnchukwuma.com/training/riak...

introduction to riak (ian plosker)

riak a successful failure

riak intro to

riak & wooga_geeek2geeek meetup2014 berlin

timeseries data in riak - riak meetup stockholm 1/11/2012

node riak frank06

riak at shareaholic

yokozuna, distributed search you don't think about

riak intro

node.js + riak

riak perf wins

riak at ideeli

the future of distributed search in riak basho...

comparison between dynamo and riak

putting together the platform: riak, redis, solr and...

riak from small to large

raspberry pi cluster running riak - 33rd...

yokozuna bath*party enjoy your happy bathtimes. : 4979508...