yokozuna, distributed search you don't think about

YokozunaDistributed Search You Don’t Think About

Ryan Zezeski May 14th 20131Tuesday, May 21, 13

Live Demo

2Tuesday, May 21, 13

Live Demos


PROBLEM?


PROBLEM?

SOLUTION!


Solution Pre-made


Piece At A Time


Goals• Don’t screw up

• Show how Yokozuna doesn’t make you think (too hard)

• Teach you about Search

• Neat things you can do with Yokozuna


PROBLEM: SEARCH FOR COMMITS

ABOUT SPECIFIC FEATURE/BUG

MAKE IT GOOGLE-LIKE


SOLUTION: INDEX COMMITS IN YOKOZUNA -

“COMMIT LOG SEARCHER” (CLS)


Anatomy of a Commit Msg


Primary Key


Any Node Will Do


Term Query


Query Any Node


Boolean (1)repo:riak_kv repo:riak_core


Boolean (2)repo:riak_kv AND author:”Ryan Zezeski”


Boolean (3)commit_author:"Ryan Zezeski" OR

commit_author:"Joseph Blomstedt" NOT commit_repo:riak_kv


Range (1)commit_repo:riak_core

AND commit_dt:[NOW-1YEAR TO NOW]


Range (2)commit_repo:riak_core

AND commit_dt:[NOW-1YEAR TO NOW]

I RAN THIS ON 2013-05-10

sort=dt asc


Wildcard (1)*:*

GET TOTAL COUNT FIRST


Wildcard (2)commit_repo:riak_*

NOTICE COUNT IS LOWER


WHAT ABOUT SEARCHING

SUMMARY AND BODY?


THE INVERTED INDEX


AN INDEX - BUT INVERTED


EVERYONE KNOWS WHAT IT IS


EVEN NON-TECH PEOPLE


YES...EVEN YOUR PARENTS


What’s In A Book?


• WORDS

• PARAGRAPHS

• SECTIONS

• CHAPTERS

• ETC.


AND PAGE NUMBERS


PAGE NUMBERS ARE AN IMPLICIT INDEX


PAGE NUMBER TO WORDS

WORD TO PAGE NUMBERS

INVERTED


STOLEN FROM BLOG OF RICKY HO: http://horicky.blogspot.com/2013/02/text-processing-

part-2-inverted-index.html


http://horicky.blogspot.com/2013/02/text-processing-part-2-inverted-index.html






HOW DO YOU GET THE WORDS IN THE

FIRST PLACE?


Analysis - The Iceberg That Sunk The Titanic


Phrase (1)subject:hinted OR subject:handoff OR body:hinted OR body:handoff


Phrase (2)subject:”hinted handoff” OR body:”hinted handoff”


Phrase (3)subject:”partition vnode” OR body:”partition vnode”


Phrase (4)subject:”partition vnode”~4 OR body:”partition vnode”~4


Exact Termsubject:behavior OR body:behavior


Fuzzy Termsubject:behavior~1 OR body:behavior~1


Ranking

ADD SCORE TO FL

SCORE ADDED TO EVERY RESULT


RECALL, PRECISION, AND RELEVANCY, OH

MY!


RELEVANCY - FOR A GIVEN QUERY &

DOC SET THERE IS AN IDEAL ANSWER OF ONLY RELEVANT

DOCS


RECALL = WHAT % OF IDEAL ANSWER SET WAS RETRIEVED


PRECISION = WHAT % OF ANSWER IS

RELEVANT


RECALLvs.PRECISION AS YOU INCREASE

RECALL YOU DEGRADE PRECISION


SOLR DETERMINES RELEVANCY VIA THE

NOTION OF SIMILARITY


SOLR USES TF-IDF: TERM FREQUENCY,

INVERSE DOCUMENT FREQUENCY


Dismax + Facets + Highlighting

FACETSHIGHLIGHTING

DISMAX


FACET - A TAXONOMY OF

YOUR QUERY BASED ON FIELD’S VALUES


FACETS ALLOW “DRILL DOWN” - THEY GUIDE THE

USER


HIGHLIGHTING GIVES YOUR RESULTS CONTEXT - ALLOWS

QUICKER DETERMINATION OF

RELEVANCY


DISMAX - DISjunction MAX - A QUERY

HANDLER MEANT FOR DIRECT USER

INPUT


All Nodes Up


All Nodes Up - Query


Node 4 Down


Node 4 Down - Query


Node 3 & 4 Down


Node 3 & 4 Down - Query


REPLICATION PROVIDES HIGH

AVAILABILITY

2 3 41

START WITH 4 NODES


Write 3 Replicas

2 3 41


Take 2 Nodes Down

2 3 41

1 REPLICA STILL AVAILABLE


WHAT IF DATA IS WRITTEN WHILE

NODES ARE DOWN?


YZ Not Stored Yet


Store YZ Log


Query YZ - Node 1& 2


Set XFer Limit To 0


Start Nodes 3 & 4


Query Solr Direct

WHEN MAKING THIS DEMO I WAS

EXPECTING THIS TO BE 0 BUT I FORGOT ABOUT

AAE WHICH STARTED KICKING IN BEFORE

HANDOFF - SELF HEALING FTW!


Set Xfer Limit To 64


Handoff Occurs


0 Pending Xfers


Solr Direct (Again)

NOTICE IT’S NOW 301, UP FROM 54, MORE

PROOF THAT HANDOFF OCCURRED - NOTE

THIS QUERY IS GOING DIRECT TO ONLY 1

SHARD


Query Node 4 YZ

NOW HIT YOKOZUNA ON NODE4 (NOTICE CHANGE IN PORT #) - THIS WILL RUN A DIST

SEARCH AND THUS RETURN CORRECT

COUNT


Data Ownership

A VNODE

THE RING


Node DownX

X

X

XXX

X

X

XX


Write FallbackX

X

X

XXX

X

X

XX


Node Up

HINTED HANDOFF WILL MOVE REPLICA

TO PRIMARY


WHAT IF YOU RM -RF THE INDEX DIR?


Kill The Data

RM -RF THE INDEX DIRECTORY

KILL THE SOLR PROC


Auto Restart

YOKOZUNA NOTICES SOLR DIED AND AUTOMATICALLY RESTARTS

IT


Node 4 - 0 Results


AAE Notices Missing Data


Node 4 - 13 Results

DATA IS RE-INDEXED OVER TIME


More AAE Repair


Node 4 - 128 Results

MORE INDEXES ARE REPAIRED, THIS CONTINUES UNTIL AAE

REPAIRS ALL INDEXES


WHAT EVEN IS ACTIVE ANTI-

ENTROPY?


Mo Systems Mo Failure

• index update could get lost

• files can become truncated/corrupted

• accidental `rm -rf`

• segfault at right time

• etc...


MYRAID OF FAILURE SCENARIOS - FROM

OBVIOUS TO NEARLY INVISIBLE


ENTROPY IS DAMAGE

AAE IS SELF HEALING

STRIKER!!!! EHEM, I MEAN, ENTROPY!!!!


REPAIR EFFICIENTLY - NOT STUPIDLY


Learn You Some Merkle For A Great Good

BIG UPS TO @jtuple FOR THE AAE DIAGRAMS


SegmentsEACH SEGMENT IS LIST OF

KEY-HASH PAIRS


Segment HashesHASH OF HASHES IN

SEGMENT


Hash O’ Hashes


WHAT HAPPENS DURING

EXCHANGE?


Start With 2 Trees


Compare Top Hashes

TOP HASHES DON’T MATCH -

SOMETHING IS DIFFERENT


Compare Child Hashes

NARROW DOWN THE DIVERGENT

SEGMENT


Recur

NARROW DOWN THE DIVERGENT

SEGMENT CONT...


Iter Key-Hash Pairs

ITER FINAL LIST OF HASHES TO FIND DIVERGENT KEYS


Repair Divergent Keys

REPAIR (RE-INDEX) KEYS THAT ARE

DIVERGENT (RED)


CODE FOR DETECTION AND

REPAIR - NOT PREVENTION


WHAT HAPPENS IF 3 NODES GO DOWN?


Stop 3 Nodes


Query


CONSISTENCY vs.

AVAILABILITY


Uptime - Story of 9s

UPTIME = (MTBF - MTTR) / MTBF


Uptime is Flawed

IF THE SYSTEM IS DOWN, BUT NO

ONE MAKES A REQUEST, IS IT

REALLY DOWN?112Tuesday, May 21, 13

Yield - Uptime of the People

YIELD = QUERIES COMPLETED / QUERIES OFFERED


Harvest vs. YieldHARVEST = DATA AVAIL / COMPLETE DATA

IF FACE OF FAILURE YOU CAN’T HAVE BOTH FOR A SINGLE REQUEST


IN TIMES OF TROUBLE -

YOKOZUNA CHOOSES HARVEST

FOR QUERIES115Tuesday, May 21, 13

TECHNICALLY - YOKOZUNA IS ALWAYS < 100%

HARVEST IN A NON-QUIESCENT CLUSTER


YOKOZUNA FAVORS YIELD FOR WRITES


ONCE RIAK 1.4 SHIPS - YOKOZUNA LANDS

IN MASTER


THANK YOU

HTTP://GITHUB.COM/BASHO/YOKOZUNA




yokozuna, distributed search you don't think about

Technology

query node

kv repo

hinted handoffor body

partition vnodeor body

queryyz node

query solr directwhen

agiven query doc set

hinted handoff39tuesday