yokozuna, distributed search you don't think about

YokozunaDistributed Search You Don’t Think About

Ryan Zezeski May 14th 20131Tuesday, May 21, 13

Live Demo

2Tuesday, May 21, 13

Live Demos

3Tuesday, May 21, 13

PROBLEM?

4Tuesday, May 21, 13

PROBLEM?

SOLUTION!

5Tuesday, May 21, 13

Solution Pre-made

6Tuesday, May 21, 13

Piece At A Time

7Tuesday, May 21, 13

Goals• Don’t screw up

• Show how Yokozuna doesn’t make you think (too hard)

• Teach you about Search

• Neat things you can do with Yokozuna

8Tuesday, May 21, 13

PROBLEM: SEARCH FOR COMMITS

ABOUT SPECIFIC FEATURE/BUG

MAKE IT GOOGLE-LIKE

9Tuesday, May 21, 13

SOLUTION: INDEX COMMITS IN YOKOZUNA -

“COMMIT LOG SEARCHER” (CLS)

10Tuesday, May 21, 13

Anatomy of a Commit Msg

11Tuesday, May 21, 13

Primary Key

12Tuesday, May 21, 13

Any Node Will Do

13Tuesday, May 21, 13

Term Query

14Tuesday, May 21, 13

Query Any Node

15Tuesday, May 21, 13

Boolean (1)repo:riak_kv repo:riak_core

16Tuesday, May 21, 13

Boolean (2)repo:riak_kv AND author:”Ryan Zezeski”

17Tuesday, May 21, 13

Boolean (3)commit_author:"Ryan Zezeski" OR

commit_author:"Joseph Blomstedt" NOT commit_repo:riak_kv

18Tuesday, May 21, 13

Range (1)commit_repo:riak_core

AND commit_dt:[NOW-1YEAR TO NOW]

19Tuesday, May 21, 13

Range (2)commit_repo:riak_core

AND commit_dt:[NOW-1YEAR TO NOW]

I RAN THIS ON 2013-05-10

sort=dt asc

20Tuesday, May 21, 13

Wildcard (1)*:*

GET TOTAL COUNT FIRST

21Tuesday, May 21, 13

Wildcard (2)commit_repo:riak_*

NOTICE COUNT IS LOWER

22Tuesday, May 21, 13

WHAT ABOUT SEARCHING

SUMMARY AND BODY?

23Tuesday, May 21, 13

THE INVERTED INDEX

24Tuesday, May 21, 13

AN INDEX - BUT INVERTED

25Tuesday, May 21, 13

EVERYONE KNOWS WHAT IT IS

26Tuesday, May 21, 13

EVEN NON-TECH PEOPLE

27Tuesday, May 21, 13

YES...EVEN YOUR PARENTS

28Tuesday, May 21, 13

What’s In A Book?

29Tuesday, May 21, 13

• WORDS

• PARAGRAPHS

• SECTIONS

• CHAPTERS

• ETC.

30Tuesday, May 21, 13

AND PAGE NUMBERS

31Tuesday, May 21, 13

32Tuesday, May 21, 13

PAGE NUMBERS ARE AN IMPLICIT INDEX

33Tuesday, May 21, 13

PAGE NUMBER TO WORDS

WORD TO PAGE NUMBERS

INVERTED

34Tuesday, May 21, 13

STOLEN FROM BLOG OF RICKY HO: http://horicky.blogspot.com/2013/02/text-processing-

part-2-inverted-index.html

35Tuesday, May 21, 13

HOW DO YOU GET THE WORDS IN THE

FIRST PLACE?

36Tuesday, May 21, 13

Analysis - The Iceberg That Sunk The Titanic

37Tuesday, May 21, 13

Phrase (1)subject:hinted OR subject:handoff OR body:hinted OR body:handoff

38Tuesday, May 21, 13

Phrase (2)subject:”hinted handoff” OR body:”hinted handoff”

39Tuesday, May 21, 13

Phrase (3)subject:”partition vnode” OR body:”partition vnode”

40Tuesday, May 21, 13

Phrase (4)subject:”partition vnode”~4 OR body:”partition vnode”~4

41Tuesday, May 21, 13

Exact Termsubject:behavior OR body:behavior

42Tuesday, May 21, 13

Fuzzy Termsubject:behavior~1 OR body:behavior~1

43Tuesday, May 21, 13

Ranking

ADD SCORE TO FL

SCORE ADDED TO EVERY RESULT

44Tuesday, May 21, 13

RECALL, PRECISION, AND RELEVANCY, OH

MY!

45Tuesday, May 21, 13

RELEVANCY - FOR A GIVEN QUERY &

DOC SET THERE IS AN IDEAL ANSWER OF ONLY RELEVANT

DOCS

46Tuesday, May 21, 13

RECALL = WHAT % OF IDEAL ANSWER SET WAS RETRIEVED

47Tuesday, May 21, 13

PRECISION = WHAT % OF ANSWER IS

RELEVANT

48Tuesday, May 21, 13

RECALLvs.PRECISION AS YOU INCREASE

RECALL YOU DEGRADE PRECISION

49Tuesday, May 21, 13

SOLR DETERMINES RELEVANCY VIA THE

NOTION OF SIMILARITY

50Tuesday, May 21, 13

SOLR USES TF-IDF: TERM FREQUENCY,

INVERSE DOCUMENT FREQUENCY

51Tuesday, May 21, 13

Dismax + Facets + Highlighting

FACETSHIGHLIGHTING

DISMAX

52Tuesday, May 21, 13

FACET - A TAXONOMY OF

YOUR QUERY BASED ON FIELD’S VALUES

53Tuesday, May 21, 13

FACETS ALLOW “DRILL DOWN” - THEY GUIDE THE

USER

54Tuesday, May 21, 13

HIGHLIGHTING GIVES YOUR RESULTS CONTEXT - ALLOWS

QUICKER DETERMINATION OF

RELEVANCY

55Tuesday, May 21, 13

DISMAX - DISjunction MAX - A QUERY

HANDLER MEANT FOR DIRECT USER

INPUT

56Tuesday, May 21, 13

All Nodes Up

57Tuesday, May 21, 13

All Nodes Up - Query

58Tuesday, May 21, 13

Node 4 Down

59Tuesday, May 21, 13

Node 4 Down - Query

60Tuesday, May 21, 13

Node 3 & 4 Down

61Tuesday, May 21, 13

Node 3 & 4 Down - Query

62Tuesday, May 21, 13

REPLICATION PROVIDES HIGH

AVAILABILITY

2 3 41

START WITH 4 NODES

63Tuesday, May 21, 13

Write 3 Replicas

2 3 41

64Tuesday, May 21, 13

Take 2 Nodes Down

2 3 41

1 REPLICA STILL AVAILABLE

65Tuesday, May 21, 13

WHAT IF DATA IS WRITTEN WHILE

NODES ARE DOWN?

66Tuesday, May 21, 13

YZ Not Stored Yet

67Tuesday, May 21, 13

Store YZ Log

68Tuesday, May 21, 13

Query YZ - Node 1& 2

69Tuesday, May 21, 13

Set XFer Limit To 0

70Tuesday, May 21, 13

Start Nodes 3 & 4

71Tuesday, May 21, 13

Query Solr Direct

WHEN MAKING THIS DEMO I WAS

EXPECTING THIS TO BE 0 BUT I FORGOT ABOUT

AAE WHICH STARTED KICKING IN BEFORE

HANDOFF - SELF HEALING FTW!

72Tuesday, May 21, 13

Set Xfer Limit To 64

73Tuesday, May 21, 13

Handoff Occurs

74Tuesday, May 21, 13

0 Pending Xfers

75Tuesday, May 21, 13

Solr Direct (Again)

NOTICE IT’S NOW 301, UP FROM 54, MORE

PROOF THAT HANDOFF OCCURRED - NOTE

THIS QUERY IS GOING DIRECT TO ONLY 1

SHARD

76Tuesday, May 21, 13

Query Node 4 YZ

NOW HIT YOKOZUNA ON NODE4 (NOTICE CHANGE IN PORT #) - THIS WILL RUN A DIST

SEARCH AND THUS RETURN CORRECT

COUNT

77Tuesday, May 21, 13

Data Ownership

A VNODE

THE RING

78Tuesday, May 21, 13

Node DownX

X

X

XXX

X

X

XX

79Tuesday, May 21, 13

Write FallbackX

X

X

XXX

X

X

XX

80Tuesday, May 21, 13

Node Up

HINTED HANDOFF WILL MOVE REPLICA

TO PRIMARY

81Tuesday, May 21, 13

WHAT IF YOU RM -RF THE INDEX DIR?

82Tuesday, May 21, 13

Kill The Data

RM -RF THE INDEX DIRECTORY

KILL THE SOLR PROC

83Tuesday, May 21, 13

Auto Restart

YOKOZUNA NOTICES SOLR DIED AND AUTOMATICALLY RESTARTS

IT

84Tuesday, May 21, 13

Node 4 - 0 Results

85Tuesday, May 21, 13

AAE Notices Missing Data

86Tuesday, May 21, 13

Node 4 - 13 Results

DATA IS RE-INDEXED OVER TIME

87Tuesday, May 21, 13

More AAE Repair

88Tuesday, May 21, 13

Node 4 - 128 Results

MORE INDEXES ARE REPAIRED, THIS CONTINUES UNTIL AAE

REPAIRS ALL INDEXES

89Tuesday, May 21, 13

WHAT EVEN IS ACTIVE ANTI-

ENTROPY?

90Tuesday, May 21, 13

Mo Systems Mo Failure

• index update could get lost

• files can become truncated/corrupted

• accidental `rm -rf`

• segfault at right time

• etc...

91Tuesday, May 21, 13

MYRAID OF FAILURE SCENARIOS - FROM

OBVIOUS TO NEARLY INVISIBLE

92Tuesday, May 21, 13

ENTROPY IS DAMAGE

AAE IS SELF HEALING

STRIKER!!!! EHEM, I MEAN, ENTROPY!!!!

93Tuesday, May 21, 13

REPAIR EFFICIENTLY - NOT STUPIDLY

94Tuesday, May 21, 13

Learn You Some Merkle For A Great Good

BIG UPS TO @jtuple FOR THE AAE DIAGRAMS

95Tuesday, May 21, 13

SegmentsEACH SEGMENT IS LIST OF

KEY-HASH PAIRS

96Tuesday, May 21, 13

Segment HashesHASH OF HASHES IN

SEGMENT

97Tuesday, May 21, 13

Hash O’ Hashes

98Tuesday, May 21, 13

WHAT HAPPENS DURING

EXCHANGE?

99Tuesday, May 21, 13

Start With 2 Trees

100Tuesday, May 21, 13

Compare Top Hashes

TOP HASHES DON’T MATCH -

SOMETHING IS DIFFERENT

101Tuesday, May 21, 13

Compare Child Hashes

NARROW DOWN THE DIVERGENT

SEGMENT

102Tuesday, May 21, 13

Recur

NARROW DOWN THE DIVERGENT

SEGMENT CONT...

103Tuesday, May 21, 13

Iter Key-Hash Pairs

ITER FINAL LIST OF HASHES TO FIND DIVERGENT KEYS

104Tuesday, May 21, 13

Repair Divergent Keys

REPAIR (RE-INDEX) KEYS THAT ARE

DIVERGENT (RED)

105Tuesday, May 21, 13

CODE FOR DETECTION AND

REPAIR - NOT PREVENTION

106Tuesday, May 21, 13

WHAT HAPPENS IF 3 NODES GO DOWN?

107Tuesday, May 21, 13

Stop 3 Nodes

108Tuesday, May 21, 13

Query

109Tuesday, May 21, 13

CONSISTENCY vs.

AVAILABILITY

110Tuesday, May 21, 13

Uptime - Story of 9s

UPTIME = (MTBF - MTTR) / MTBF

111Tuesday, May 21, 13

Uptime is Flawed

IF THE SYSTEM IS DOWN, BUT NO

ONE MAKES A REQUEST, IS IT

REALLY DOWN?112Tuesday, May 21, 13

Yield - Uptime of the People

YIELD = QUERIES COMPLETED / QUERIES OFFERED

113Tuesday, May 21, 13

Harvest vs. YieldHARVEST = DATA AVAIL / COMPLETE DATA

IF FACE OF FAILURE YOU CAN’T HAVE BOTH FOR A SINGLE REQUEST

114Tuesday, May 21, 13

IN TIMES OF TROUBLE -

YOKOZUNA CHOOSES HARVEST

FOR QUERIES115Tuesday, May 21, 13

TECHNICALLY - YOKOZUNA IS ALWAYS < 100%

HARVEST IN A NON-QUIESCENT CLUSTER

116Tuesday, May 21, 13

YOKOZUNA FAVORS YIELD FOR WRITES

117Tuesday, May 21, 13

ONCE RIAK 1.4 SHIPS - YOKOZUNA LANDS

IN MASTER

118Tuesday, May 21, 13

THANK YOU

HTTP://GITHUB.COM/BASHO/YOKOZUNA

119Tuesday, May 21, 13