adam fuchs' accumulo talk at nosql now! 2013

Securely explore your data

SQRRL ENTERPRISE +

APACHE ACCUMULO:

A secure, scalable, real-time analysis framework

Adam Fuchs, CTO

Sqrrl Data, Inc.

August 21, 2013

© 2013 Sqrrl | All Rights Reserved | Proprietary and Confidential

OUTLINE

Two Halves of “Real-Time”

Accumulo and Sqrrl Technology

Data-Centric Security

Table Designs

Performance Benchmarks


TWO HALVES OF REAL-TIME

Real-Time reduce event to reaction time Real-Time reduce ingest to query latency

Data-Driven Query-Driven


1. SPE queries NoSQL to enrich streaming data

2. SPE persists results in NoSQL for future query

3. SPE takes action automatically

4. SPE issues data-driven alerts

5. Sqrrl provides context for dashboards

6. Analysis tools query use Sqrrl to search and manipulate historical data

Data-Driven + Query-Driven Real-Time Ecosystem

Data

NoSQL+

SPE

Dashboards

Actions

InteractiveAnalysis Tools(Discovery + Forensics)

1 2

3

5

4

6

© 2013 Sqrrl | All Rights Reserved | Proprietary and Confidential © 2013 Sqrrl | All Rights Reserved | Proprietary and Confidential 5

This talk focuses on the database.

Dashboards

InteractiveAnalysis Tools(Discovery + Forensics)

1. SPE queries NoSQL to enrich streaming data2. SPE persists results in NoSQL for future query3. SPE takes action automatically4. SPE issues data-driven alerts5. Sqrrl provides context for dashboards6. Analysis tools query use Sqrrl to search and manipulate historical data

Data

Actions

SPE4

3

NoSQL+6

5

21


OUTLINE




Table Designs



ACCUMULO DATA FORMAT

© 2013 Sqrrl | All Rights Reserved | Proprietary and Confidential 7

Accumulo Key/Value Example

An Accumulo key is a 5-tuple, consisting of:

- Row: Controls Atomicity- Column Family: Controls Locality - Column Qualifier: Controls Uniqueness- Visibility Label: Controls Access- Timestamp: Controls Versioning


ACCUMULO TABLETS

Collections of KV pairs form Tables

Tables are partitioned into Tablets

Metadata tablets hold info about other tablets, forming a 3-level hierarchy

A Tablet is a unit of work for a Tablet Server


Root Tablet-∞ to ∞

Metadata Tablet 1-∞ to “Encyclopedia:Ocelot”

Data Tablet-∞ : thing

Data Tabletthing : ∞

Data Tablet-∞ : Ocelot

Data TabletOcelot : Yak

Data TabletYak : ∞

Data Tablet-∞ to ∞

Metadata Tablet 2 “Encyclopedia:Ocelot” to ∞

Well-Known Location

(zookeeper)

Table: Adam’s Table Table: Encyclopedia Table: Foo


ACCUMULO PROCESSES


Tablet Server

Tablet

Tablet Server

Tablet

Tablet Server

Tablet

Application

Zookeeper

Zookeeper

Zookeeper

Master

HDFS

Read/Write

Store/Replicate

Assign/Balance

Delegate Authority

Application

Application


TABLET DATA FLOW


In-Memory Map

Write AheadLog

(For Recovery)

Sorted, Indexed

File

Sorted, Indexed

File

Sorted, Indexed

File

Tablet

ReadsIterator

TreeMinor

Compaction

Merging / Major Compaction

Iterator Tree

Writes Iterator Tree

Scan


WORD COUNT:


Summing Aggregating Iterator

Input Corpus


ITERATOR FRAMEWORK


Iterator Operations:

- File Reads- Block Caching- Merging- Deletion- Isolation- Locality Groups- Range Selection- Column Selection- Cell-level Security- Versioning- Filtering- Aggregation- Partitioned Joins


ACCUMULO LATENCIES


Ingesters QueriersTablet Servers

Input BatchWriter

In-Memory

Map

ScanIterators

Scanner/Batch

Scanner

In-Memory

Map

RFile

Compaction

Iterators

ScanIterators

RFile

Compaction

Iterators

In-Memory

Map

RFiles

CompactionIterators

ScanIterators

Output

~ms~ms ~ms

ms

- m

in


ACCUMULO THROUGHPUT


Ingesters QueriersTablet Servers

Input BatchWriter

In-Memory

Map

ScanIterators

Scanner/Batch

Scanner

In-Memory

Map

RFile

Compaction

Iterators

ScanIterators

RFile

Compaction

Iterators

In-Memory

Map

RFiles

CompactionIterators

ScanIterators

Output

~ms~ms ~ms

ms

- m

in

Read-Modify-Write Latency: ~ms

>1K entries/s challenging with R-M-W

Ingest:up to 500K entries/s

per node

Scan:up to 1M entries/s

per node


SQRRL ENTERPRISE


Built on Apache Accumulo

Sqrrl Server

Sqrrl API over Apache Thrift RPC(JSON, Graph, Aggregation, Search, etc.)

• Sqrrl proprietary• Automated indexing• Custom iterators• Lucene integration• Security extensions Accumulo RPC

(Sorted Key/Value I/O)

Hadoop RPC(File I/O)

• Open source (including Sqrrl contributions)

• Open source or commercial distributions

Graph + Document I/O

Exploratory / Operational Apps

Bulk Processing Integration


OUTLINE




Table Designs




DATA-CENTRIC SECURITY


Definition: Data carries with it information that is required to make policy decisions on its releasability.

User 1 User 2Sqrrl/

Accumulo


SECURITY


Example Accumulo Key/Value Pairs

Accumulo is the only NoSQL database with cell-level access controls


DATA-CENTRIC SECURITY ECOSYSTEM


Data Labeler Sqrrl Enterprise

Apps

User Attributes

Audits

Policies

End Users

Auth. Service

Policy Engine

Key Mgmt


OUTLINE




Table Designs




HIERARCHICAL DECOMPOSITION


Row:

Column Family:

Column Qualifier:

Value:

<person>

attribute purchases

age

<age>

discount

<cost>

sneakers

<rate>

returns

hat

<cost>


MATERIALIZED TABLE


Row: george

attribute purchases

age

27 $83

sneakers

bill

attribute purchases

40%

sneakers

$100

discount

49

age

Key/Value Pair

Column Family:

Column Qualifier:

Value:


FORWARD AND INVERTED INDEX


Table:

Row:

Column Family:

Value:

Forward Index

<UUID>

<Type>

<Field>

<Term>

Inverted Index

<Term>

<UUID>

<Type+Field>

<Digest of Event>

Column Qualifier:


FORWARD AND INVERTED INDEX



CUSTOM INDEXING


Table:

Row:

Geo Index

<GeoHash>

<Event Type>

<UUID>

<Digest of Event>

Latitude10110101001

Longitude00111010010

101001110111010101011100001011100

Depth11010110110

Column Family:

Column Qualifier:

Value:


D4M 2.0 SCHEMA FOR TWITTER DATA


Table:

Row:

Column Family:

Tedge

<UUID>

“stat”

<stat>

“1”

“time”

<time>

“1”

“user”

<user>

“1”

“word”

<word>

“1”

TedgeT

<value>

“stat”

<UUID>

“1”

“time”

<UUID>

“1”

“user”

<UUID>

“1”

“word”

<UUID>

“1”

Column Qualifier:

Value:




Table:

Row:

Column Family:

TedgeDegT

<value>

“stat”

“degree”

<count>

“time”

“degree”

<count>

“user”

“degree”

<count>

“word”

“degree”

<count>

Ttext

<UUID>

Column Qualifier:

Value:

“text”

-

<text>




Source: D4M 2.0 Schema: A General Purpose High Performance Schema for the Accumulo Database , Kepner et. al., HPEC 2013


OUTLINE




Table Designs




ACCUMULO WITH D4M 2.0 SCHEMA PERFORMANCE


Source: D4M 2.0 Schema: A General Purpose High Performance Schema for the Accumulo Database , Kepner et. al., HPEC 2013

Maximizing throughput on an 8-node, 192-core cluster:


ACCUMULO SCALABILITY: GRAPH500 BENCHMARK


source: http://www.pdl.cmu.edu/SDI/2013/slides/big_graph_nsa_rd_2013_56002v1.pdf


ATOMIC INCREMENT PERFORMANCE COMPARISON


Read/Modify/Write (HBase) vs. Iterators/Combiners (Accumulo)


QUESTIONS?


Adam Fuchs, CTOSqrrl Data, Inc.

adam fuchs' accumulo talk at nosql now! 2013

Technology

cto sqrrl data

encyclopedia data tablet

foo data tablet

accumulo data format

historical data

halves of realtime data

adams table data tablet

streaming data spe