the perils and triumphs of using cassandra at a .net/microsoft shop

reimagining the business of apps

©2013 NativeX Holdings, LLC

The Perils and Triumphs of using Cassandra at a .NET/Microsoft Shop

About the PresentersJeff Smoley – Sr. Infrastructure Architect

Derek Bromenshenkel – Infrastructure Architect

Agenda• About NativeX

• Why Cassandra?

• Challenges

• Auto Id Generation

• FluentCassandra

• Hector

• IKVM.NET

• HectorNet

• Reporting Integration

• Data Modeling

• Lessons Learned


About NativeX• Formerly W3i

• Home Office in Sartell, MN

• 75 miles NW of Minneapolis

• Remote Offices in MSP and SF

• ~120 Employees


What NativeX Does• Marketing technology

platform that enables developers to build successful business around their apps.

• We provide Publishers with a way to monetize and Advertisers with a way to gain distribution through a native ad experience.

Mobile Vanity Metrics• Over 1B unique devices

• 1000s of Apps

• > 135M Monthly Active Users

• > 200GB of data ingest per week


• Why Cassandra?

• Challenges


• FluentCassandra

• Hector

• IKVM.NET

• HectorNet


• Data Modeling

• Lessons Learned


Backstory• In early 2012 realized

infrastructure needed to change to support high growth business.

• From 100M session/quarter to 6B+.

2011 Q4

2012 Q1

2012 Q2

2012 Q3

2012 Q4

2013 Q1

2013 Q2

2013 Q3

0

1,000,000,000

2,000,000,000

3,000,000,000

4,000,000,000

5,000,000,000

6,000,000,000

7,000,000,000

API Requests


Original OLTP Architecture• Microsoft SQL Server

• 2 Node Cluster (Failover)

• 12 cores / node

• 192 GB mem / node• Compellent SAN

• 172 Tiered Disk

• SSD, FC, SATA


Objectives

Scale

• Horizontal• Incremental cost

structure

Resiliency

• No single point of failure

• Geographically distributed


What is NoSQL• Stands for Not Only SQL.

• The NoSQL movement is about understanding problems and focusing on solutions.

• It’s not about silver bullets and black boxes.

• It is about using the right tool for the right problem.


Researched Products• Compared features like:

• Distributed / Shared Nothing

• Multi-Cluster Support

• Maturity & Popularity

• Documentation

• .NET Support


Selecting CassandraDB Distributed Maturity High Availability Style Documentation Native Language Drivers Popularity

MongoDB Yes Medium Yes Document - NoSQL Excellent Major Languages High

VoltDB Yes Low Yes RDBMS - SQL Good Major Languages Low

MySQL Cluster Yes High Yes RDBMS - SQL & Key/Value Excellent Major Languages Medium

MySQL ScaleDB Yes Low Yes RDBMS - SQL Good Major Languages Low

Cassandra Yes Medium Yes Key/Value - Column Family Excellent Major; Poor .Net High

CouchDB No Medium Yes Document - NoSQL ? No - REST only Medium

RavenDB Yes? Low No Document - NoSQL Poor C#, JS, REST Medium

Couchbase Yes Medium Yes Key/Value - Document Good Major Languages Medium

*Disclaimer, this data was complied in spring of 2012 and my not reflect the current state of each database system shown here.

http://nosql.mypopescu.com/ is a helpful site for discovering and learning about different DB Systems.

http://nosql.mypopescu.com/

http://nosql.mypopescu.com/


Top Choices• MySQL Cluster

• Relational and very familiar.

• Has physical row limitations.

• MongoDB

• Data modeling was simpler than C*.

• Not very clear if it had multi-cluster support.

• Cassandra

• At the very core it’s all about scalability and resiliency.

• Data modeling a little scary, immature .Net support.


Why Cassandra?• Multi-node

• Multi-cluster

• Highly Available

• Durable

• Shared Nothing

• Tunable Consistency


Cassandra at NativeX• C* was not a replacement DB system.

• We continue to use MS SQL Server alongside C*.

• SQL Server used for storing configuration data.

• C* solves a very specific problem for us.

• Writing large volumes of data quickly.

• Reading very specific data out of a large record set.

Challenges• C* does not have Auto Id generation.

• How to connect to C* with C#?

• Finding a connector with good Failure Tolerance.

• How to integrate our reporting system?


Auto ID Generation• Pre-existing requirements

• Unique, 64-bit positive integers

• Increasing (sortable) a plus

• Previously SQL Server Identity column

• A Time-based UUID is sortable and unique

• Changed everything we could

• The future for us


Auto ID – What are the options?• SQL dummy table

• Easy & familiar, but limited• Pre-generated range

• Proposed by Datastax’s Architect

• Distributed, but more complicated to implement• Sharding [Instagram]

• Discovered too late

• Unfamiliar with Postgres

http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram


We chose Snowflake• Built by Twitter, Apache 2.0 license

• https://github.com/twitter/snowflake

• “… network service for generating unique ID numbers at high scale..”

• Same motivation; MySQL -> C*

• A few tweaks for our Windows environment

https://github.com/twitter/snowflake




Technical reasons for Snowflake• Meets all requirements

• Tested in high transaction system

• Java based [Scala] implementation

• Thrift server

• Run as a Windows service with Apache Daemon

• Con: Requires Apache Zookeeper

• Coordinate the worker id


Connecting to Snowflake• Built our own .NET

Snowflake Client

• Snowflake server on each web node

• Local instance is primary

• Round robin failover to other nodes

• Auto failover AND recovery

• “Circuit Breaker” pattern

Web App SF

Server 1

Web App

SF

Server 3

Web App

SF

Server 2

Web App

SF

Server 4


Connecting to Cassandra with C#• Thrift alone too low level

• Needs

• CQL support

• Active development / support

• Wants

• ADO.NET / LINQ feel

• ????

• FluentCassandra is where we started


Vetting FluentCassandra• Pros

• Open source - https://github.com/fluentcassandra/fluentcassandra

• Nick Berardi, project owner, is excellent

• Designed for CQL

• Familiar feel

• Were able to start project development with it

https://github.com/fluentcassandra/fluentcassandra




Vetting FluentCassandra• Cons

• Immaturity

• Few users with high transaction system

• Permanent node blacklisting

• Lacked auto retry

• Couldn’t live with these limitations

• Tried adding resources dedicated to maturing it


Hector: Yes, please• Popular C* connector

• Use cases matching ours

• Good maturity

• Auto node discovery

• Auto retry

• Auto failure recovery

• Written in Java – major roadblock


Help!• We knew we still needed help.

• We found a company named Concord.

• Based out of the Twin Cites.

• Specialize in System, Process, and Data Integration.

• http://concordusa.com/

http://concordusa.com/




Concord’s Recommendation• Concord recommended that we use IKVM.NET to port

Hector to a .NET assembly.

• They had previous success using IKVM for other Java to .NET ports.

• They felt that maturing FluentCassandra was going to take longer than our timeline allowed.


About the IKVM.NET Project• http://www.ikvm.net/

• Open Source Project.

• Main contributor is Jeroen Frijters.

• He is actively contributing to the project.

• License allows for use in commercial applications.

http://www.ikvm.net/


What is IKVM.NET?• IKVM.NET includes the following components:

• A Java Virtual Machine implemented in .NET.

• A .NET implementation of the Java class libraries.

• Set of tools that enable Java and .NET interoperability.


Uses for IKVM• Drop-in JVM

• Included is a distribution of a .NET implementation of a Java Virtual Machine.

• Allows you to run jar files using the .NET stack.

• Example: ikvm -jar myapp.jar


Uses for IKVM• Use Java libraries in your .NET applications

• Using ikvmc you can compile Java bytecode to .NET IL.

• Example: ikvmc -target:library mylib.jar


Uses for IKVM• Develop .NET applications in Java

• Write code in Java.

• Compile to JVM bytecode.

• Use ikvmc to produce a .NET Executable.

• Can also use .NET API’s in Java code using the ikvmstub application to generate a Java jar file.

• Example: ikvmstub MyDotNetAssemblyName


Hector Converted to .NET• Per Concord’s recommendation we chose to compile

the Hector jar into a .NET Assembly.

• Hector and all of it’s dependencies are pulled into one .NET dll that can be referenced by any .NET assembly.

• In addition you will have to reference some core IKVM assemblies.

• Each Java dependency is given it’s own namespace with in the .NET dll.


HectorNet• Concord also created a dll called HectorNet that

wraps some of the Hector behaviors and makes it feel more like .NET.

• Such as supporting connection strings.

• Mapping Thrift byte arrays to .NET data types.

• Mapping to native .NET collections instead of using Java collections.

Why Not DataStax C# Driver?• We built everything using CQL 2.0.

• Wasn’t ready in time for our launch date.


Integrating Reporting

OLTPC*

Extract Transform

CUBESSAS

OLAPMS SQL

Load

ETL - SSIS


Integrating Reporting• The SSIS Extract process uses C# Script Tasks.

• Script Task needs references to HectorNet and all of its dependencies.

• SSIS can only reference assemblies that are in the GAC.

• Assemblies in the GAC have to be Signed.


• Why Cassandra?

• Challenges


• FluentCassandra

• Hector

• IKVM.NET

• HectorNet


• Data Modeling

• Lessons Learned


Data Classification• NativeX has three major classifications of data.

• Configuration or Master Data

• Activity Tracking

• Device History


Configuration Data

• Also referred to as Lookup Data or Master Data.

• This data is relatively small in terms of record counts.

• 10s – 100,000s of records not millions.

• Is used to operationally run our products.


Configuration Data• Examples in NativeX’s business:

• Mobile Apps

• Offers

• Campaigns

• Restrictions

• Queue Settings

Relational DataConfiguration data is typically relational in nature and therefore we continue to store it in MS SQL Server.


C* Data Modeling Basics• Data is stored inside of Column Families using nested

Key/Value Pairs.

• A CF can be thought of as a Table.

• They are made up of Rows and Columns.

• However, CFs do not have direct relationships to each other.

• You typically deal with one row at a time.


RowsA Row is the first level of the nested Key/Value pairs.

• A Row consists of:

• A Row Key (unique to the CF).

• A Row Value which is 1 to many Columns.

• A Row will typically represent:

• Single Entity/Record.

• Multiple records (known as a Wide Row CF).


ColumnsA Column is the second level of the nested Key/Value pairs.

• A Column consists of:

• A Column Name (Key) (unique to the Row).

• A Column Value.


Column Name• Column Names can consists of a value of any data

type.

• String, Integer, Date, UUID (GUID), etc.

• The Column Name is stored as part of every column.

• This means it has an impact to the size of your data.

• Can also use the Column Name to store data.


Column Value• A Column Value will typically contain:

• A single value such as an integer, string, date, etc.

• A whole record usually represented in XML, JSON, or some other document or object structure.


CF - Putting it all Together


Wide Row CF• A collection of like records organized into a single row.

• Each record is stored as a distinct column.

• Not unheard of for each row to have millions of columns.

• Data is often denormalized into XML or JSON documents.

• Good for storing:

• Time Series Data

• Event Series Data

• Logging Data

• Tracking Data


Wide Row Examples


• Why Cassandra?

• Challenges


• FluentCassandra

• Hector

• IKVM.NET

• HectorNet


• Data Modeling

• Lessons Learned


Lessons Learned• Get into production early

• Migration is hard

• Data Import = Reality

• Dev team needs to be integrated right away

• Training

• Operations / Troubleshooting

• Understanding your I/O profile is really important

• Are you sure you’re write heavy?

• Effects your hardware config, i.e. SSDs for us


Lessons Learned• Cluster sizing and hardware selection

• Dependent on data set + workload• You might get it wrong the first time• Enterprise vs. ‘commodity’

• Cassandra changes quickly• You need to keep up• Leverage mailing list, forums, release notes

• Scalable systems like C* have a massive amount of knobs, you need to know them


ProjectionsMonth Aug-13 Sep-13 Oct-13 Nov-13 Dec-13 Jan-14 Feb-14 Mar-14 Apr-14

Pub DAU 18,500,000 28,500,000 33,500,000 38,500,000 43,500,000 48,500,000 53,500,000 58,500,000 63,500,000Adv DAU 12,000,000 6,600,000 7,600,000 8,600,000 9,600,000 10,600,000 11,600,000 12,600,000 13,600,000

Total Devices 1,060,000,000 1,120,000,000 1,180,000,000 1,240,000,000 1,300,000,000 1,360,000,000 1,420,000,000 1,480,000,000 1,540,000,000Nodes Need for Disk 9 12 13 15 16 17 19 20 21Nodes Need for BF 22 16 17 18 19 20 21 22 23Nodes Need for RR 14 16 19 22 24 27 30 33 35

CapacitiesNumber of Nodes 30.00Usable Space/Node (GB) 600.00Total Usable Space (GB) 18,000.00Memory/Node (GB) 64.00JVM Heap Size (GB) 8.00BF Size / Node (GB) 1.50Replication Factor 3.00Read Requests/Node 1,000.00

Understand which KPI represents Node capacity.

DSE for the Win!• We use DataStax Enterprise.

• Mainly for support, which continues to be a life saver.


Thank you!• Join the MSP C* Meetup

• http://www.meetup.com/Minneapolis-St-Paul-Cassandra-Meetup/

• Contact us

• [email protected]

• [email protected] @breakingtrail

• Slide Deck

• http://www.slideshare.net/jjsmoley/the-perils-and-triumphs-of-using-cassandra-at-a-netmicrosoft-shop

http://www.meetup.com/Minneapolis-St-Paul-Cassandra-Meetup/

http://www.meetup.com/Minneapolis-St-Paul-Cassandra-Meetup/

mailto:[email protected]

mailto:[email protected]

http://www.slideshare.net/jjsmoley/the-perils-and-triumphs-of-using-cassandra-at-a-netmicrosoft-shop



the perils and triumphs of using cassandra at a .net/microsoft shop

Technology

nativex holdings

nativex c

llc cassandra

llc backstory

llc auto id generation

llc technical reasons

gb of data

specific data