couchbase at cox automotive – couchbase live new york 2015

Couchbase at Cox Automotive

3

Who we are

4

Who we are

5

Who we are

6

What we do at

7

What we do at

8

What we do at

9

Our data changes quickly

Couchbase at Cox AutomotiveKeeping pace with our data

11

What sort of data do we store?

12


13


14

How do we keep pace with our data?

15


16


17


18


Two weeks to get to production!

Couchbase at Cox AutomotiveAccelerating our pace

20

We churn through truckloads of data

21

We churn through truckloads of data

22

An API with Couchbase storage

23


24


25

Implementation

API design features

• RESTful endpoints• Use of JSON format and immutable documents• Cached access to SQL Server look ups for configuration• Couchbase storage for speed and flexibility

26

It’s simple to move our data into documents

SQL Server crashes when we try to list all our “staging” tables!

27

It’s simple to move our data into documents{ "ImageURLs": [ "http://www.xhjsdfk.com/15234.jpg", "http://www.xhjsdfk.com/93784.jpg", "http://wwwxhjsdfk.com/13378.jpg" ], "Properties": { "BodyStyle": "2dr Car", "TransmissionSpeed": 6, "TransmissionTypeCode": "6-Speed Ma", "VehicleCondition": "New", "Year": 2021, }, "VIN": "1TAGP28135G148557" }, "Timestamp": "2015-10-05T13:38:46.2979624-04:00", "Version": 1}

JSON lends itself to flexible storage

There’s no schema

28


We coded an additive storage process.

Say we get this input first….

29


And then we get this data, for the same vehicle…

30


We produce a resulting document…

31


Now we’ve solved our

archiving problem andour rollback problem!

Version 1 Version 2

32

How we got operational in two data centers

Now we use Couchbase as our database of record for this data,

we want resiliency with XDCR

33

How we got operational in two data centers

Remember that TTL setting for the cached data?

Here’s some free advice

Couchbase at Cox AutomotiveStaying at the front of the pack

35

Wow; we’re maxed out again!

We have a fancy SQL Server*doing the heavy lifting

- 4 TB of primary data- Gigabytes of change per minute- 60 CPU cores!- 2TB RAM- 12TB Fusion IO cards!

*cluster of 3 machines

36

Data model migration

Forward plans for SQL and NoSQL

OPERATIONAL CACHEDCOMPUTED

RAW DATA

37

What functions should Couchbase serve?

• Easing the load for heavy-read objects from SQL Server• Applications where ‘eventually consistent’ is OK• Outbound ‘file’ storage for computed documents

• Novel queries against unstructured documents can leverage N1QL• When the query load is unpredictable, it can scale fast

38

Future steps: How to migrate production data?

Our traditional approach

PRODUCTION STAGINGRESTORE

QA ENV.

C# APP

39

Future steps: How to migrate production data?

With Couchbase 4.0

PRODUCTION STAGINGRESTORE

QA ENV.

XDCRFilter

Couchbase at Cox AutomotiveConclusion

41

How we began with Couchbase

“Start where you are.”- Chris Gardner

Couchbase at Cox AutomotiveQuestions

couchbase at cox automotive – couchbase live new york 2015

Software