james corcoran, head of engineering emea, first derivatives, "simplifying big data...

Simplifying Big Data Architecture

Paris 1st December 2016

2 | KX SYSTEMS

Agenda

• About

• Kx Technology

• Solutions

• Technical Overview

• Resources

• Q & A

3 | KX SYSTEMS

About

First Derivatives plc

People1,600 FTEs

LocationsIreland, London, New York,

Singapore,Sydney, Tokyo, Toronto, Zurich

HistoryFounded in 1996

Public listed CompanyAlways Profitable

Divisions

KxBig Fast Data

Capital Markets Consulting

Domain expertsTechnology experts

MRP/Prelytix Digital marketing

resources

Our Software

KxStreaming Analytics, In Memory Computing, - the market standard

Tools for KxDevelop, Deploy,

Visualize

Industry SolutionsPowered by Kx

Implemented and supported by us

4 | KX SYSTEMS

Kx Technology Overview

• Integrated columnar database & programming system

• Streaming, real time and historical data

• Built for massive data volumes

• In-database analytics

• Parallelism

• Compression

• More

The kdb+ database Features

• Interpreted

• Event-driven

• Functional

• Array / Vector

• Query

• Time-series

q Programming Language

5 | KX SYSTEMS

Tools for Kx Solutions

Solutions Overview

• Dashboards for Kx

• Analyst for Kx

• Control for Kx

• Stream for Kx

• Kx for Algos

• Kx for Flow

• Kx for Monitoring

• Kx for Pharma

• Kx for Surveillance

• Kx for Sensors

• Kx for Telcos

Visualise&

Discover

Architect&

Manage

Enterprise Concepts

6 | KX SYSTEMS

kdb+

Technical Overview

7 | KX SYSTEMS

kdb+

• Integrated columnar database & programming system

• Streaming, real time and historical data

• 32 bit free version

• 64 bit licensed version

8 | KX SYSTEMS

kdb+ Features

• Standard OS & hardware

• In-database analytics

• Compression

• Parallelism

• Interfaces

9 | KX SYSTEMS

q programming language

• Functional

• Array/Vector

• Query

• Interpreted

• Time-series

10

q LanguageSeamless database and data manipulation language • We are fast not only due to data architecture, our native

programming language, q, runs inside the database not in separate processes with costly data passing.

• We are one of the few fully 64 bit databases and unique in having time as a native type, with nanosecond resolution and a full set of operations over time.

Quick to Deliver• Unlike many compile-link and run approaches q is

dynamic allowing much shorter development and deployment cycles

Allows you to get more from less code• q is terse - this means a smaller code base to maintain

and debug (one line in q versus many of Java).

Other Key Features• Array/Vector Language• Parallelism • Native Compression

Java

Project Euler Problem 1* in Java

Project Euler Problem 1* in q

*Find the Sum of all multiples of 3 or 5 below 1000. For more information on the Euler Programming Problems see - https://projecteuler.net

11

Sample q query

select open: first price, high: max price, low: min price, close: last price from trade where date = 2013.05.01, sym = `AAPL

12

q vs. standard SQL

• q supports order for faster execution• q eliminates many joins• Standard SQL does not enforce row order• Built-in support for CSV, JSON & XML• higher productivity & less maintenance

13

Parallel queries in q

Serial:

calcEvent each eventTypesParallel:

calcEvent peach eventTypes

14

Time-Series Analysis

• Native temporal data types• high-precision (nanosecond)• temporal arithmetic• asof join/bi-temporal joins• join time-series on-the-fly

15

Time-Series Analysis cont’d

select count i by userAgent,5 xbar time.minute from events where date within (.z.D - 7;.z.D), eventType=`login

aj[`time; select time,price from trade where date=last date,sym=`MSFT; select time,bid,ask from quote where date=last date,sym=`MSFT]

16

Streaming, Real-Time and Historical Data

17 | KX SYSTEMS

Sample Architecture

18 | KX SYSTEMS

Kx Solutions

Technical Overview

19 | KX SYSTEMS

Dashboards for Kx

20 | KX SYSTEMS

Analyst for Kx

21 | KX SYSTEMS

Streams for KxFast Data Management

• Clustered services

• Load-balancing

• Failover

• Automated recovery

• Query Manager

• Messaging Server

• Alert Framework

• File Watchers & Loaders

• Scheduling

22 | KX SYSTEMS

Kx

Resources

23 | KX SYSTEMS

Resources – Wiki Page

24 | KX SYSTEMS

Resources – Kx Community

25 | KX SYSTEMS

Resources – Forum & Google Group

26 | KX SYSTEMS

Resources – Github

27 | KX SYSTEMS

Resources – Q For Mortals

28 | KX SYSTEMS

Resources – Q Tips

29 | KX SYSTEMS

Resources – Free 32-bit download

30 | KX SYSTEMS

Resources – Kx Community Meetups

31 | KX SYSTEMS

Resources – Lecture Series

32 | KX SYSTEMS

Thank You.

kx.com@kxsystems

33 | KX SYSTEMS

Appendix

• Kx Wiki • Kx Community • Google Group • Kx Github • Q For Mortals• Q Tips • STAC benchmarks• Free version• Kx Meetups• FD Lecture Series