Big Data

What is it ?

What trends in IT support it ?

Some examples

What tools are used ?

The future ?

Big Data What is it ?

Very large data set volumes

Very long / unacceptable processing times

Very large data velocity ( inputs / outputs )

Very large varieties of data

High level of complexity

Big Data Supporting Trends

Moore's Law

An observation that the number of transistors on integrated circuits doubles every two years.

Big Data Supporting Trends

Kryder's Law

The density of storage is increasing and the cost decreasing at a rate faster than Moore's Law.

Big Data Supporting Trends

Butter's Law

Relates to network capacity and states that the cost of sending data over an optical network halves every nine months.

Big Data Supporting Trends

Parallel Processing

Task parallelism, breaking the task down into its constituent parts and processing them simultaneously.

Big Data Examples

NASA Climate Simulation

32 petabytes

The Large Hadron Collider

25 petabytes annually, 200 petabytes after replication

Wall mart

2.5 petabytes per hour

Big Data Tools

Hadoop

Hadoop is often used at the server level to organise the cluster along with a NoSQL database for data storage.

NoSQL

Databases ( non sql ) that use looser consistency models than relational databases. Performance gains via simplification using key value stores.

MPP

Massively parallel processing and analytics databases. Fast for data aggregation but slow for data loading.

Big Data The Future

Data sets will continue to grow

Storage unit costs will continue to decrease

Processing costs will decrease

Network capacity will continue to grow

Data growth may exceed processing capacity

Contact Us

Feel free to contact us at

www.semtech-solutions.co.nz

info@semtech-solutions.co.nz

We offer IT project consultancy

We are happy to hear about your problems

You can just pay for those hours that you need

To solve your problems