building big data applications with sql-mapreduce and...
TRANSCRIPT
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.1
Title of Presenter | Date
Building Big Data Applications with SQL-MapReduce and Aster Developer Express
Peter Pawlowski, Software Engineer
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.2
Big Data: The Trend
Many companies have anever-increasing amount of data.
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.3
Big Data: The Challenge
How do we enable all these companies to build useful
applications using this data?
“Big Data Applications”
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.4
Not Everyone is Google
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.5
Our Goal: Big Data Applications Made Easy
1. We enable storing terabytes to petabytes of data in our scalable, parallel database.
2. We provide a framework for running rich, interesting analytics on this data.
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.6
Our Goal: Big Data Applications Made Easy
1. We enable storing terabytes to petabytes of data in our scalable, parallel database.
2. We provide a framework for running rich, interesting, and fast analytics on this data.
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.7
Aster Data nCluster
Workers (Parallel Processing)
Loading/Export
Queries
Backup
Queries/Results
Reports, Advanced Analytics, Applications(SQL, SQL-MapReduce, ODBC , JDBC)
Queen
Data Data
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.8
We started with a SQL interface
• Great fit for many data problems
• Many people know it
• Plugs into many applications & tools
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.9
… but SQL …
• Is a bad fit for many kinds of analytic applications
• Can be hard to write & maintain
• Can be hard to execute efficiently
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.10
Enter SQL-MapReduce
• SQL-MapReduce is our framework for pushing analytic code into the database
• Enables much richer analytics to happen on the data in the database
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.11
Two Steps to SQL-MapReduce
• Write a SQL-MapReduce function in Java*
• Invoke it from SQL
* Other languages supported, too. See our website.
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.12
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.13
Test Environment
Deploy3Test2Develop1
SQL / SQL-MR App
SQL / SQL-MR App
Aster Data Developer Express
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.14
Step 1: Develop
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.15
Step 2: Test Locally
Desktop Test Environment
Input OutputInput Files
Output Files
• Run tests on your own machine without requiring an nCluster instance.
• Configure with JSON.
Application
Results
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.16
Step 2: Test Locally
Desktop Test Environment
Input OutputInput Files
Output Files
• Run tests on your own machine without requiring an nCluster instance.
• Configure with JSON.
Application
Results
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.17
Step 2: Test Locally
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.18
Step 3: Deploy
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.19
Step 3: Deploy
Aster Data nCluster
App App AppApp App App
SQL / SQL-MR App
• Push application into Aster database directly from within IDE
• Initiate and validate application with actual data stored inside nCluster
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.20
Aster Data Developer Express is freely available
Download atwww.asterdata.com