data management research forecast: hot with an increasing chance of clouds
Post on 23-Feb-2016
26 Views
Preview:
DESCRIPTION
TRANSCRIPT
Data Management Research Forecast:Hot With an Increasing
Chance of Clouds...Michael Carey
Information Systems GroupCS Department
UC Irvine
2
Cloud-Related DB Bandwagons• MapReduce and Hadoop– Parallel programming for dummies– But now Hive, Pig, Scope, Jaql, … MapReduce is the new runtime
• DFSs and HDFS (and CSS)– Scalable, self-managed, Really Big Files– But now BigTable, HBase, … HDFS (or CSS) is the new file storage
• Key-value stores– Charter members of the “NoSQL movement”– Includes S3, Dynamo, HBase, Cassandra, … Key-value stores are the new record managers
Perhaps Our Brains Are Clouded!
• Some of the possible symptoms include…– A number of us are focused on tweaking Hadoop– Most of us are compiling our declarative data analysis
languages into Hadoop jobs (map/reduce/combine)– Some of us are building “cloud database systems” as a layer
on top of “givens” such as HDFS– Some other possible signs of “cloud dementia” include…• We keep forgetting where we put our schemas• We don’t remember working on parallel DBs in the 80’s• We’re about to learn some hard lessons all over again…
Thou Shalt Worship DFS& BigTable
Thou Shalt MapReduce!
3
4
DB-ers: Let’s Do This Stuff “Right”!• In my opinion:– Yes, the OS/DS folks out-scaled us (oops!)– But, we’d be “nuts” to simply build on their
current (i.e., first-generation) foundation• Let’s bring our DB experience to the party…!– Recall important past architectural lessons• I’ll give some quick examples of this…
– Plus, it can be “windy” in the stratosphere• I’ll give one quick example of this too…
5
Current Data Layers
HDFS
HBase
Cloud DBMS?
Ordered byte streamsAppend onlyRandomly partitioned
Sets of records/columnsKey-based retrievalRange partitioned
Some data model?Some query language?
6
…
…
7
8
!!
9
Let’s Do This Stuff “Right”! (cont.)
• Identify the lessons, then do it “right”– Open-source S/W on commodity H/W– Non-monolithic software components– Equal opportunity data access (external sources)– Fault-tolerant query execution (for data analysis)– Little pre-planning or DBA-type work required– Tolerant of flexible / nested / absent schemas– Types and declarative languages (duh…! )
10
What If We’d Meant To Do This?• What is the “right” basis for analyzing and
managing the data of the future?– Storage and data distribution layers?– Query runtime layer (and division of labor)?– ….
• Let’s explore how to build new information management systems to live in the cloud that…– Scale to thousands of nodes (and beyond)– Make effective use of shared (and elastic) resources– Execute queries in the face of partial failures– Seamlessly support external data access– Don’t require five-star wizard administrators– ….
11
In Short…
• Ask not what cloud software can do for you, but what you can do for cloud software…!– We are, after all, the data experts!– We just have to remember our roots and bring our experiences and
past lessons to bear on cloud issues• We’re asking this question at UCI (and UCSD/UCR):– ASTERIX: A new data platform to ingest, store, manage, index, query,
analyze, and publish vast quantities of semi-structured information– Hyracks: A new foundation for scalable data analysis – both a
runtime system for ASTERIX and an extensible parallel data analysis platform in its own right
– Open-source sharing planned for both (eventually)
12
• Semistructured data management– Core work exists– XML & XQuery, JSON, …– Time to parallelize and scale out
• Parallel database systems– Research quiesced in mid-1990’s– Renewed industrial interest– Time to scale up and de-schema-tize
• Data-intensive computing– MapReduce and Hadoop quite popular– Language efforts even more popular (Pig, Hive, Jaql, …)– Ripe for parallel DB ideas (e.g., for query processing) and support
for stored, indexed data sets
The ASTERIX ProjectSemistructured
Data Management
Parallel Database Systems
Data-Intensive
Computing
13
ASTERIX Project Overview
Disk
MainMemory
Disk
CPU(s)
ADMData
MainMemory
Disk
CPU(s)
ADMData
ADMData
Hi-Speed Interconnect
Data loads & feeds from external sources (XML,
JSON, …)
AQL queries & scripting
requests and programs
Data publishing to
external sources and
appsASTERIX Goal: To ingest, digest, persist, index, manage, query, analyze, and publish massive quantities of semistructured information…
(ADM = ASTERIX
Data Model;AQL =
ASTERIX Query
Language)
MainMemory
CPU(s)
14
ASTERIX User Model• Data model (ADM) with
open and closed types• Query language (AQL)
for nested and semi-structured querying
• Support for both stored and external datasets
for $user in dataset(‘User’)where some $i in $user.interests satisfies $i =“movies”return {“name”: $user.name };
15
Hyracks Runtime• Partitioned-parallel platform for data-intensive computing• Job = dataflow DAG of operators and connectors
– Operators consume/produce partitions of data– Connectors repartition/route data between operators
• Hyracks vs. the “competition”– Based on time-tested parallel database principles– vs. Hadoop: More flexible model, less “pessimistic”implementation– vs. Dryad: Supports data as a first-class citizen
16
ASTERIX Research Issue Sampler• Semistructured data modeling
– Open/closed types, type evolution, relationships, ….– Efficient physical storage scheme(s)
• Scalable storage and indexing– Self-managing scalable partitioned datasets– Ditto for indexes (hash, range, spatial, fuzzy; combos)
• Large scale parallel query processing– Division of labor between compiler and runtime– Query plan decision-making timing and basis– Model-independent complex object algebra (AQUA)– Fuzzy matching as well as exact-match queries
• Multiuser workload management (scheduling)– Uniformly cited: Facebook, Yahoo!, eBay, Teradata, ….
17
Closing Thoughts• IMO, this is the most exciting time to be a data
management researcher in the last 20-30 years!– Data coming out our ears (and other openings) thanks to
WWW, Twitter, FaceBook, ... (so-called “data exhaust”)– Data models and query languages totally up for grabs (witness
the “NoSQL movement”)– System layers and architectures also up for grabs ()
• Many interesting challenges to Cloud our thinking– Opportunities for DB, OS, and PL research to converge– Important to learn from the past as we plan for the future– Resource management at scale is one of the biggies– Consistency models / programming models are another
top related