@daniel_abadi

Yale University

*Big Data and the Database Community

*Big Data

*The Big Data phenomenon is the best thing that could have happened to the database community

*Despite other definitions related to ‘3 Vs’ --- Big Data means BIG Data

*Which means we need scalable database systems

*Still two main components of Big Data

*Performing data analysis at scale

*Performing requests on data at scale

*Performing Data Analysis at Scale

*Database community has won the battle

*Some thought that MapReduce might replace traditional database technology as the primary means to perform analysis at scale

* Just about every MapReduce vendor has abandoned this goal

*Hadapt, Impala, Tez, and several others are in a race to see who can add the most traditional database execution technology to Hadoop fastest

*Everyone is going in the direction of cost-based optimizers, traditional database operators, and push-based query execution

*Performing Requests on Data

at Scale

*The database community is losing the battle

*NoSQL systems still have very little traditional database technology inside (despite adding SQL interfaces)

*No race to add DB technology --- why?

* Don’t blame CAP --- CAP is only relevant when there’s a network partition

* We never figured out how to do ACID and active replication at scale

* Many new proposals make simplifying assumptions in order to handle scale

* It’s been 30 years ---- why can’t we build a distributed database that can handle distributed transactions over actively replicated data at scale?