dzone java 8 block buster: query databases using streams
Post on 25-Jan-2017
398 Views
Preview:
TRANSCRIPT
Meetup Gothenburg Dec 28 2015 Per-Åke Minborgminborg@speedment.com
Palo Alto Office Speedment Java One Presentation
Background
Before Speedment
Connector
JDBC
ORM
JPA
ApplicationPROSWidely spread standard
• Many providers• Many developers
You can work with a relational database as if it were object oriented Increased productivity Provides a certain degree of abstraction (you can replace your DBMS)
CONS Slows down the application and gives unpredictable performance You can not access all Java 8 featuresTime consuming to write applications for existing databases, You must still write SQL, HQL, et. AlHigher the risk for errors
Java
ORM
JPA
Speedment- a Java Development Tool
Connector
JDBC
Application Object Oriented functional programming
Fast to develop Fast Execution Easy to test Secure
Make use of all Java 8 features Domain Model reflects your existing
datasource Automatic generation of Code (”error free”)
Java
Speedment
Java 8
Application
Speedment- a Java Development Tool
Code Examples• Embrace Java 8 paradigms with stream(), filter(), sort(), limit() etc.
• Forget about SQL, JQL et. al
Database Table “hare”
mysql> explain hare;+-------+-------------+------+-----+---------+----------------+| Field | Type | Null | Key | Default | Extra |+-------+-------------+------+-----+---------+----------------+| id | int(11) | NO | PRI | NULL | auto_increment || name | varchar(45) | NO | | NULL | || color | varchar(45) | NO | | NULL | || age | int(11) | NO | | NU | |+-------+-------------+------+-----+---------+----------------+4 rows in set (0.01 sec)
Code Generationpublic interface Hare extends Entity<Hare> {
public final static ReferenceComparableField<Hare, Integer> ID = … public final static ReferenceComparableStringField<Hare> NAME = … public final static ReferenceComparableStringField<Hare> COLOR = … public final static ReferenceComparableField<Hare, Integer> AGE = …
Integer getId(); String getName(); String getColor(); Integer getAge();
Hare setId(Integer id); Hare setName(String name); Hare setColor(String color); Hare setAge(Integer age);
/** Graph-like traversal methods eliminating JOINs */ Stream<Carrot> findCarrotsByOwner(); Stream<Carrot> findCarrotsByRival(); Stream<Carrot> findCarrots();}
Querying using Java 8 StreamsList<Hare> oldHares = hares.stream() .filter(Hare.AGE.greaterThan(4)) .collect(toList());
SELECT * from `hare`WHERE`age` > 4;
Stream Fundamentals Detailed description Slide 35-45
Depends on Terminating OperationOptional<Hare> oldHare = hares.stream() .filter(Hare.AGE.greaterThan(4)) .findAny();
SELECT * from `hare`WHERE`age` > 4 LIMIT 1;
Depends on Terminating Operationlong noOldHares = hares.stream() .filter(Hare.AGE.greaterThan(4)) .count();
SELECT count(*) from `hare`WHERE`age` > 4;
Entities are linked
Optional<Carrot> carrot = hares.stream() .filter(Hare.NAME.equal(”Spire”)) // Carrot is a foreign key. .flatmap(Hare::findCarrots) .findAny();
Multi-threading
hares.stream() .parallel() .forEach(doExpensiveOperation());
PersistenceHare harry = hares.newInstance() .setName("Harry") .setColor("Gray") .setAge(3) .persist();
// OR JPA-style
entityManager.persist(harry);
Delete Using Streams users.stream() .filter(User.ID.greaterThan(4)) .forEach(User::delete);
DELETE FROM `user`WHERE `id` > 4;
Initialization
Speedment speedment = new HareApplication().build();Manager<Hare> hares = speedment.managerOf(Hare.class);
Speedment Insane Mode•Extreme Performance, real time big Data
•Convert large databases into In-JVM-memory Java Objects
•Scales Linearly -More CPUs, More Insane
In Memory Initialization
Speedment speedment = new HareApplication()
.with(OnHeapReadOnlyCacheComponent.class) .build();
How does it Work?
23
1 In-JVM-memory cache2 Java Objects in Graph View3 Separates Reads and WritesReflection in
real time
ApplicationJVM
Read
Write
Why will in-JVM-Memory give Insane Speed?
Database In JVM Memory
Compare latencies using the Speed of Light
Why will in-JVM Memory give Insane Speed?• Keep everything in-JVM
• No need to check if an object is in cache
• We can organize objects knowing that we have them all
• We do not even have to look at individual objects -> O(1)
• Everything is Java
Size
Time
O(1)
O(n)
Speedment – OSS and Enterprise
Cloud Deployment
RDBMS
Speedment
Speedment
Speedment Speedment
Speedment
• Spin up nodes as you grow
• Partition data over nodes
How can I fit my full Database in JVM Memory?
~24 GB
>1 TB
>4 TB
∞
On-HeapOff-Heap
SSD
SQL
Database Size(After Compression)
Scale out with our partner Hazelcast• Set up a cluster in the Cloud• Scale out your existing database• Set up Terabytes of RAM
Properties:• The API will remain the same regardless of selected Storage Engine
• Massively concurrent and lock free
• Scale with CPU performance and number of cores
• Eventually consistent
• Transaction safe, if you want
Configuration• Groovy script like Gradle
• Config file is placed at the same location as the POM
dbms { schema { table { name = ”hare"; column { name = "name"; } } }}
Ease of use:• Single Maven dependency in POM
• Maven targets
• Works with Gradle
• GUI to automatically derive the groovy config file from existing databases
Properties:• The database ”owns” the data
• Data is reflected into Materialized Object Views (MOVs)
• MOVs and application resides in the same JVM
• Most operations are O(1)
• persons.byName(“Bob") -> Map<PK,Person> with 1000 persons in 100 ns (30 m)
• TCP/IP RTT 100 us -> +1000 "finds" per TCP RT.
• In-memory DBs with zero query latency are snails by definition
http://localhost:4567/trades/1001/200? from=0&to=1441147840000&callback=Ext.data.JsonP.callback1
Ext.data.JsonP.callback1([{'start':1420131600000,'stop':1420236681000,'average':100876,'open':100000,'close':102209,'high':104060,'low':98714},{'start':1420236682000,'stop':1420243199000,'average':102339,'open':102224,'close':103348,'high':103366,'low':101603},{'start':1420477200000,'stop':1420502399000,'average':102272,'open':103353,'close':103316,'high':103608,'low':100819},{'start':1420563600000,'stop':1420657005000,'average':101682,'open':103315,'close':102292,'high':103755,'low':99241},
… (200 rows in total) ...{'start':1441147839000,'stop':1441147839000,'average':142165,'open':142165,'close':142165,'high':142165,'low':142165}]);
Objectives:
Stream Fundamentals Per-Åke Minborgminborg@speedment.com
Setup
36
DB App
Pwd ?
In-JVM Cache
Stream<Trade> stream = trades.stream();
37
DB
Data Source
Trade5 Trade4 Trade3 Trade2 Trade1
trades.stream().parallel();
38
TradeN
Trade4
Trade3
Trade2
Trade1
TradeX
Thread 1
Thread 2
Thread N
DB
Data Source
trades.stream().filter(Predicate<Trade> predicate);
39
Trade2 Trade1predicate.test(trade)Trade5 Trade4
Trade3
false
true
trades.stream().map(Function<Trade,String> mapper);
40
Json2 Json1Trade5 Trade4mapper.
apply(trade)
e.g. Trade::toJson
trades.stream().collect(Collector collector);
41
TradeN
Trade4
Trade3
Trade2
Trade1
TradeX
apply(trade)
apply(trade)
apply(trade)
Complete Stream
42
DB
Data Source
Finding first and last date/time in an interval
43
from to
selected
trades.stream().parallel().filter(DATE.between(from, to)).filter(STOCK.equal(stock)).mapToLong(Trade::getDate).summaryStatistics();
44
2 3 4
2 3 4
2 3 4
5
12345
DB
Data Source
45
DB
Data Source 1
4
4
4
52
3
trades.stream().parallel().filter(DATE.between(from, to)).filter(STOCK.equal(stock)).mapToLong(Trade::getDate).summaryStatistics();
12345
Generating a REST APIIntroduction to Speedment and Sencha
Emil Forslundemil@speedment.com
Outline●The Speedment + Sencha Software Stack●Using the Pivotgrid Plugin for Speedment
Overview of the Software Stack
DB
Javascript Framework
ServerFramework
DB Abstraction Layer
HTML5RESTJava 8JDBC
Database Layer●Holds all the data collected from various sources●High Level of Normalization
Sale
ProductSalesperson
Customer
Country
Region
Office
Cost Price
Count
Database Abstraction Layer●Presents an Object-Oriented view of the database●Write expressive and powerful queries●Optimize reading by using in-memory caching●Better testability and higher productivity
speedment.managerOf(Sale.class) .stream() .filter(Sale.PRODUCT.equal(105)) .collect(toList());
Server Framework●A lightweight webserver●Separates client/server-responsabilitys●Execute selected business logic as response to REST commands●Low Level of Abstraction
http://localhost:8123/salesinfo/?c=callback&filter=[{...}]
Javascript Framework●Presents data in user-friendly components●User can navigate the data in the browser●Automatically requests additional data from the server
Introduction to the Pivot Grid for Speedment●Generate a full-fledged server application from your database●Select exactly which data to expose in the API●Define aliases for columns and tables●Map between different data types●Create virtual columns calculated in real-time
Step 1 – Create a New Project
Step 2 – Find Pivotgrid Archetype
Step 3 – Name Project
Step 4 – Run Speedment GUI
Step 5 – Press Generate
Step 6 – Run Application
Example Application: Salesinfo
Thanks!
top related