exploring four datomic superpowers

Lucas Cavalcanti & Edward Wible

Exploring Four Datomic Superpowers

2

Why build a bank from scratch in Brazil using Clojure and Datomic?

3

3

4

1 Audit Trail

5

Customer joins waiting list for a card01 NOV 10:00

event stream

5

Customer joins waiting list for a card01 NOV 10:00

Robot 437aae3 approves R$3K limit01 NOV 11:00

event stream

5

Customer joins waiting list for a card01 NOV 10:00

Robot 437aae3 approves R$3K limit01 NOV 11:00

Mastercard purchase, Starbucks, R$10009 NOV 08:00

event stream

5

Customer joins waiting list for a card01 NOV 10:00

Robot 437aae3 approves R$3K limit01 NOV 11:00

Mastercard purchase, Starbucks, R$10009 NOV 08:00

Support agent increases limit to R$5K15 NOV 15:00

event stream

5

Customer joins waiting list for a card01 NOV 10:00

Robot 437aae3 approves R$3K limit01 NOV 11:00

Mastercard purchase, Starbucks, R$10009 NOV 08:00

Support agent increases limit to R$5K15 NOV 15:00

Customer blocks card15 NOV 17:05

event stream

facts over time

6

01 NOV 10:00

01 NOV 11:00

09 NOV 08:00

15 NOV 15:00

15 NOV 17:05

[<customer> :customer/id #uuid “b2c90…”]

[<account> :account/customer <customer>][<account> :account/limit 3000][<card> :card/account <account>][<card> :card/status :card.status/active]

[<purchase> :purchase/card <card>][<purchase> :purchase/amount 100][<purchase> :purchase/merchant “Starbucks”]

[<account> :account/limit 5000][<account> :account/limit 3000]

[<card> :card/status :card.status/blocked][<card> :card/status :card.status/active]

facts over time

6

01 NOV 10:00

01 NOV 11:00

09 NOV 08:00

15 NOV 15:00

15 NOV 17:05

[<customer> :customer/id #uuid “b2c90…”]

[<account> :account/customer <customer>][<account> :account/limit 3000][<card> :card/account <account>][<card> :card/status :card.status/active]

[<purchase> :purchase/card <card>][<purchase> :purchase/amount 100][<purchase> :purchase/merchant “Starbucks”]

[<account> :account/limit 5000][<account> :account/limit 3000]

[<card> :card/status :card.status/blocked][<card> :card/status :card.status/active]

entity attribute value

7

7

don’t lose data

7

don’t lose data

what changed

when

7

don’t lose data

what changed

when

as-of since

7

don’t lose data

what changed

when

as-of since

fork merge

8

What was the initial limit for the card?

8

What was the initial limit for the card?

At the time the Starbucks transaction occurred, which fraud triggers would have activated?

8

What was the initial limit for the card?

At the time the Starbucks transaction occurred, which fraud triggers would have activated?

How long did the customer spend on each stage of the acquisition funnel?

8

What was the initial limit for the card?

At the time the Starbucks transaction occurred, which fraud triggers would have activated?

How long did the customer spend on each stage of the acquisition funnel?

How frequently do we see amount changes on Starbucks transactions?

9

What sequence of events resulted in the Starbucks transaction being persisted?

9

What sequence of events resulted in the Starbucks transaction being persisted?

iOShttp-inkafka-outkafka-in

iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A

9

What sequence of events resulted in the Starbucks transaction being persisted? correlation-id

iOShttp-inkafka-outkafka-in

iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A

Who was responsible for the credit limit increase?

9

What sequence of events resulted in the Starbucks transaction being persisted? correlation-id

iOShttp-inkafka-outkafka-in

iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A

Who was responsible for the credit limit increase?

9

What sequence of events resulted in the Starbucks transaction being persisted? correlation-id

iOShttp-inkafka-outkafka-in

iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A

#uuid “b2c90…”“lucas@nubank.com.br”:kafka-LIMIT-CHANGED:robot-437aae3

Who was responsible for the credit limit increase?

9

What sequence of events resulted in the Starbucks transaction being persisted? correlation-id

iOShttp-inkafka-outkafka-in

iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A

user#uuid “b2c90…”“lucas@nubank.com.br”:kafka-LIMIT-CHANGED:robot-437aae3

Who was responsible for the credit limit increase?

9

What sequence of events resulted in the Starbucks transaction being persisted?

Why was the customer’s card blocked?

correlation-id

iOShttp-inkafka-outkafka-in

iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A

user#uuid “b2c90…”“lucas@nubank.com.br”:kafka-LIMIT-CHANGED:robot-437aae3

Who was responsible for the credit limit increase?

9

What sequence of events resulted in the Starbucks transaction being persisted?

Why was the customer’s card blocked?

correlation-id

iOShttp-inkafka-outkafka-in

iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A

:fraud-preventative:recurring-scheduled:late-payment:data-migration

user#uuid “b2c90…”“lucas@nubank.com.br”:kafka-LIMIT-CHANGED:robot-437aae3

Who was responsible for the credit limit increase?

9

What sequence of events resulted in the Starbucks transaction being persisted?

Why was the customer’s card blocked?

correlation-id

iOShttp-inkafka-outkafka-in

iZb iZb.jnAiZb.jnA.9CdiZb.jnA.9Cd.l9A

reason:fraud-preventative:recurring-scheduled:late-payment:data-migration

user#uuid “b2c90…”“lucas@nubank.com.br”:kafka-LIMIT-CHANGED:robot-437aae3

(defn block-card [card reason] (d/transact (conn) [[:db/add card :card/status :card.status/blocked]

]))

10

storing transaction metadata

(defn block-card [card reason] (d/transact (conn) [[:db/add card :card/status :card.status/blocked]

]))

{:db/id (d/tempid :db.part/tx) :audit/user "lucas@nubank.com.br" :audit/cid "iZb.jnA.9Cd.l9A" :audit/tags :fraud-preventative}

10

storing transaction metadata

11

[<customer> :customer/id #uuid “b2c90…”] 01 NOV 10:00

[<account> :account/customer <customer>][<account> :account/limit 3000M][<card> :card/account <account>][<card> :card/status :card.status/active]

01 NOV 11:00

[<purchase> :purchase/card <card>][<purchase> :purchase/amount 100M][<purchase> :purchase/merchant “Starbucks”]

09 NOV 08:00

[<account> :account/limit 5000M][<account> :account/limit 3000M]15 NOV 15:00

[<card> :card/status :card.status/blocked][<card> :card/status :card.status/active]15 NOV 17:05

transactions over time with metadata

11

[<customer> :customer/id #uuid “b2c90…”] 01 NOV 10:00

[<account> :account/customer <customer>][<account> :account/limit 3000M][<card> :card/account <account>][<card> :card/status :card.status/active]

01 NOV 11:00

[<purchase> :purchase/card <card>][<purchase> :purchase/amount 100M][<purchase> :purchase/merchant “Starbucks”]

09 NOV 08:00

[<account> :account/limit 5000M][<account> :account/limit 3000M]15 NOV 15:00

[<card> :card/status :card.status/blocked][<card> :card/status :card.status/active]15 NOV 17:05

transactions over time with metadata

12

querying transaction metadata

(defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))

12

querying transaction metadata

(defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))

[?card :card/status ?status ?tx]entity attribute value txn

12

querying transaction metadata

(defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))

12

querying transaction metadata

(defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))

12

querying transaction metadata

(defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))

13

2 Authorization

GET /purchases/1337/comments

should we return data?

GET /purchases/1337/comments

200 OK 401 Unauthorized

should we return data?

customer

account

card purchase

payment

comment

GET /purchases/1337/comments

200 OK 401 Unauthorized

should we return data?

customer

account

card purchase

payment

comment

GET /purchases/1337/comments

200 OK 401 Unauthorized

should we return data?

customer

account

card purchase

payment

comment

GET /purchases/1337/comments

200 OK 401 Unauthorized

should we return data?

15

explicit relationship traversal

(defn owns? [customer-id purchase-id db] (d/q '{:find [?pur .] :in [$ ?customer-id ?purchase-id] :where [[?pur :purchase/id ?purchase-id] [?pur :purchase/account ?acc] [?acc :account/customer ?cus] [?cus :customer/id ?customer-id]]} db customer-id purchase-id))

15

explicit relationship traversal

(defn owns? [customer-id purchase-id db] (d/q '{:find [?pur .] :in [$ ?customer-id ?purchase-id] :where [[?pur :purchase/id ?purchase-id] [?pur :purchase/account ?acc] [?acc :account/customer ?cus] [?cus :customer/id ?customer-id]]} db customer-id purchase-id))

generic relationship traversal

16

recursive rule (logical OR)

generic relationship traversal

16

recursive rule (logical OR)

(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])

generic relationship traversal

16

recursive rule (logical OR)

(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])

generic relationship traversal

16

recursive rule (logical OR)

(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])

generic relationship traversal

16

recursive rule (logical OR)

(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])

generic relationship traversal

16

recursive rule (logical OR)

(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])

generic relationship traversal

16

recursive rule (logical OR)

(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])

generic relationship traversal

16

recursive rule (logical OR)

query using rule

(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])

generic relationship traversal

16

recursive rule (logical OR)

query using rule

(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])

(owns? customer-id [:purchase/id id] db)

(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity owner-rules))

generic relationship traversal

16

recursive rule (logical OR)

query using rule

(def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])

(owns? customer-id [:purchase/id id] db)

(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity owner-rules))

17

Recursion rules!

But can we go further?

Can we prevent queries from returning entities owned by

other customers?

all owned entities

18

query binding attribute

all owned entities

18

query binding attribute

(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))

all owned entities

18

query binding attribute

(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))

all owned entities

18

query binding attribute

attribute unbound

(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))

all owned entities

18

query binding attribute

attribute unbound

(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))

(defn owned-entities [customer-id db] (d/q '{:find [[?e ...]] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e)]} db customer-id owner-rules))

all owned entities

18

query binding attribute

attribute unbound

(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))

(defn owned-entities [customer-id db] (d/q '{:find [[?e ...]] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e)]} db customer-id owner-rules))

all owned entities

18

query binding attribute

attribute unbound

(defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))

(defn owned-entities [customer-id db] (d/q '{:find [[?e ...]] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e)]} db customer-id owner-rules))

filtered db (as value)

19

(defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))

filtered db (as value)

19

(defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))

filtered db (as value)

19

(defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))

filtered db (as value)

19

(defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))

filtered db (as value)

19

(defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))

filtered db (as value)

19

filtered db will not contain datoms from other owners!

(defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))

passing a filtered db is transparent for queries

20

(defn all-purchases [account-id db] (d/q '{:find [[(pull ?p [*]) ...]] :in [$ ?acc] :where [[?p :purchase/account ?acc]]} db [:account/id account-id]))

(all-purchases account-id db) (all-purchases account-id (owned-db customer-id db))

passing a filtered db is transparent for queries

20

(defn all-purchases [account-id db] (d/q '{:find [[(pull ?p [*]) ...]] :in [$ ?acc] :where [[?p :purchase/account ?acc]]} db [:account/id account-id]))

(all-purchases account-id db) (all-purchases account-id (owned-db customer-id db))

21

We’re Mobile!

Bandwidth is costly

22

3 HTTP Cache

23

GET /accounts/1234/purchases200 OKLast-Modified: Fri, 14 Nov 2014 14:28:50 UTC{"purchases": [...]}

http last modified header

23

If-Modified-Since: Fri, 14 Nov 2014 14:28:50 UTCGET /accounts/1234/purchases304 Not Modified

GET /accounts/1234/purchases200 OKLast-Modified: Fri, 14 Nov 2014 14:28:50 UTC{"purchases": [...]}

http last modified header

24

How to keep track of a good last-modified date for a URL?

24

How to keep track of a good last-modified date for a URL?

The last time any owned entity changed!

If no customer-owned data changed, 304

25

(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))

If no customer-owned data changed, 304

25

(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))

If no customer-owned data changed, 304

25

(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))

If no customer-owned data changed, 304

25

(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))

If no customer-owned data changed, 304

25

(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))

If no customer-owned data changed, 304

25

(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))

If no customer-owned data changed, 304

25

(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))

If no customer-owned data changed, 304

25

(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))

If no customer-owned data changed, 304

25

(defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))

26

Cache hits are awesome, but what about cache misses?

27

4 Mobile Sync

desired hypermedia from the API

28

GET /accounts/1223/purchases200 OK{"purchases": [...] "_links": { "updates": "https://...?since=2014-11-10T11:12:13Z" }}

serve purchases added or modified since our last sync

29

(defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id]))

(updated-purchases account-id db #inst "2014-11-14"))

serve purchases added or modified since our last sync

29

(defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id]))

(updated-purchases account-id db #inst "2014-11-14"))

serve purchases added or modified since our last sync

29

(defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id]))

(updated-purchases account-id db #inst "2014-11-14"))

serve purchases added or modified since our last sync

29

(defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id]))

(updated-purchases account-id db #inst "2014-11-14"))

serve purchases added or modified since our last sync

29

(defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id]))

(updated-purchases account-id db #inst "2014-11-14"))

serve purchases added or modified since our last sync

29

(defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id]))

(updated-purchases account-id db #inst "2014-11-14"))

serve purchases added or modified since our last sync

29

(defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id]))

(updated-purchases account-id db #inst "2014-11-14"))

30

Bonus

31

5 Future DBs

(defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases))))

generating a virtual db

32

(defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases))))

generating a virtual db

32

(defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases))))

generating a virtual db

32

(defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases))))

generating a virtual db

32

(defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases))))

generating a virtual db

32

33

6 Testing

db testing - locally scoped

34

(defn virtual-db [updates db] (:db-after (d/with db updates)))(let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])

db testing - locally scoped

34

(defn virtual-db [updates db] (:db-after (d/with db updates)))(let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])

db testing - locally scoped

34

(defn virtual-db [updates db] (:db-after (d/with db updates)))(let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])

db testing - locally scoped

34

(defn virtual-db [updates db] (:db-after (d/with db updates)))(let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])

db testing - locally scoped

34

(defn virtual-db [updates db] (:db-after (d/with db updates)))(let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])

db testing - locally scoped

34

(defn virtual-db [updates db] (:db-after (d/with db updates)))(let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])

db testing - vector of datoms

35

(fact "on last-modification-for" (let [customer-id (uuid) db [[42 :account/customer-id customer-id 99] [99 :db/txInstant #inst "2014-10-10T12:00:00.909Z" 99] [1337 :purchase/account 42 100] [1337 :purchase/id (uuid) 100] [1337 :purchase/merchant-name "Coiso" 100] [100 :db/txInstant #inst "2014-10-11T12:00:00.888Z" 100] [9001 :line-item/account 42 101] [9001 :charge/id (uuid) 101] [9001 :line-item/precise-amount 100.00M 101] [101 :db/txInstant #inst "2014-10-12T13:00:00.777Z" 101] [45 :account/customer-id (uuid) 102] [102 :db/txInstant #inst "2014-10-15T12:00:00.909Z" 102]]] (last-modification-for customer-id db) => #nu/time "2014-10-12T13:00:00.777Z"))

db testing - vector of datoms

35

(fact "on last-modification-for" (let [customer-id (uuid) db [[42 :account/customer-id customer-id 99] [99 :db/txInstant #inst "2014-10-10T12:00:00.909Z" 99] [1337 :purchase/account 42 100] [1337 :purchase/id (uuid) 100] [1337 :purchase/merchant-name "Coiso" 100] [100 :db/txInstant #inst "2014-10-11T12:00:00.888Z" 100] [9001 :line-item/account 42 101] [9001 :charge/id (uuid) 101] [9001 :line-item/precise-amount 100.00M 101] [101 :db/txInstant #inst "2014-10-12T13:00:00.777Z" 101] [45 :account/customer-id (uuid) 102] [102 :db/txInstant #inst "2014-10-15T12:00:00.909Z" 102]]] (last-modification-for customer-id db) => #nu/time "2014-10-12T13:00:00.777Z"))

36

7 Schema Extension

schema is data!

37

(d/transact conn [{:db/valueType :db.type/string, :db.install/_attribute :db.part/db, :db/id (d/tempid :db.part/db), :db/cardinality :db.cardinality/one, :db/ident :customer/name }])

:nubank/transform :pii

schema is data!

37

(d/transact conn [{:db/valueType :db.type/string, :db.install/_attribute :db.part/db, :db/id (d/tempid :db.part/db), :db/cardinality :db.cardinality/one, :db/ident :customer/name }])

38

8 Sharding Reads

39

sharding to the transactor

customers

notification

auth

processor

acquisition

accounts

39

sharding to the transactor

customers

notification

auth

processor

acquisition

accounts

customers

1 2 …

39

sharding to the transactor

customers

notification

auth

processor

acquisition

accounts

customers

1 2 …

39

sharding to the transactor

customers

notification

auth

processor

acquisition

accounts

shard-specific peer cache

1

customers

1 2 …

39

sharding to the transactor

customers

notification

auth

processor

acquisition

accounts

shard-specific peer cache

1

inter-shard ACID transactions

2

40

9 Db Aggregation

41

data science aggregation

customers

notification

auth

processor

acquisition

accounts

data science

42

querying multiple databases

(defn multi-join [customer-id cus-db acc-db acq-db ntf-db] (d/q '{:find [...] :in [$cus $acc $acq $ntf ?cus-id] :where [[$cus ?cus :customer/id ?cus-id] [$acc ?acc :account/customer-id ?cus-id] [$acq ?ar :request/customer-id ?cus-id] [$ntf ?evt :event/customer-id ?cus-id] [...]]} cus-db acc-db acq-db ntf-db customer-id))

42

querying multiple databases

(defn multi-join [customer-id cus-db acc-db acq-db ntf-db] (d/q '{:find [...] :in [$cus $acc $acq $ntf ?cus-id] :where [[$cus ?cus :customer/id ?cus-id] [$acc ?acc :account/customer-id ?cus-id] [$acq ?ar :request/customer-id ?cus-id] [$ntf ?evt :event/customer-id ?cus-id] [...]]} cus-db acc-db acq-db ntf-db customer-id))

43

1 Audit Trail

2 Authorization

3 HTTP Cache

4 Mobile Sync

5 Future DBs

6 Testing

7 Schema Extension

8 Sharding Reads

9 Db Aggregation

44

We’re hiring

45

We’re hiring

Lucas Cavalcanti & Edward Wible@lucascslucas@nubank.com.bredward@nubank.com.br

Thanks!