intelligent distributed systems optimizations
TRANSCRIPT
Martin Zapletal @zapletal_martin#ReactiveSummit
Intelligent Distributed SystemOptimizations
@cakesolutions
val in: Array[Int]
var i = 0var result = 0while (i <= in.size) { result += in(i) i += 1}
Motivation
val in: Array[Int]
in.foldLeft(0)(_ + _)
Distributed System Optimizations
● Performance, cost, reliability, uptime● Logical, physical, intelligence, .... Optimized
single machine
Optimized single
machine
Optimized single
machine
Optimized distributed
system
val users = spark.read.json("...").as[User]val posts = spark.read.json("...").as[Post]
users .where(not(not($"active"))) .join(posts, posts("author") === users("id")) .select($"userName", initcap($"title")) .filter($"userName" !== "Mike") .filter(_ => 1 == 1) .groupBy($"userName") .agg(count("*")) .select($"userName", $"count(1)")
Author, text, title
Active, id, age, name, userName
NOT(NOT(Active))
userName, initcap(title)
Username != Mike
userName, count
posts
users
filter
join
project
filter
aggregate
project
Graph Optimizations
Graph Optimizations
posts users
Author, text, title Active, id, age, name, userName
filterActive && Username != Mike
join
project userName
aggregate
project userName, count
posts
usersAuthor, text, title
Active, id, age, name, userName
filter NOT(NOT(Active))
join
project userName, initcap(title)
filter Username != Mike
aggregate
project userName, count
projectprojectAuthor, title Id, userName
Id == author
Id == author
Graph Optimizations
posts users
Author, text, title Active, id, age, name, userName
filterActive && Username != Mike
join
userName
aggregate
project userName, count
posts
usersAuthor, text, title
Active, id, age, name, userName
filter NOT(NOT(Active))
join
project userName, initcap(title)
filter Username != Mike
aggregate
project userName, count
projectprojectAuthor, title Id, userName
Id == author
Id == author
project
Physical Optimizations
posts users
filter
5 billion records, 500 GB
join
projectproject
100 million records 20 GB
t1.value=200 1 million records, 100 MB
Id == author
[3, 8]
Whole Stage Code Generationprivate void agg_doAggregateWithKeys() throws java.io.IOException { agg_hashMap = agg_plan.createHashMap();
while (inputadapter_input.hasNext() && !stopEarly()) { InternalRow inputadapter_row = (InternalRow) inputadapter_input.next(); boolean inputadapter_isNull = inputadapter_row.isNullAt(0); UTF8String inputadapter_value = inputadapter_isNull ? null : (inputadapter_row.getUTF8String(0)); long inputadapter_value1 = inputadapter_row.getLong(1);
UnsafeRow agg_unsafeRowAggBuffer = null; UnsafeRow agg_fastAggBuffer = null;
if (agg_fastAggBuffer == null) { agg_holder.reset();
agg_rowWriter.zeroOutNullBytes();
if (inputadapter_isNull) { agg_rowWriter.setNullAt(0); } else { agg_rowWriter.write(0, inputadapter_value); }
[0, 1, 2, 4, 5, 7]
Manual Optimizations● Coordination avoidance● Single writer principle● Load balancing● Speculative Execution● Colocation● Decoupling● Caching● Publish Subscribe● Batching● Rate Limiting● Low Level Optimizations (memory, object allocation, GC, CPU, IO, parallelization, …)● ...
Machine learning
● Stock market predictions● Product recommendations● Facial recognition● Object recognition● Speech understanding● Self-driving cars● Distributed system optimizations● ...
Observability
● Observability○ Logs○ Metrics○ Traces
● Analytics● Understanding● Actions● Reactive
[9]
Metrics and Monitoring
private[this] val latency = Histogram .build() .name(s"requests_latency_seconds") .help("Request latency in seconds.") .register(registry)
val timer = latency.startTimer()
…
timer.observeDuration()
[9]
Metrics and Monitoring
● Time series database● Aggregate queries
Actor Processing Time (max)
200
150
100
50
0
600
400
200
0
800
600
400
200
0
Running Actors
Processed Messages (1 Minute Rate)
17:30 17:45 18:00 18:15
17:30 17:45 18:00 18:15
17:30 17:45 18:00 18:15
6
4
2
0
Actor mailbox Time (max)
17:30 17:45 18:00 18:15
[10, 11]
Logs and Context
● Automated context
log.logInfo( s"""{ | "traceId"="$traceId", | "userId"="$userId", | "scope"="$scope", | "service"="$service", | "userData"=${userData.asset} | "message"="Don't log like this! Automate context!" |""".stripMargin)
Logs and Context{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"}{ "datetime": "2017-09-20T15:01:28.920Z", "level": "info", "scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler", "traceId": "5760302508966145110", "service": "microservice2", "uid": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "eventPublished": { "someAttribute1": "2", "someOtherAttribute1": "None", "dateTimePublished": "2017-09-20T15:01:28.917Z", "eventType": "updateAsset", "company": "cakeSolutions", "assetDescription": "some asset description", "updateId": "0", "userId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "userDetails": "no way to notice the userId here actually is incorrect", "version": "v1", "data": { "updateAssetData": "http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breachingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xURl.ie18418stretchedbastretchingxm62d1faraway0URl.ieny0expandedNe1longish419Smallr39far%2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxloftynfar%2BreachingtallMooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2a01spun%2Bout50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Shortlinksfaraway78Redirxdrawn%2Bout3Doiop" } }}
Logs and Context{
"datetim
e": "2017-09-20T15:01:28.834Z",
"le
vel":
"info",
"sc
ope": "net.c
akeso
lutions.m
icrose
rvice
1.dispatch
er.Asyn
cRequestD
ispatch
er",
"tra
ceId": "
5760302508966145110",
"se
rvice
": "micr
oservi
ce1",
"use
rId": "
067e6162-3b6f-4ae2-a171-2470b63dff0
0",
"asse
tId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"se
rvice
Url": "h
ttp://c
ake.qa.us-e
ast-1.so
lutions.n
et/micr
oservi
ce2/067e6162-3b6f-4
ae2-a171-2470b63dff00"
}{
"datetim
e": "2017-09-20T15:01:28.920Z",
"le
vel":
"info",
"sc
ope": "net.c
akeso
lutions.m
icrose
rvice
2.dispatch
er.Disp
atchRequestH
andler",
"tra
ceId": "
5760302508966145110",
"se
rvice
": "micr
oservi
ce2",
"use
rId": "
067e6162-3b6f-4ae2-a171-2470b63dff0
0",
"asse
tId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"eve
ntPublished": {
"someAttri
bute1": "2",
"someOtherAttri
bute1": "None",
"dateTimePublish
ed": "2017-09-20T15:01:28.917Z",
"eventTyp
e": "updateAsse
t",
"company":
"cake
Solutions",
"assetDescr
iption": "
some asse
t descr
iption",
"updateId": "0",
"userId
": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"userD
etails": "n
o way to notice
the use
rId here actu
ally is i
ncorre
ct",
"versi
on": "v1
",
"data": {
"updateAsse
tData":
"http://w
ww.longurlm
aker.c
om/go?id=7s80lengthened87EzU
RL11aDecentURLsd
136ShoterLinkdeepSnipURLcfa
r%2Breac
hingt108cNutsh
ellURLgbsp
read%2Bout13EasyURLendurin
gtall0xURl.ie
18418stretch
edbastretch
ingxm62d1faraway0
URl.ie
ny0exp
andedNe1longish419Smallr3
9far%2Bofflo
fty1EzU
RL151Shim1aURLPieaShredURLURL.co
.ukNanoRefendurin
gxlof
tynfar%
2BreachingtallM
ooURL57i09stretch
edk014longish
090rangyn9c5
h5YATUCremoteb1Doiop00dgreat2a01spun%2Bo
ut50301URL2aspread%2Bout01blnkf5
1gangling2101k9far%
2Boff0a0910d0Shortli
nksfaraway7
8Redirxdrawn%2Bout3Doio
p"
}
}
}
"datetime": "
2017-09-20T15:01:28.834Z",
"le
vel":
"info",
"sc
ope": "net.c
akeso
lutions.m
icrose
rvice
1.dispatch
er.Asyn
cRequestD
ispatch
er",
"tra
ceId": "
5760302508966145110",
"se
rvice
": "micr
oservi
ce1",
"use
rId": "
067e6162-3b6f-4ae2-a171-2470b63dff0
0",
"asse
tId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"se
rvice
Url": "h
ttp://c
ake.qa.us-e
ast-1.so
lutions.n
et/micr
oservi
ce2/067e6162-3b6f-4
ae2-a171-2470b63dff00"
}{
"datetim
e": "2017-09-20T15:01:28.834Z",
"le
vel":
"info",
"sc
ope": "net.c
akeso
lutions.m
icrose
rvice
1.dispatch
er.Asyn
cRequestD
ispatch
er",
"tra
ceId": "
5760302508966145110",
"se
rvice
": "micr
oservi
ce1",
"use
rId": "
067e6162-3b6f-4ae2-a171-2470b63dff0
0",
"asse
tId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"se
rvice
Url": "h
ttp://c
ake.
{
"timesta
mp": "2017-09-20T15:01:28.931Z",
"lo
gLevel":
"warn",
"sc
ope": "net.c
akeso
lutions.m
icrose
rvice
1.asset.E
ventParse
r.reco
verFromMiss
ingData",
"tra
ceId": "
5760302508966145110",
"se
rvice
": "micr
oservi
ce1",
"use
rId": "
067e6162-3b6f-4ae2-a171-2470b63dff0
0",
"asse
tId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"m
issingDataFailure": {
"Miss
ingData": {
"failure": {
"property
": "referre
r",
"optio
nal": "tru
e"
}
}
}
}
"datetime": "
2017-09-20T15:01:28.834Z",
"le
vel":
"info",
"sc
ope": "net.c
akeso
lutions.m
icrose
rvice
1.dispatch
er.Asyn
cRequestD
ispatch
er",
"tra
ceId": "
5760302508966145110",
"se
rvice
": "micr
oservi
ce1",
"use
rId": "
067e6162-3b6f-4ae2-a171-2470b63dff0
0",
"asse
tId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"se
rvice
Url": "h
ttp://c
ake.qa.us-e
ast-1.so
lutions.n
et/micr
oservi
ce2/067e6162-3b6f-4
ae2-a171-2470b63dff00"
}{
"datetim
e": "2017-09-20T15:01:28.834Z",
"le
vel":
"info",
"sc
ope": "net.c
akeso
lutions.m
icrose
rvice
1.dispatch
er.Asyn
cRequestD
ispatch
er",
"tra
ceId": "
5760302508966145110",
"se
rvice
": "micr
oservi
ce1",
"use
rId": "
067e6162-3b6f-4ae2-a171-2470b63dff0
0",
"asse
tId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"se
rvice
Url": "h
ttp://c
ake.
{ "dataTime": "2017-09-20T15:01:28.925Z", "level": "info", "scope": "net.cakesolutions.microservice1.asset.EventProcessor",
"traceId": "5760302508966145110", "service": "microservice2", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "assetDetails": "(List(AssetIdentifier(Provider,12341567433.1122-2344,11220303,0128479),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,09320418u7.1092384,p91234,0182374))),(List(AssetIdentifier(Provider,
10293890.10000,12-83,107237977799)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/asse
ts/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-
8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.140350974038424,275420
8,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(Ass
etIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http:/
/example.org/some/other/Descriptor/file/video.mp4,0,0,0))"}{ "dateTime": "2017-09-20T15:01:28.925Z", "logLevel": "info", "scope": "net.cakesolutions.microservice1.asset.EventParser",
"traceId": "9044230965916828392", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "assetDetails": "(List(AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718)),null),(List(AssetIden
tifier(Provider,19487588.140350974038400,2754237,19487588)),AssetUrl(http://cake.qa.solutio
ns.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,Ams
Id(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.14
0350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/
file.mp4,0,0,0)),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),De
scriptorFile(Format,http://example.org/some/other/Descriptor/file/video.mp4,0,0,0))"
}{ "timestamp": "2017-09-20T15:01:28.931Z", "logLevel": "warn", "scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData",
"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "missingDataFailure": { "MissingData": { "failure": { "property": "referrer", "optional": "true" }
} }}
{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}
{ "timestamp": "2017-09-20T15:01:28.931Z", "logLevel": "warn", "scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData",
"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "missingDataFailure": { "MissingData": { "failure": { "property": "referrer", "optional": "true" }
} }}
{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067 "datetime":
"2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.e6162-3b6f-4ae2-a171-2470b63dff00"}
{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"}{ "datetime": "2017-09-20T15:01:28.920Z", "level": "info", "scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler", "traceId": "5760302508966145110", "service": "microservice2", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "eventPublished": { "someAttribute1": "2", "someOtherAttribute1": "None", "dateTimePublished": "2017-09-20T15:01:28.917Z", "eventType": "updateAsset", "company": "cakeSolutions", "assetDescription": "some asset description", "updateId": "0", "userId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "userDetails": "no way to notice the userId here actually is incorrect", "version": "v1", "data": { "updateAssetData": "http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breachingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xURl.ie18418stretchedbastretchingxm62d1faraway0URl.ieny0expandedNe1longish419Smallr39far%2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxloftynfar%2BreachingtallMooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2a01spun%2Bout50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Shortlinksfaraway78Redirxdrawn%2Bout3Doiop" } }}{ "dataTime": "2017-09-20T15:01:28.925Z", "level": "info", "scope": "net.cakesolutions.microservice1.asset.EventProcessor", "traceId": "5760302508966145110", "service": "microservice2", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "assetDetails": "(List(AssetIdentifier(Provider,12341567433.1122-2344,11220303,0128479), AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718), AssetIdentifier(Provider,09320418u7.1092384,p91234,0182374))),(List(AssetIdentifier(Provider,10293890.10000,12-83,107237977799)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.140350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/some/other/Descriptor/file/video.mp4,0,0,0))"}{ "dateTime": "2017-09-20T15:01:28.925Z", "logLevel": "info", "scope": "net.cakesolutions.microservice1.asset.EventParser", "traceId": "9044230965916828392", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "assetDetails": "(List(AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718), AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718), AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718)),null),(List(AssetIdentifier(Provider,19487588.140350974038400,2754237,19487588)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.140350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/some/other/Descriptor/file/video.mp4,0,0,0))"}{ "timestamp": "2017-09-20T15:01:28.931Z", "logLevel": "warn", "scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "missingDataFailure": { "MissingData": { "failure": { "property": "referrer", "optional": "true" } } }}
{ "datetime": "2017-09-20T15:01:28.834Z",
"level": "info",
"scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}{ "datetime": "2017-09-20T15:01:28.920Z",
"level": "info",
"scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler",
"traceId": "5760302508966145110",
"service": "microservice2",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"eventPublished": {
"someAttribute1": "2",
"someOtherAttribute1": "None",
"dateTimePublished": "2017-09-20T15:01:28.917Z",
"eventType": "updateAsset",
"company": "cakeSolutions",
"assetDescription": "some asset description",
"updateId": "0",
"userId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"userDetails": "no way to notice the userId here actually is incorrect",
"version": "v1",
"data": {
"updateAssetData":
"http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breac
hingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xURl.ie18418stretchedbastretchingxm62d1faraway0URl.ie
ny0expandedNe1longish419Smallr39far%2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxlof
tynfar%2BreachingtallMooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2a01spun%2Bo
ut50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Shortlinksfaraway78Redirxdrawn%2Bout3Doio
p" }
}} "datetime": "2017-09-20T15:01:28.834Z",
"level": "info",
"scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}{ "datetime": "2017-09-20T15:01:28.834Z",
"level": "info",
"scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"serviceUrl": "http://cake.
{ "tim
estamp": "2017-09-20T15:01:28.931Z",
"logLevel": "warn",
"scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData",
"traceId": "5760302508966145110",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"missingDataFailure": {
"MissingData": {
"failure": {
"property": "referrer",
"optional": "tru
e"
}
}
}} "datetime": "2017-09-20T15:01:28.834Z",
"level": "info",
"scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}{ "datetime": "2017-09-20T15:01:28.834Z",
"level": "info",
"scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"serviceUrl": "http://cake.
{ "d
atet
ime"
: "20
17-0
9-20
T15:
01:2
8.83
4Z",
"l
evel
": "in
fo",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.d
ispa
tche
r.Asy
ncR
eque
stD
ispa
tche
r",
"t
race
Id":
"576
0302
5089
6614
5110
",
"ser
vice
": "m
icro
serv
ice1
",
"use
rId":
"067
e616
2-3b
6f-4
ae2-
a171
-247
0b63
dff0
0",
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"s
ervi
ceU
rl": "
http
://ca
ke.q
a.us
-eas
t-1.s
olut
ions
.net
/mic
rose
rvic
e2/0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
} { "d
atet
ime"
: "20
17-0
9-20
T15:
01:2
8.92
0Z",
"l
evel
": "in
fo",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e2.d
ispa
tche
r.Dis
patc
hReq
uest
Han
dler
",
"tra
ceId
": "5
7603
0250
8966
1451
10",
"s
ervi
ce":
"mic
rose
rvic
e2",
"u
serId
": "0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
,
"ass
etId
": "5
4947
df8-
0e9e
-447
1-a2
f9-9
af50
9fb5
889"
,
"eve
ntP
ublis
hed"
: {
"
som
eAttr
ibut
e1":
"2",
"so
meO
ther
Attr
ibut
e1":
"Non
e",
"da
teTi
meP
ublis
hed"
: "20
17-0
9-20
T15:
01:2
8.91
7Z",
"ev
entT
ype"
: "up
date
Ass
et",
"co
mpa
ny":
"cak
eSol
utio
ns",
"as
setD
escr
iptio
n": "
som
e as
set d
escr
iptio
n",
"up
date
Id":
"0",
"us
erId
": "5
4947
df8-
0e9e
-447
1-a2
f9-9
af50
9fb5
889"
,
"
user
Det
ails
": "n
o w
ay to
not
ice
the
user
Id h
ere
actu
ally
is in
corr
ect",
"ve
rsio
n": "
v1",
"da
ta":
{
"up
date
Ass
etD
ata"
: "h
ttp://
ww
w.lo
ngur
lmak
er.c
om/g
o?id
=7s8
0len
gthe
ned8
7EzU
RL1
1aD
ecen
tUR
Lsd1
36S
hote
rLin
kdee
pSni
pUR
Lcfa
r%2B
reac
hing
t108
cNut
shel
lUR
Lgbs
prea
d%2B
out1
3Eas
yUR
Lend
urin
gtal
l0xU
Rl.i
e184
18st
retc
hedb
astre
tchi
ngxm
62d1
fara
way
0UR
l.ie
ny0e
xpan
dedN
e1lo
ngis
h419
Sm
allr3
9far
%2B
offlo
fty1E
zUR
L151
Shi
m1a
UR
LPie
aShr
edU
RLU
RL.
co.u
kNan
oRef
endu
ringx
lof
tynf
ar%
2Bre
achi
ngta
llMoo
UR
L57i
09st
retc
hedk
014l
ongi
sh09
0ran
gyn9
c5h5
YA
TUC
rem
oteb
1Doi
op00
dgre
at2a
01sp
un%
2Bo
ut50
301U
RL2
aspr
ead%
2Bou
t01b
lnkf
51ga
nglin
g210
1k9f
ar%
2Bof
f0a0
910d
0Sho
rtlin
ksfa
raw
ay78
Red
irxdr
awn%
2Bou
t3D
oio
p"
}
}} {
"dat
aTim
e": "
2017
-09-
20T1
5:01
:28.
925Z
",
"lev
el":
"info
",
"sco
pe":
"net
.cak
esol
utio
ns.m
icro
serv
ice1
.ass
et.E
vent
Pro
cess
or",
"t
race
Id":
"576
0302
5089
6614
5110
",
"ser
vice
": "m
icro
serv
ice2
",
"use
rId":
"067
e616
2-3b
6f-4
ae2-
a171
-247
0b63
dff0
0",
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"a
sset
Det
ails
": "(
List
(Ass
etId
entif
ier(
Pro
vide
r,123
4156
7433
.112
2-23
44,1
1220
303,
0128
479)
, A
sset
Iden
tifie
r(P
rovi
der,1
9460
718.
1403
5097
4037
952,
2762
537,
1946
0718
), A
sset
Iden
tifie
r(P
rovi
der,0
9320
418u
7.10
9238
4,p9
1234
,018
2374
))),(
List
(Ass
etId
entif
ier(
Pro
vide
r,102
9389
0.10
000,
12-8
3,10
7237
9777
99))
,Ass
etU
rl(ht
tp://
cake
.qa.
solu
tions
.us-
east
-1.n
et/v
4/as
sets
/des
crib
e/ad
8ecd
76-7
36e-
4816
-8c5
9-48
4ba1
d229
a4/
desc
riptio
r,Am
sId(
ad8e
cd76
-736
e-48
16-8
c59-
484b
a1d2
29a4
),0,0
,0))
,(Lis
t(Ass
etId
entif
ier(
Pro
vide
r,194
8756
.140
3509
740
3842
4,27
5420
8,19
4875
6)),D
escr
ipto
rFile
(For
mat
,http
://ex
ampl
e.or
g/so
me/
Des
crip
tor/f
ile.m
p4,0
,0,0
)),(L
ist(A
sset
Iden
tifie
r(P
rovi
der,1
9487
20.1
4035
0974
0383
7,27
5426
4,19
4872
0)),D
escr
ipto
rFile
(For
mat
,http
://ex
ampl
e.or
g/so
me/
othe
r/Des
crip
tor/f
ile/
vide
o.m
p4,0
,0,0
))"
} { "d
ateT
ime"
: "20
17-0
9-20
T15:
01:2
8.92
5Z",
"l
ogLe
vel":
"inf
o",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.a
sset
.Eve
ntP
arse
r",
"t
race
Id":
"904
4230
9659
1682
8392
",
"ser
vice
": "m
icro
serv
ice1
",
"use
rId":
"067
e616
2-3b
6f-4
ae2-
a171
-247
0b63
dff0
0",
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"a
sset
Det
ails
": "(
List
(Ass
etId
entif
ier(
Pro
vide
r,194
6071
8.14
0350
9740
3795
2,27
6253
7,19
4607
18),
Ass
etId
entif
ier(
Pro
vide
r,194
6071
8.14
0350
9740
3795
2,27
6253
7,19
4607
18),
Ass
etId
entif
ier(
Pro
vide
r,194
6071
8.14
0350
9740
3795
2,27
6253
7,19
4607
18))
,nul
l),(L
ist(A
sset
Iden
tifie
r(P
rovi
der,1
9487
588.
1403
5097
4038
400,
2754
237,
1948
7588
)),A
sset
Url(
http
://ca
ke.q
a.so
lutio
ns.u
s-ea
st-1
.net
/v4/
asse
ts/d
escr
ibe/
ad8e
cd76
-736
e-4
816-
8c59
-484
ba1d
229a
4/de
scrip
tior,A
msI
d(ad
8ecd
76-7
36e-
4816
-8c5
9-48
4ba1
d229
a4),0
,0,0
)),(L
ist(A
sset
Iden
tifie
r(P
rovi
der,1
9487
56.1
4035
0974
0384
24,2
7542
08,1
9487
56))
,Des
crip
torF
ile(F
orm
at,h
ttp://
exam
ple.
org/
som
e/D
escr
ipto
r/file
.mp4
,0,0
,0)
),(Li
st(A
sset
Iden
tifie
r(P
rovi
der,1
9487
20.1
4035
0974
0383
7,27
5426
4,19
4872
0)),D
escr
ipto
rFile
(For
mat
,http
://ex
ampl
e.or
g/so
me/
othe
r/Des
crip
tor/f
ile/v
ideo
.mp4
,0,0
,0))
"} {
"tim
esta
mp"
: "20
17-0
9-20
T15:
01:2
8.93
1Z",
"l
ogLe
vel":
"war
n",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.a
sset
.Eve
ntP
arse
r.rec
over
From
Mis
sing
Dat
a",
"t
race
Id":
"576
0302
5089
6614
5110
",
"ser
vice
": "m
icro
serv
ice1
",
"use
rId":
"067
e616
2-3b
6f-4
ae2-
a171
-247
0b63
dff0
0",
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"m
issi
ngD
ataF
ailu
re":
{
"
Mis
sing
Dat
a": {
"f
ailu
re":
{
"p
rope
rty":
"ref
erre
r",
"opt
iona
l": "t
rue"
}
}
}}
{ "datetime": "2017-09-20T15:01:28.834Z",
"level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}{
"datetime": "2017-09-20T15:01:28.920Z",
"level": "info", "scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler",
"traceId": "5760302508966145110",
"service": "microservice2",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"eventPublished": { "someAttribute1": "2",
"someOtherAttribute1": "None",
"dateTimePublished": "2017-09-20T15:01:28.917Z",
"eventType": "updateAsset",
"company": "cakeSolutions",
"assetDescription": "some asset description",
"updateId": "0", "userId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"userDetails": "no way to notice the userId here actually is incorrect",
"version": "v1", "data": { "updateAssetData":
"http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breac
hingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xURl.ie18418stretchedbastretchingxm62d1faraway0URl.ie
ny0expandedNe1longish419Smallr39far%2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxlof
tynfar%2BreachingtallMooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2a01spun%2Bo
ut50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Shortlinksfaraway78Redirxdrawn%2Bout3Doio
p" } }
}{
"dataTime": "2017-09-20T15:01:28.925Z",
"level": "info", "scope": "net.cakesolutions.microservice1.asset.EventProcessor",
"traceId": "5760302508966145110",
"service": "microservice2",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"assetDetails": "(List(AssetIdentifier(Provider,12341567433.1122-2344,11220303,0128479),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,09320418u7.1092384,p91234,0182374))),(List(AssetIdentifier(Provider,10293890.10000,12-83,10
7237977799)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a
4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.1403509740
38424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(AssetIdentifier(P
rovider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/some/other/Descriptor/file/
video.mp4,0,0,0))"}
{ "dateTime": "2017-09-20T15:01:28.925Z",
"logLevel": "info", "scope": "net.cakesolutions.microservice1.asset.EventParser",
"traceId": "9044230965916828392",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"assetDetails": "(List(AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718)),null),(List(AssetIdentifier(Provider,19487588.14
0350974038400,2754237,19487588)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4
816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provide
r,1948756.140350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)
),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/so
me/other/Descriptor/file/video.mp4,0,0,0))"
}{
"timestamp": "2017-09-20T15:01:28.931Z",
"logLevel": "warn", "scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData",
"traceId": "5760302508966145110",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"missingDataFailure": {
"MissingData": { "failure": {
"property": "referrer",
"optional": "true"
} } }}
{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}
{ "timestamp": "2017-09-20T15:01:28.931Z", "logLevel": "warn", "scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "missingDataFailure": { "MissingData": { "failure": { "property": "referrer", "optional": "true" } } }}
{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"}{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067 "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"}{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.e6162-3b6f-4ae2-a171-2470b63dff00"
}
{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"}{ "datetime": "2017-09-20T15:01:28.920Z", "level": "info", "scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler", "traceId": "5760302508966145110", "service": "microservice2", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "eventPublished": { "someAttribute1": "2", "someOtherAttribute1": "None", "dateTimePublished": "2017-09-20T15:01:28.917Z", "eventType": "updateAsset", "company": "cakeSolutions", "assetDescription": "some asset description", "updateId": "0", "userId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "userDetails": "no way to notice the userId here actually is incorrect", "version": "v1", "data": { "updateAssetData": "http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breachingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xURl.ie18418stretchedbastretchingxm62d1faraway0URl.ieny0expandedNe1longish419Smallr39far%2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxloftynfar%2BreachingtallMooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2a01spun%2Bout50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Shortlinksfaraway78Redirxdrawn%2Bout3Doiop" } }}{ "dataTime": "2017-09-20T15:01:28.925Z", "level": "info", "scope": "net.cakesolutions.microservice1.asset.EventProcessor", "traceId": "5760302508966145110", "service": "microservice2", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "assetDetails": "(List(AssetIdentifier(Provider,12341567433.1122-2344,11220303,0128479), AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718), AssetIdentifier(Provider,09320418u7.1092384,p91234,0182374))),(List(AssetIdentifier(Provider,10293890.10000,12-83,107237977799)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.140350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/some/other/Descriptor/file/video.mp4,0,0,0))"}{ "dateTime": "2017-09-20T15:01:28.925Z", "logLevel": "info", "scope": "net.cakesolutions.microservice1.asset.EventParser", "traceId": "9044230965916828392", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "assetDetails": "(List(AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718), AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718), AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718)),null),(List(AssetIdentifier(Provider,19487588.140350974038400,2754237,19487588)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.140350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/some/other/Descriptor/file/video.mp4,0,0,0))"}{ "timestamp": "2017-09-20T15:01:28.931Z", "logLevel": "warn", "scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "missingDataFailure": { "MissingData": { "failure": { "property": "referrer", "optional": "true" } } }}
{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}{ "datetime": "2017-09-20T15:01:28.920Z", "level": "info", "scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler",
"traceId": "5760302508966145110", "service": "microservice2", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"eventPublished": { "someAttribute1": "2", "someOtherAttribute1": "None", "dateTimePublished": "2017-09-20T15:01:28.917Z",
"eventType": "updateAsset", "company": "cakeSolutions", "assetDescription": "some asset description",
"updateId": "0", "userId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"userDetails": "no way to notice the userId here actually is incorrect",
"version": "v1", "data": { "updateAssetData": "http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breac
hingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xURl.ie18418stretchedbastretchingxm62d1faraway0URl.ie
ny0expandedNe1longish419Smallr39far%2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxlof
tynfar%2BreachingtallMooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2a01spun%2Bo
ut50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Shortlinksfaraway78Redirxdrawn%2Bout3Doio
p" } }}{ "dataTime": "2017-09-20T15:01:28.925Z", "level": "info", "scope": "net.cakesolutions.microservice1.asset.EventProcessor",
"traceId": "5760302508966145110", "service": "microservice2", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"assetDetails": "(List(AssetIdentifier(Provider,12341567433.1122-2344,11220303,0128479),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,09320418u7.1092384,p91234,0182374))),(List(AssetIdentifier(Provider,10293890.10000,12-83,10
7237977799)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a
4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.1403509740
38424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(AssetIdentifier(P
rovider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/some/other/Descriptor/file/
video.mp4,0,0,0))"}{ "dateTime": "2017-09-20T15:01:28.925Z", "logLevel": "info", "scope": "net.cakesolutions.microservice1.asset.EventParser",
"traceId": "9044230965916828392", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"assetDetails": "(List(AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718)),null),(List(AssetIdentifier(Provider,19487588.14
0350974038400,2754237,19487588)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4
816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provide
r,1948756.140350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)
),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/so
me/other/Descriptor/file/video.mp4,0,0,0))"}{ "timestamp": "2017-09-20T15:01:28.931Z",
"logLevel": "warn", "scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData",
"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"missingDataFailure": { "MissingData": { "failure": { "property": "referrer", "optional": "true" } } }}
{
"datetime": "
2017-09-20T15:01:28.834Z",
"le
vel":
"info",
"sc
ope": "net.c
akeso
lutions.m
icrose
rvice
1.dispatch
er.Asyn
cRequestD
ispatch
er",
"tra
ceId": "
5760302508966145110",
"se
rvice
": "micr
oservi
ce1",
"use
rId": "
067e6162-3b6f-4ae2-a171-2470b63dff0
0",
"asse
tId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"se
rvice
Url": "h
ttp://c
ake.qa.us-e
ast-1.so
lutions.n
et/micr
oservi
ce2/067e6162-3b6f-4
ae2-a171-2470b63dff00"
}{
"datetim
e": "2017-09-20T15:01:28.920Z",
"le
vel":
"info",
"sc
ope": "net.c
akeso
lutions.m
icrose
rvice
2.dispatch
er.Disp
atchRequestH
andler",
"tra
ceId": "
5760302508966145110",
"se
rvice
": "micr
oservi
ce2",
"use
rId": "
067e6162-3b6f-4ae2-a171-2470b63dff0
0",
"asse
tId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"eve
ntPublished": {
"someAttri
bute1": "2",
"someOtherAttri
bute1": "None",
"dateTimePublish
ed": "2017-09-20T15:01:28.917Z",
"eventTyp
e": "updateAsse
t",
"company":
"cake
Solutions",
"assetDescr
iption": "
some asse
t descr
iption",
"updateId": "0",
"userId
": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"userD
etails": "n
o way to notice
the use
rId here actu
ally is i
ncorre
ct",
"versi
on": "v1
",
"data": {
"updateAsse
tData":
"http://w
ww.longurlm
aker.c
om/go?id=7s80lengthened87EzU
RL11aDecentURLsd
136ShoterLinkdeepSnipURLcfa
r%2Breac
hingt108cNutsh
ellURLgbsp
read%2Bout13EasyURLendurin
gtall0xURl.ie
18418stretch
edbastretch
ingxm62d1faraway0
URl.ie
ny0exp
andedNe1longish419Smallr3
9far%2Bofflo
fty1EzU
RL151Shim1aURLPieaShredURLURL.co
.ukNanoRefendurin
gxlof
tynfar%
2BreachingtallM
ooURL57i09stretch
edk014longish
090rangyn9c5
h5YATUCremoteb1Doiop00dgreat2a01spun%2Bo
ut50301URL2aspread%2Bout01blnkf5
1gangling2101k9far%
2Boff0a0910d0Shortli
nksfaraway7
8Redirxdrawn%2Bout3Doio
p"
}
}
}
"datetime": "
2017-09-20T15:01:28.834Z",
"le
vel":
"info",
"sc
ope": "net.c
akeso
lutions.m
icrose
rvice
1.dispatch
er.Asyn
cRequestD
ispatch
er",
"tra
ceId": "
5760302508966145110",
"se
rvice
": "micr
oservi
ce1",
"use
rId": "
067e6162-3b6f-4ae2-a171-2470b63dff0
0",
"asse
tId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"se
rvice
Url": "h
ttp://c
ake.qa.us-e
ast-1.so
lutions.n
et/micr
oservi
ce2/067e6162-3b6f-4
ae2-a171-2470b63dff00"
}{
"datetim
e": "2017-09-20T15:01:28.834Z",
"le
vel":
"info",
"sc
ope": "net.c
akeso
lutions.m
icrose
rvice
1.dispatch
er.Asyn
cRequestD
ispatch
er",
"tra
ceId": "
5760302508966145110",
"se
rvice
": "micr
oservi
ce1",
"use
rId": "
067e6162-3b6f-4ae2-a171-2470b63dff0
0",
"asse
tId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"se
rvice
Url": "h
ttp://c
ake.
{
"timesta
mp": "2017-09-20T15:01:28.931Z",
"lo
gLevel":
"warn",
"sc
ope": "net.c
akeso
lutions.m
icrose
rvice
1.asset.E
ventParse
r.reco
verFromMiss
ingData",
"tra
ceId": "
5760302508966145110",
"se
rvice
": "micr
oservi
ce1",
"use
rId": "
067e6162-3b6f-4ae2-a171-2470b63dff0
0",
"asse
tId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"m
issingDataFailure": {
"Miss
ingData": {
"failure": {
"property
": "referre
r",
"optio
nal": "tru
e"
}
}
}
}
"datetime": "
2017-09-20T15:01:28.834Z",
"le
vel":
"info",
"sc
ope": "net.c
akeso
lutions.m
icrose
rvice
1.dispatch
er.Asyn
cRequestD
ispatch
er",
"tra
ceId": "
5760302508966145110",
"se
rvice
": "micr
oservi
ce1",
"use
rId": "
067e6162-3b6f-4ae2-a171-2470b63dff0
0",
"asse
tId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"se
rvice
Url": "h
ttp://c
ake.qa.us-e
ast-1.so
lutions.n
et/micr
oservi
ce2/067e6162-3b6f-4
ae2-a171-2470b63dff00"
}{
"datetim
e": "2017-09-20T15:01:28.834Z",
"le
vel":
"info",
"sc
ope": "net.c
akeso
lutions.m
icrose
rvice
1.dispatch
er.Asyn
cRequestD
ispatch
er",
"tra
ceId": "
5760302508966145110",
"se
rvice
": "micr
oservi
ce1",
"use
rId": "
067e6162-3b6f-4ae2-a171-2470b63dff0
0",
"asse
tId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"se
rvice
Url": "h
ttp://c
ake.
{ "
datetime": "
2017-09-20T15:01:28.834Z",
"level": "
info",
"scope": "
net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "
5760302508966145110",
"service": "
microservice1",
"userId": "
067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"serviceUrl":
"http://cake.qa.us-east-1.solutions.net/m
icroservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}{
"datetim
e": "2017-09-20T15:01:28.920Z",
"level": "
info",
"scope": "
net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler",
"traceId": "
5760302508966145110",
"service": "
microservice2",
"userId": "
067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"eventPublished": {
"s
omeAttribute1": "
2",
"s
omeOtherAttribute1": "
None",
"d
ateTimePublished": "2017-09-20T15:01:28.917Z",
"e
ventType": "updateAsset",
"c
ompany": "cakeSolutions",
"a
ssetDescription": "some asset description",
"u
pdateId": "0",
"u
serId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"u
serDetails": "no way to notice the userId here actually is incorre
ct",
"v
ersion": "v1",
"d
ata": {
"updateAssetData":
"http://www.longurlm
aker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breac
hingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xURl.ie18418stretchedbastretchingxm62d1faraway0URl.ie
ny0expandedNe1longish419Smallr39far%
2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxlof
tynfar%2BreachingtallM
ooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2a01spun%2Bo
ut50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Shortlin
ksfaraway78Redirxdrawn%2Bout3Doio
p"
}
}}
"datetim
e": "2017-09-20T15:01:28.834Z",
"level": "
info",
"scope": "
net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "
5760302508966145110",
"service": "
microservice1",
"userId": "
067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"serviceUrl":
"http://cake.qa.us-east-1.solutions.net/m
icroservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}{
"datetim
e": "2017-09-20T15:01:28.834Z",
"level": "
info",
"scope": "
net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "
5760302508966145110",
"service": "
microservice1",
"userId": "
067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"importantPresentationMessage": "
It is very simple to miss things in such amount of data. I a
m pretty sure noone will
ever read this message. R
elationships, context and the overall data can be used to im
prove the system as a whole.
Consider for example understanding failures, predicting failures, understanding system and its
behaviour in specific
context
and much more. Also, I h
ope you have a great day!"
{ "
timestamp": "
2017-09-20T15:01:28.931Z",
"logLevel": "
warn",
"scope": "
net.cakesolutions.microservice1.asset.EventParser.re
coverFromMissingData",
"traceId": "
5760302508966145110",
"service": "
microservice1",
"userId": "
067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"missingDataFailure": {
"M
issingData": {
"failure": {
"property": "
referrer",
"optional": "
true"
}
}
}}
"datetim
e": "2017-09-20T15:01:28.834Z",
"level": "
info",
"scope": "
net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "
5760302508966145110",
"service": "
microservice1",
"userId": "
067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"serviceUrl":
"http://cake.qa.us-east-1.solutions.net/m
icroservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}{
"datetim
e": "2017-09-20T15:01:28.834Z",
"level": "
info",
"scope": "
net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "
5760302508966145110",
"service": "
microservice1",
"userId": "
067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "
54947df8-0e9e-4471-a2f9-9af509fb5889",
"serviceUrl":
"http://cake.
{ "datetime": "2017-09-20T15:01:28.834Z",
"level": "info",
"scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}{
"datetime": "2017-09-20T15:01:28.920Z",
"level": "info",
"scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler",
"traceId": "5760302508966145110",
"service": "microservice2",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"eventPublished": {
"someAttribute1": "2",
"someOtherAttribute1": "None",
"dateTimePublished": "2017-09-20T15:01:28.917Z",
"eventType": "updateAsset",
"company": "cakeSolutions",
"assetDescription": "some asset description",
"updateId": "0",
"userId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"userDetails": "no way to notice the userId here actually is incorrect",
"version": "v1",
"data": {
"updateAssetData":
"http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breac
hingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xURl.ie18418stretchedbastretchingxm62d1faraway0URl.ie
ny0expandedNe1longish419Smallr39far%2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxlof
tynfar%2BreachingtallMooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2a01spun%2Bo
ut50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Shortlinksfaraway78Redirxdrawn%2Bout3Doio
p" } }}
{ "dataTime": "2017-09-20T15:01:28.925Z",
"level": "info",
"scope": "net.cakesolutions.microservice1.asset.EventProcessor",
"traceId": "5760302508966145110",
"service": "microservice2",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"assetDetails": "(List(AssetIdentifier(Provider,12341567433.1122-2344,11220303,0128479),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,09320418u7.1092384,p91234,0182374))),(List(AssetIdentifier(Provider,10293890.10000,12-83,10
7237977799)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a
4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.1403509740
38424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(AssetIdentifier(P
rovider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/some/other/Descriptor/file/
video.mp4,0,0,0))"
}{
"dateTime": "2017-09-20T15:01:28.925Z",
"logLevel": "info",
"scope": "net.cakesolutions.microservice1.asset.EventParser",
"traceId": "9044230965916828392",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"assetDetails": "(List(AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718)),null),(List(AssetIdentifier(Provider,19487588.14
0350974038400,2754237,19487588)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4
816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provide
r,1948756.140350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)
),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/so
me/other/Descriptor/file/video.mp4,0,0,0))"
}{
"timestamp": "2017-09-20T15:01:28.931Z",
"logLevel": "warn",
"scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData",
"traceId": "5760302508966145110",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"missingDataFailure": {
"MissingData": {
"failure": {
"property": "referrer",
"optional": "true"
} } }}
{ "d
atet
ime"
: "20
17-0
9-20
T15:
01:2
8.83
4Z",
"l
evel
": "in
fo",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.d
ispa
tche
r.Asy
ncR
eque
stD
ispa
tche
r",
"t
race
Id":
"576
0302
5089
6614
5110
",
"s
ervi
ce":
"mic
rose
rvic
e1",
"u
serId
": "0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
,
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"s
ervi
ceU
rl":
"http
://ca
ke.q
a.us
-eas
t-1.s
olut
ions
.net
/mic
rose
rvic
e2/0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
} { "t
imes
tam
p": "
2017
-09-
20T1
5:01
:28.
931Z
",
"l
ogLe
vel":
"war
n",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.a
sset
.Eve
ntPa
rser
.reco
verF
rom
Mis
sing
Dat
a",
"t
race
Id":
"576
0302
5089
6614
5110
",
"s
ervi
ce":
"mic
rose
rvic
e1",
"u
serId
": "0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
,
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"m
issi
ngD
ataF
ailu
re":
{
"M
issi
ngD
ata"
: {
"f
ailu
re":
{
"pro
perty
": "r
efer
rer"
,
"opt
iona
l": "t
rue"
}
}
}} {
"dat
etim
e": "
2017
-09-
20T1
5:01
:28.
834Z
",
"l
evel
": "in
fo",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.d
ispa
tche
r.Asy
ncR
eque
stD
ispa
tche
r",
"t
race
Id":
"576
0302
5089
6614
5110
",
"s
ervi
ce":
"mic
rose
rvic
e1",
"u
serId
": "0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
,
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"s
ervi
ceU
rl":
"http
://ca
ke.q
a.us
-eas
t-1.s
olut
ions
.net
/mic
rose
rvic
e2/0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
} { "d
atet
ime"
: "20
17-0
9-20
T15:
01:2
8.83
4Z",
"l
evel
": "in
fo",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.d
ispa
tche
r.Asy
ncR
eque
stD
ispa
tche
r",
"t
race
Id":
"576
0302
5089
6614
5110
",
"s
ervi
ce":
"mic
rose
rvic
e1",
"u
serId
": "0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
,
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"s
ervi
ceU
rl": "
http
://ca
ke.q
a.us
-eas
t-1.s
olut
ions
.net
/mic
rose
rvic
e2/0
67
"dat
etim
e":
"201
7-09
-20T
15:0
1:28
.834
Z",
"l
evel
": "in
fo",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.d
ispa
tche
r.Asy
ncR
eque
stD
ispa
tche
r",
"t
race
Id":
"576
0302
5089
6614
5110
",
"s
ervi
ce":
"mic
rose
rvic
e1",
"u
serId
": "0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
,
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"s
ervi
ceU
rl":
"http
://ca
ke.q
a.us
-eas
t-1.s
olut
ions
.net
/mic
rose
rvic
e2/0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
} { "d
atet
ime"
: "20
17-0
9-20
T15:
01:2
8.83
4Z",
"l
evel
": "in
fo",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.d
ispa
tche
r.Asy
ncR
eque
stD
ispa
tche
r",
"t
race
Id":
"576
0302
5089
6614
5110
",
"s
ervi
ce":
"mic
rose
rvic
e1",
"u
serId
": "0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
,
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"s
ervi
ceU
rl": "
http
://ca
ke.e
6162
-3b6
f-4ae
2-a1
71-2
470b
63df
f00"
}
{ "datetime": "2017-09-20T15:01:28.834Z",
"level": "info",
"scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"serviceUrl":
"http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}{ "datetime": "2017-09-20T15:01:28.920Z",
"level": "info",
"scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler",
"traceId": "5760302508966145110",
"service": "microservice2",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"eventPublished": {
"someAttribute1": "2",
"someOtherAttribute1": "None",
"dateTimePublished": "2017-09-20T15:01:28.917Z",
"eventType": "updateAsset",
"company": "cakeSolutions",
"assetDescription": "some asset description",
"updateId": "0",
"userId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"userDetails": "no way to notice the userId here actually is incorrect",
"version": "v1",
"data": {
"updateAssetData":
"http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLink
deepSnipURLcfar%2Breachingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xU
Rl.ie18418stretchedbastretchingxm62d1faraway0URl.ieny0expandedNe1longish419Smallr39fa
r%2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxloftynfar%2Bre
achingtallMooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2
a01spun%2Bout50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Short
linksfaraway78Redirxdrawn%2Bout3Doiop"
}
}}{ "dataTime": "2017-09-20T15:01:28.925Z",
"level": "info",
"scope": "net.cakesolutions.microservice1.asset.EventProcessor",
"traceId": "5760302508966145110",
"service": "microservice2",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"assetDetails": "(List(AssetIdentifier(Provider,12341567433.1122-2344,11220303,0128479),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,09320418u7.1092384,p91234,0182374))),(List(AssetIdentifier(Provider,
10293890.10000,12-83,107237977799)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/asse
ts/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-
8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.140350974038424,275420
8,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(Ass
etIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http:/
/example.org/some/other/Descriptor/file/video.mp4,0,0,0))"
}{ "dateTime": "2017-09-20T15:01:28.925Z",
"logLevel": "info",
"scope": "net.cakesolutions.microservice1.asset.EventParser",
"traceId": "9044230965916828392",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"assetDetails":
"(List(AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718)),null),(List(AssetIden
tifier(Provider,19487588.140350974038400,2754237,19487588)),AssetUrl(http://cake.qa.solutio
ns.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,Ams
Id(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.14
0350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/
file.mp4,0,0,0)),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),De
scriptorFile(Format,http://example.org/some/other/Descriptor/file/video.mp4,0,0,0))"
}{ "timestamp": "2017-09-20T15:01:28.931Z",
"logLevel": "warn",
"scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData",
"traceId": "5760302508966145110",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"missingDataFailure": {
"MissingData": {
"failure": {
"property": "referrer",
"optional": "true"
}
}
}}
{ "d
atet
ime"
: "20
17-0
9-20
T15:
01:2
8.83
4Z",
"l
evel
": "in
fo",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.d
ispa
tche
r.Asy
ncR
eque
stD
ispa
tche
r",
"t
race
Id":
"576
0302
5089
6614
5110
",
"s
ervi
ce":
"mic
rose
rvic
e1",
"u
serId
": "0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
,
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"s
ervi
ceU
rl": "
http
://ca
ke.q
a.us
-eas
t-1.s
olut
ions
.net
/mic
rose
rvic
e2/0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
} { "d
atet
ime"
: "20
17-0
9-20
T15:
01:2
8.92
0Z",
"l
evel
": "in
fo",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e2.d
ispa
tche
r.Dis
patc
hReq
uest
Han
dler
",
"t
race
Id":
"576
0302
5089
6614
5110
",
"s
ervi
ce":
"mic
rose
rvic
e2",
"u
serId
": "0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
,
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"e
vent
Publ
ishe
d": {
"so
meA
ttrib
ute1
": "2
",
"so
meO
ther
Attri
bute
1": "
Non
e",
"da
teTi
meP
ublis
hed"
: "20
17-0
9-20
T15:
01:2
8.91
7Z",
"ev
entT
ype"
: "up
date
Asse
t",
"co
mpa
ny":
"cak
eSol
utio
ns",
"as
setD
escr
iptio
n": "
som
e as
set d
escr
iptio
n",
"up
date
Id":
"0",
"us
erId
": "5
4947
df8-
0e9e
-447
1-a2
f9-9
af50
9fb5
889"
,
"us
erD
etai
ls":
"no
way
to n
otic
e th
e us
erId
her
e ac
tual
ly is
inco
rrect
",
"ve
rsio
n": "
v1",
"da
ta":
{
"
upda
teAs
setD
ata"
:
"http
://w
ww
.long
urlm
aker
.com
/go?
id=7
s80l
engt
hene
d87E
zUR
L11a
Dec
entU
RLs
d136
Shot
erLi
nkde
epSn
ipU
RLc
far%
2Bre
ac
hing
t108
cNut
shel
lUR
Lgbs
prea
d%2B
out1
3Eas
yUR
Lend
urin
gtal
l0xU
Rl.i
e184
18st
retc
hedb
astre
tchi
ngxm
62d1
fara
way
0UR
l.ie
ny0e
xpan
dedN
e1lo
ngis
h419
Smal
lr39f
ar%
2Bof
flofty
1EzU
RL1
51Sh
im1a
UR
LPie
aShr
edU
RLU
RL.
co.u
kNan
oRef
endu
ringx
lof
tynf
ar%
2Bre
achi
ngta
llMoo
UR
L57i
09st
retc
hedk
014l
ongi
sh09
0ran
gyn9
c5h5
YATU
Cre
mot
eb1D
oiop
00dg
reat
2a01
spun
%2B
o
ut50
301U
RL2
aspr
ead%
2Bou
t01b
lnkf
51ga
nglin
g210
1k9f
ar%
2Bof
f0a0
910d
0Sho
rtlin
ksfa
raw
ay78
Red
irxdr
awn%
2Bou
t3D
oio
p"
}
}}
"dat
etim
e": "
2017
-09-
20T1
5:01
:28.
834Z
",
"l
evel
": "in
fo",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.d
ispa
tche
r.Asy
ncR
eque
stD
ispa
tche
r",
"t
race
Id":
"576
0302
5089
6614
5110
",
"s
ervi
ce":
"mic
rose
rvic
e1",
"u
serId
": "0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
,
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"s
ervi
ceU
rl": "
http
://ca
ke.q
a.us
-eas
t-1.s
olut
ions
.net
/mic
rose
rvic
e2/0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
} { "d
atet
ime"
: "20
17-0
9-20
T15:
01:2
8.83
4Z",
"l
evel
": "in
fo",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.d
ispa
tche
r.Asy
ncR
eque
stD
ispa
tche
r",
"t
race
Id":
"576
0302
5089
6614
5110
",
"s
ervi
ce":
"mic
rose
rvic
e1",
"u
serId
": "0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
,
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"s
ervi
ceU
rl": "
http
://ca
ke.
{ "t
imes
tam
p": "
2017
-09-
20T1
5:01
:28.
931Z
",
"l
ogLe
vel":
"war
n",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.a
sset
.Eve
ntPa
rser
.reco
verF
rom
Mis
sing
Dat
a",
"t
race
Id":
"576
0302
5089
6614
5110
",
"s
ervi
ce":
"mic
rose
rvic
e1",
"u
serId
": "0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
,
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"m
issi
ngD
ataF
ailu
re":
{
"M
issi
ngD
ata"
: {
"f
ailu
re":
{
"pro
perty
": "r
efer
rer"
,
"opt
iona
l": "t
rue"
}
}
}}
"dat
etim
e": "
2017
-09-
20T1
5:01
:28.
834Z
",
"l
evel
": "in
fo",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.d
ispa
tche
r.Asy
ncR
eque
stD
ispa
tche
r",
"t
race
Id":
"576
0302
5089
6614
5110
",
"s
ervi
ce":
"mic
rose
rvic
e1",
"u
serId
": "0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
,
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"s
ervi
ceU
rl": "
http
://ca
ke.q
a.us
-eas
t-1.s
olut
ions
.net
/mic
rose
rvic
e2/0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
} { "d
atet
ime"
: "20
17-0
9-20
T15:
01:2
8.83
4Z",
"l
evel
": "in
fo",
"s
cope
": "n
et.c
akes
olut
ions
.mic
rose
rvic
e1.d
ispa
tche
r.Asy
ncR
eque
stD
ispa
tche
r",
"t
race
Id":
"576
0302
5089
6614
5110
",
"s
ervi
ce":
"mic
rose
rvic
e1",
"u
serId
": "0
67e6
162-
3b6f
-4ae
2-a1
71-2
470b
63df
f00"
,
"a
sset
Id":
"549
47df
8-0e
9e-4
471-
a2f9
-9af
509f
b588
9",
"s
ervi
ceU
rl": "
http
://ca
ke.
{ "datetime": "2017-09-20T15:01:28.834Z",
"level": "info",
"scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",
"traceId": "5760302508966145110",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"
}{ "datetime": "2017-09-20T15:01:28.920Z",
"level": "info",
"scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler",
"traceId": "5760302508966145110",
"service": "microservice2",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"eventPublished": {
"someAttribute1": "2",
"someOtherAttribute1": "None",
"dateTimePublished": "2017-09-20T15:01:28.917Z",
"eventType": "updateAsset",
"company": "cakeSolutions",
"assetDescription": "some asset description",
"updateId": "0",
"userId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"userDetails": "no way to notice the userId here actually is incorrect",
"version": "v1",
"data": {
"updateAssetData":
"http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breac
hingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xURl.ie18418stretchedbastretchingxm62d1faraway0URl.ie
ny0expandedNe1longish419Smallr39far%2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxlof
tynfar%2BreachingtallMooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2a01spun%2Bo
ut50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Shortlinksfaraway78Redirxdrawn%2Bout3Doio
p" }
}}{ "dataTime": "2017-09-20T15:01:28.925Z",
"level": "info",
"scope": "net.cakesolutions.microservice1.asset.EventProcessor",
"traceId": "5760302508966145110",
"service": "microservice2",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"assetDetails": "(List(AssetIdentifier(Provider,12341567433.1122-2344,11220303,0128479),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,09320418u7.1092384,p91234,0182374))),(List(AssetIdentifier(Provider,10293890.10000,12-83,10
7237977799)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a
4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.1403509740
38424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(AssetIdentifier(P
rovider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/some/other/Descriptor/file/
video.mp4,0,0,0))"
}{ "dateTime": "2017-09-20T15:01:28.925Z",
"logLevel": "info",
"scope": "net.cakesolutions.microservice1.asset.EventParser",
"traceId": "9044230965916828392",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"assetDetails": "(List(AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),
AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718)),null),(List(AssetIdentifier(Provider,19487588.14
0350974038400,2754237,19487588)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4
816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provide
r,1948756.140350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)
),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/so
me/other/Descriptor/file/video.mp4,0,0,0))"
}{ "tim
estamp": "2017-09-20T15:01:28.931Z",
"logLevel": "warn",
"scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData",
"traceId": "5760302508966145110",
"service": "microservice1",
"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",
"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",
"missingDataFailure": {
"MissingData": {
"failure": {
"property": "referrer",
"optional": "tru
e"
}
}
}}
Traces
[12]
clientservice1
service1service1service1
service1service1service1
service2
service2
service2
service2
service2
service2service2
service2
service2
service2
service3
181.126ms : client-calls-service1180.527ms : service1-internal-method1
605µs : service1-internal-method2
54.152ms : service1-internal-method3
394µs : service1-internal-method4
46µs : service1-internal-method540.910ms : service1-internal-method6
1.000ms : service1-internal-method7
41.194ms : service2-internal-method1
32.659ms : service2-internal-method2
12.489ms : service2-internal-method3
11.492ms : service2-internal-method4105µs : service2-internal-method5
11.494ms : service2-internal-method610.511ms : service2-internal-method7
85µs : service2-internal-method8
29.816ms : service2-internal-method912.153ms : service2-internal-method10
9.712ms : service3-internal-method1
client x4 service1 x10 service3 x2 service2 x11unknown-service x2
Economies of scale
● Find the perfect balance● Align the risk taken by a service with the risk the business is willing to
bear● Explicit decisions
[15]
128 192 256 320 384 448 512 576 640 704 768 832 896 960 1024 1088 1152
Function 1
Function 2
Function 3
Function 4
Function 5
Function 6
...
Serverless
Automated ScalingRate of game events
Number of workers
40,000/s
30,000/s
20,000/S
10,000/s
30
20
10
0
[17, 18]
Automated ScalingRate of game events
Number of workers
40,000/s
30,000/s
20,000/S
10,000/s
30
20
10
0
Configurationval data = spark.read.format("libsvm") .load("sample_multiclass_classification_data.txt")
val splits = data.randomSplit(Array(0.6, 0.4), seed = 1234L)val train = splits(0)val test = splits(1)
val layers = Array[Int](4, 5, 4, 3)
val trainer = new MultilayerPerceptronClassifier() .setLayers(layers) .setBlockSize(128) .setSeed(1234L) .setMaxIter(100)
val model = trainer.fit(train)
val result = model.transform(test)val predictionAndLabels = result.select("prediction", "label")val evaluator = new MulticlassClassificationEvaluator() .setMetricName("accuracy")
evaluator.evaluate(predictionAndLabels)
Configurationval data = spark.read.format("libsvm") .load("sample_multiclass_classification_data.txt")
val splits = data.randomSplit(Array(0.6, 0.4), seed = 1234L)val train = splits(0)val test = splits(1)
val layers = Array[Int](4, 5, 4, 3)
val trainer = new MultilayerPerceptronClassifier() .setLayers(layers) .setBlockSize(128) .setSeed(1234L) .setMaxIter(100)
val model = trainer.fit(train)
val result = model.transform(test)val predictionAndLabels = result.select("prediction", "label")val evaluator = new MulticlassClassificationEvaluator() .setMetricName("accuracy")
evaluator.evaluate(predictionAndLabels)
Configurationval data = spark.read.format("libsvm") .load("sample_multiclass_classification_data.txt")
val splits = data.randomSplit(Array(0.6, 0.4), seed = 1234L)val train = splits(0)val test = splits(1)
val layers = Array[Int](4, 5, 4, 3)
val trainer = new MultilayerPerceptronClassifier() .setLayers(layers) .setBlockSize(128) .setSeed(1234L) .setMaxIter(100)
val model = trainer.fit(train)
val result = model.transform(test)val predictionAndLabels = result.select("prediction", "label")val evaluator = new MulticlassClassificationEvaluator() .setMetricName("accuracy")
evaluator.evaluate(predictionAndLabels)
Configurationval data = spark.read.format("libsvm") .load("sample_multiclass_classification_data.txt")
val splits = data.randomSplit(Array(0.6, 0.4), seed = 1234L)val train = splits(0)val test = splits(1)
val layers = Array[Int](4, 5, 4, 3)
val trainer = new MultilayerPerceptronClassifier() .setLayers(layers) .setBlockSize(128) .setSeed(1234L) .setMaxIter(100)
val model = trainer.fit(train)
val result = model.transform(test)val predictionAndLabels = result.select("prediction", "label")val evaluator = new MulticlassClassificationEvaluator() .setMetricName("accuracy")
evaluator.evaluate(predictionAndLabels)
Configurationval data = spark.read.format("libsvm") .load("sample_multiclass_classification_data.txt")
val splits = data.randomSplit(Array(0.6, 0.4), seed = 1234L)val train = splits(0)val test = splits(1)
val layers = Array[Int](4, 5, 4, 3)
val trainer = new MultilayerPerceptronClassifier() .setLayers(layers) .setBlockSize(128) .setSeed(1234L) .setMaxIter(100)
val model = trainer.fit(train)
val result = model.transform(test)val predictionAndLabels = result.select("prediction", "label")val evaluator = new MulticlassClassificationEvaluator() .setMetricName("accuracy")
evaluator.evaluate(predictionAndLabels)
Configurationval spark = SparkSession .builder() .config("spark.reducer.maxSizeInFlight", maxSizeInFlight) .config("spark.reducer.maxReqsInFlight", maxReqsInflight) .config("spark.shuffle.file.buffer", shuffleFileBuffer) .config("spark.memory.fraction", sparkMemoryFraction) .config("spark.shuffle.service.index.cache.entries", shuffleServiceIndexCacheEntries) .config("spark.memory.storageFraction", sparkMemoryStorageFraction) .config("spark.shuffle.memoryFraction", sparkShuffleMemoryFraction) .config("spark.storage.memoryFraction", sparkStorageMemoryFraction) .config("spark.storage.unrollFraction", sparkStorageUnrollFraction) .config("spark.broadcast.blockSize", sparkBroadcastBlockSize) .config("spark.executor.cores", sparkExecutorCores) .config("spark.default.parallelism", defaultParallelism) .config("spark.files.maxPartitionBytes", sparkFilesMaxPartitionBytes) .config("spark.files.openCostInBytes", sparkFilesOpenCostInByes) .config("spark.rpc.message.maxSize", sparkRpcMessageMaxSize) .config("spark.storage.memoryMapThreshold", sparkSorageMemoryMapThreshold) .config("spark.cores.max", sparkCoresMax) .config("spark.speculation", "true") .config("spark.speculation.interval", sparkSpeculationInterval) .config("spark.speculation.multiplier", sparkSpeculationMultiplier) .config("spark.speculation.quantile", sparkSpeculationQuantile) .config("spark.task.cpus", sparkTaskCpus)
● Labelled training examples (feature vector, label)● Classification, regression, clustering, ...● Find an algorithm that for given feature vector finds the correct label● Optimization of an objective function with respect to model parameters
Features Label
100mb,10,100kb,0.75,0.1,0.001,0.75,0.75,0.1,1g,2,8,10000,1000000,100,1g,1ms,1.5,0.01,1g
61.9799
1kb,2147483647,10mb,0.75,0.001,0.1,0.001,0.5,0.001,10mb,8,2,10000,1000000,100,1kb,1ms,1.1,0.75,512mb
12.2511
100mb,10,1mb,0.75,0.5,0.001,0.1,0.001,0.5,100mb,4,1,100000000,1000,1000,10mb,100ms,1.5,0.01,8g
20.7461
... ...
Supervised Learning
Supervised Learningdef deep_nn(x, hidden1_units, hidden2_units): with tf.name_scope('hidden1'): weights1 = tf.Variable(tf.truncated_normal([20, hidden1_units], stddev=1.0 / math.sqrt(float(20))), name='weights') biases1 = tf.Variable(tf.zeros([hidden1_units]), name='biases') hidden1 = tf.nn.relu(tf.matmul(x, weights1) + biases1)
with tf.name_scope('hidden2'): weights2 = tf.Variable(tf.truncated_normal([hidden1_units, hidden2_units], stddev=1.0 / math.sqrt(float(hidden1_units))), name='weights') biases2 = tf.Variable(tf.zeros([hidden2_units]), name='biases') hidden2 = tf.nn.relu(tf.matmul(hidden1, weights2) + biases2)
with tf.name_scope('linear'): weights3 = tf.Variable(tf.truncated_normal([hidden2_units, 1], stddev=1.0 / math.sqrt(float(hidden2_units))), name='weights') biases3 = tf.Variable(tf.zeros([1]), name='biases') logits = tf.matmul(hidden2, weights3) + biases3 return logits
Supervised Learningdef deep_nn(x, hidden1_units, hidden2_units): with tf.name_scope('hidden1'): weights1 = tf.Variable(tf.truncated_normal([20, hidden1_units], stddev=1.0 / math.sqrt(float(20))), name='weights') biases1 = tf.Variable(tf.zeros([hidden1_units]), name='biases') hidden1 = tf.nn.relu(tf.matmul(x, weights1) + biases1)
with tf.name_scope('hidden2'): weights2 = tf.Variable(tf.truncated_normal([hidden1_units, hidden2_units], stddev=1.0 / math.sqrt(float(hidden1_units))), name='weights') biases2 = tf.Variable(tf.zeros([hidden2_units]), name='biases') hidden2 = tf.nn.relu(tf.matmul(hidden1, weights2) + biases2)
with tf.name_scope('linear'): weights3 = tf.Variable(tf.truncated_normal([hidden2_units, 1], stddev=1.0 / math.sqrt(float(hidden2_units))), name='weights') biases3 = tf.Variable(tf.zeros([1]), name='biases') logits = tf.matmul(hidden2, weights3) + biases3 return logits
Supervised Learningdef deep_nn(x, hidden1_units, hidden2_units): with tf.name_scope('hidden1'): weights1 = tf.Variable(tf.truncated_normal([20, hidden1_units], stddev=1.0 / math.sqrt(float(20))), name='weights') biases1 = tf.Variable(tf.zeros([hidden1_units]), name='biases') hidden1 = tf.nn.relu(tf.matmul(x, weights1) + biases1)
with tf.name_scope('hidden2'): weights2 = tf.Variable(tf.truncated_normal([hidden1_units, hidden2_units], stddev=1.0 / math.sqrt(float(hidden1_units))), name='weights') biases2 = tf.Variable(tf.zeros([hidden2_units]), name='biases') hidden2 = tf.nn.relu(tf.matmul(hidden1, weights2) + biases2)
with tf.name_scope('linear'): weights3 = tf.Variable(tf.truncated_normal([hidden2_units, 1], stddev=1.0 / math.sqrt(float(hidden2_units))), name='weights') biases3 = tf.Variable(tf.zeros([1]), name='biases') logits = tf.matmul(hidden2, weights3) + biases3 return logits
Supervised Learningdef deep_nn(x, hidden1_units, hidden2_units): with tf.name_scope('hidden1'): weights1 = tf.Variable(tf.truncated_normal([20, hidden1_units], stddev=1.0 / math.sqrt(float(20))), name='weights') biases1 = tf.Variable(tf.zeros([hidden1_units]), name='biases') hidden1 = tf.nn.relu(tf.matmul(x, weights1) + biases1)
with tf.name_scope('hidden2'): weights2 = tf.Variable(tf.truncated_normal([hidden1_units, hidden2_units], stddev=1.0 / math.sqrt(float(hidden1_units))), name='weights') biases2 = tf.Variable(tf.zeros([hidden2_units]), name='biases') hidden2 = tf.nn.relu(tf.matmul(hidden1, weights2) + biases2)
with tf.name_scope('linear'): weights3 = tf.Variable(tf.truncated_normal([hidden2_units, 1], stddev=1.0 / math.sqrt(float(hidden2_units))), name='weights') biases3 = tf.Variable(tf.zeros([1]), name='biases') logits = tf.matmul(hidden2, weights3) + biases3 return logits
Supervised Learning example_batch, label_batch = read_example_batch(filename=FLAGS.data_file, batch_size=batch_size) x = tf.placeholder(tf.float32, [None, 20]) y = deepnn(x, hidden1_units, hidden2_units) y_ = tf.placeholder(tf.float32, [None, 1]) mse = tf.reduce_sum(tf.pow(y-y_, 2))/(2*batch_size) train_step = tf.train.GradientDescentOptimizer(learning_rate=0.25).minimize(mse)
sess = tf.Session() with sess.as_default(): sess.run(tf.global_variables_initializer()) coord = tf.train.Coordinator() threads = tf.train.start_queue_runners(coord=coord)
for i in range(1000): examples, labels = sess.run([example_batch, label_batch]) sess.run(train_step, feed_dict={x: examples, y_: labels})
Supervised Learning example_batch, label_batch = read_example_batch(filename=FLAGS.data_file, batch_size=batch_size) x = tf.placeholder(tf.float32, [None, 20]) y = deepnn(x, hidden1_units, hidden2_units) y_ = tf.placeholder(tf.float32, [None, 1]) mse = tf.reduce_sum(tf.pow(y-y_, 2))/(2*batch_size) train_step = tf.train.GradientDescentOptimizer(learning_rate=0.25).minimize(mse)
sess = tf.Session() with sess.as_default(): sess.run(tf.global_variables_initializer()) coord = tf.train.Coordinator() threads = tf.train.start_queue_runners(coord=coord)
for i in range(1000): examples, labels = sess.run([example_batch, label_batch]) sess.run(train_step, feed_dict={x: examples, y_: labels})
Supervised Learning example_batch, label_batch = read_example_batch(filename=FLAGS.data_file, batch_size=batch_size) x = tf.placeholder(tf.float32, [None, 20]) y = deepnn(x, hidden1_units, hidden2_units) y_ = tf.placeholder(tf.float32, [None, 1]) mse = tf.reduce_sum(tf.pow(y-y_, 2))/(2*batch_size) train_step = tf.train.GradientDescentOptimizer(learning_rate=0.25).minimize(mse)
sess = tf.Session() with sess.as_default(): sess.run(tf.global_variables_initializer()) coord = tf.train.Coordinator() threads = tf.train.start_queue_runners(coord=coord)
for i in range(1000): examples, labels = sess.run([example_batch, label_batch]) sess.run(train_step, feed_dict={x: examples, y_: labels})
Supervised Learning example_batch, label_batch = read_example_batch(filename=FLAGS.data_file, batch_size=batch_size) x = tf.placeholder(tf.float32, [None, 20]) y = deepnn(x, hidden1_units, hidden2_units) y_ = tf.placeholder(tf.float32, [None, 1]) mse = tf.reduce_sum(tf.pow(y-y_, 2))/(2*batch_size) train_step = tf.train.GradientDescentOptimizer(learning_rate=0.25).minimize(mse)
sess = tf.Session() with sess.as_default(): sess.run(tf.global_variables_initializer()) coord = tf.train.Coordinator() threads = tf.train.start_queue_runners(coord=coord)
for i in range(1000): examples, labels = sess.run([example_batch, label_batch]) sess.run(train_step, feed_dict={x: examples, y_: labels})
Data● What about training data?● Random runs● In-line optimization● Synthetic data● Generative models● Unsupervised learning
General case
Akka-httpKafka
Microservice CockroachDB
Dynamo
Redis Cache
Akka Cluster
Load Balancer
Lambda
General case● Complex● Long running● Large number of variables (many unknown)● Many possible actions● Temporal reward
Reinforcement Learning● Trial and error (positive reinforcement)● Set of states S● Set of actions A● Reward r
● P(s’ | a, s)● s0, a0, r1 -> s1, a1, r2 -> …● Q(s, a) = r + γ * maxa’(Q(s’, a’))
[19. 20]
Environment
Agent
Interpreter
Reward
State Action
Deep Reinforcement Learning - Gym
RewardNumber of errors, Cost
Concurrent user load 10,000 12,000 20,000 ...
Instances and sizes 1,1,2 ? ? ?
Action
Nothing
Inc inst size
Dec inst size
Add inst
Remove inst
State
Deep Reinforcement Learning - Gymdef _cost(self, dist_sys, load): cost = 0 can_handle_load = 0 for i in range(len(dist_sys)): if dist_sys[i] == INST_SIZE_0: cost += INST_SIZE_0_COST can_handle_load += INST_SIZE_0_LOAD elif dist_sys[i] == INST_SIZE_1: cost += INST_SIZE_1_COST can_handle_load += INST_SIZE_1_LOAD elif dist_sys[i] == INST_SIZE_2: cost += INST_SIZE_2_COST can_handle_load += INST_SIZE_2_LOAD errors = min((abs(min(can_handle_load - load, 0)) *
ERROR_COEFFICIENT + max(cost, 0)), MAX_COST)
Deep Reinforcement Learning - Gymdef _cost(self, dist_sys, load): cost = 0 can_handle_load = 0 for i in range(len(dist_sys)): if dist_sys[i] == INST_SIZE_0: cost += INST_SIZE_0_COST can_handle_load += INST_SIZE_0_LOAD elif dist_sys[i] == INST_SIZE_1: cost += INST_SIZE_1_COST can_handle_load += INST_SIZE_1_LOAD elif dist_sys[i] == INST_SIZE_2: cost += INST_SIZE_2_COST can_handle_load += INST_SIZE_2_LOAD errors = min((abs(min(can_handle_load - load, 0)) *
ERROR_COEFFICIENT + max(cost, 0)), MAX_COST)
Deep Reinforcement Learning - Gymdef _cost(self, dist_sys, load): cost = 0 can_handle_load = 0 for i in range(len(dist_sys)): if dist_sys[i] == INST_SIZE_0: cost += INST_SIZE_0_COST can_handle_load += INST_SIZE_0_LOAD elif dist_sys[i] == INST_SIZE_1: cost += INST_SIZE_1_COST can_handle_load += INST_SIZE_1_LOAD elif dist_sys[i] == INST_SIZE_2: cost += INST_SIZE_2_COST can_handle_load += INST_SIZE_2_LOAD errors = min((abs(min(can_handle_load - load, 0)) *
ERROR_COEFFICIENT + max(cost, 0)), MAX_COST)
Deep Reinforcement Learning - Gymdef _step(self, action): if (action == ADD_NODE): ... elif (action == INCREASE_INSTANCE_SIZE): ... elif (action == REMOVE_NODE): ... elif (action == DECREASE_INSTANCE_SIZE): ... else: self.dist_sys = self.dist_sys cost = self._cost(self.dist_sys, self.current_load[self.progress]) reward = -1 * ((2 * (cost / MAX_COST)) - 1) self.progress += 1 done = self.progress >= len(self.current_load) - 1 return self.current_load[self.progress], reward, done
Deep Reinforcement Learning - Gymdef _step(self, action): if (action == ADD_NODE): ... elif (action == INCREASE_INSTANCE_SIZE): ... elif (action == REMOVE_NODE): ... elif (action == DECREASE_INSTANCE_SIZE): ... else: self.dist_sys = self.dist_sys cost = self._cost(self.dist_sys, self.current_load[self.progress]) reward = -1 * ((2 * (cost / MAX_COST)) - 1) self.progress += 1 done = self.progress >= len(self.current_load) - 1 return self.current_load[self.progress], reward, done
Deep Reinforcement Learning - Gymdef _step(self, action): if (action == ADD_NODE): ... elif (action == INCREASE_INSTANCE_SIZE): ... elif (action == REMOVE_NODE): ... elif (action == DECREASE_INSTANCE_SIZE): ... else: self.dist_sys = self.dist_sys cost = self._cost(self.dist_sys, self.current_load[self.progress]) reward = -1 * ((2 * (cost / MAX_COST)) - 1) self.progress += 1 done = self.progress >= len(self.current_load) - 1 return self.current_load[self.progress], reward, done
Deep Reinforcement Learning - Agentclass agent(): def __init__(self, s_size,a_size): self.inputs = tf.placeholder(shape=[None, s_size],dtype=tf.float32) W = tf.Variable(tf.random_uniform([s_size,a_size],0,0.01)) self.Q_values = tf.matmul(self.inputs,W)
self.largest_Q_value_index = tf.argmax(self.Q_values,1) self.largest_Q_value = tf.reduce_max(self.Q_values)
self.next_Q_values = tf.placeholder(shape=[1,a_size],dtype=tf.float32) loss = tf.reduce_sum(tf.square(self.next_Q_values - self.Q_values)) trainer = tf.train.GradientDescentOptimizer(learning_rate=0.1) self.updateModel = trainer.minimize(loss)
[22, 23]
Deep Reinforcement Learning - Agentclass agent(): def __init__(self, s_size,a_size): self.inputs = tf.placeholder(shape=[None, s_size],dtype=tf.float32) W = tf.Variable(tf.random_uniform([s_size,a_size],0,0.01)) self.Q_values = tf.matmul(self.inputs,W)
self.largest_Q_value_index = tf.argmax(self.Q_values,1) self.largest_Q_value = tf.reduce_max(self.Q_values)
self.next_Q_values = tf.placeholder(shape=[1,a_size],dtype=tf.float32) loss = tf.reduce_sum(tf.square(self.next_Q_values - self.Q_values)) trainer = tf.train.GradientDescentOptimizer(learning_rate=0.1) self.updateModel = trainer.minimize(loss)
[22, 23]
Deep Reinforcement Learning - Agentclass agent(): def __init__(self, s_size,a_size): self.inputs = tf.placeholder(shape=[None, s_size],dtype=tf.float32) W = tf.Variable(tf.random_uniform([s_size,a_size],0,0.01)) self.Q_values = tf.matmul(self.inputs,W)
self.largest_Q_value_index = tf.argmax(self.Q_values,1) self.largest_Q_value = tf.reduce_max(self.Q_values)
self.next_Q_values = tf.placeholder(shape=[1,a_size],dtype=tf.float32) loss = tf.reduce_sum(tf.square(self.next_Q_values - self.Q_values)) trainer = tf.train.GradientDescentOptimizer(learning_rate=0.1) self.updateModel = trainer.minimize(loss)
[22, 23]
Deep Reinforcement Learning - Agentclass agent(): def __init__(self, s_size,a_size): self.inputs = tf.placeholder(shape=[None, s_size],dtype=tf.float32) W = tf.Variable(tf.random_uniform([s_size,a_size],0,0.01)) self.Q_values = tf.matmul(self.inputs,W)
self.largest_Q_value_index = tf.argmax(self.Q_values,1) self.largest_Q_value = tf.reduce_max(self.Q_values)
self.next_Q_values = tf.placeholder(shape=[1,a_size],dtype=tf.float32) loss = tf.reduce_sum(tf.square(self.next_Q_values - self.Q_values)) trainer = tf.train.GradientDescentOptimizer(learning_rate=0.1) self.updateModel = trainer.minimize(loss)
[22, 23]
Deep Reinforcement Learning - Agentwith tf.Session() as sess: sess.run(init) for i in range(num_episodes): state = env.reset() d = False j = 0 while j < env.episode_length() + 1: j+=1 fixed_in = fixed_length_history(episode_history[:,0]) chosen_action, all_Q_values = sess.run( [myAgent.largest_Q_value_index,myAgent.Q_values], feed_dict={myAgent.inputs:fixed_in}) if np.random.rand(1) < random_action_probability: chosen_action[0] = env.action_space.sample()
[22, 23]
Deep Reinforcement Learning - Agentwith tf.Session() as sess: sess.run(init) for i in range(num_episodes): state = env.reset() d = False j = 0 while j < env.episode_length() + 1: j+=1 fixed_in = fixed_length_history(episode_history[:,0]) chosen_action, all_Q_values = sess.run( [myAgent.largest_Q_value_index,myAgent.Q_values], feed_dict={myAgent.inputs:fixed_in}) if np.random.rand(1) < random_action_probability: chosen_action[0] = env.action_space.sample()
[22, 23]
Deep Reinforcement Learning - Agentwith tf.Session() as sess: sess.run(init) for i in range(num_episodes): state = env.reset() d = False j = 0 while j < env.episode_length() + 1: j+=1 fixed_in = fixed_length_history(episode_history[:,0]) chosen_action, all_Q_values = sess.run( [myAgent.largest_Q_value_index,myAgent.Q_values], feed_dict={myAgent.inputs:fixed_in}) if np.random.rand(1) < random_action_probability: chosen_action[0] = env.action_space.sample()
[22, 23]
Deep Reinforcement Learning - Agent next_state,r,d,_ = env.step(chosen_action[0])
episode_history.append([state, chosen_action, r, next_state]) next_largest_Q_value = sess.run( myAgent.largest_Q_value, feed_dict={myAgent.inputs:next_fixed_in}) all_Q_values[0,a[0]] = r + gamma * next_largest_Q_value _ = sess.run( myAgent.updateModel, feed_dict={myAgent.inputs:fixed_in,myAgent.next_Q_values:all_Q_values}) state = next_state
[22, 23]
Deep Reinforcement Learning - Agent next_state,r,d,_ = env.step(chosen_action[0])
episode_history.append([state, chosen_action, r, next_state]) next_largest_Q_value = sess.run( myAgent.largest_Q_value, feed_dict={myAgent.inputs:next_fixed_in}) all_Q_values[0,a[0]] = r + gamma * next_largest_Q_value _ = sess.run( myAgent.updateModel, feed_dict={myAgent.inputs:fixed_in,myAgent.next_Q_values:all_Q_values}) state = next_state
[22, 23]
Deep Reinforcement Learning - Agent next_state,r,d,_ = env.step(chosen_action[0])
episode_history.append([state, chosen_action, r, next_state]) next_largest_Q_value = sess.run( myAgent.largest_Q_value, feed_dict={myAgent.inputs:next_fixed_in}) all_Q_values[0,a[0]] = r + gamma * next_largest_Q_value _ = sess.run( myAgent.updateModel, feed_dict={myAgent.inputs:fixed_in,myAgent.next_Q_values:all_Q_values}) state = next_state
[22, 23]
Deep Reinforcement Learning - Agent next_state,r,d,_ = env.step(chosen_action[0])
episode_history.append([state, chosen_action, r, next_state]) next_largest_Q_value = sess.run( myAgent.largest_Q_value, feed_dict={myAgent.inputs:next_fixed_in}) all_Q_values[0,a[0]] = r + gamma * next_largest_Q_value _ = sess.run( myAgent.updateModel, feed_dict={myAgent.inputs:fixed_in,myAgent.next_Q_values:all_Q_values}) state = next_state
[22, 23]
Conclusion
● Optimize your code and architecture● Measure, gather data● Use the data!● Continuously improve and evolve
0845 617 1200
@zapletal_martin @cakesolutions
MANCHESTER LONDON NEW YORK
We are hiringhttp://www.cakesolutions.net/careers
We are hiringhttp://www.cakesolutions.net/careers
References[0] https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-sql-whole-stage-codegen.html[1] https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/6122906529858466/293651311471490/5382278320999420/latest.html[2] http://hydronitrogen.com/in-the-code-spark-sql-query-planning-and-execution.html[3] https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html[4] http://www.vldb.org/pvldb/vol4/p539-neumann.pdf[5] https://blog.acolyer.org/2016/05/23/efficiently-compiling-efficient-query-plans-for-modern-hardware/[6] https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-rdd-lineage.html[7] https://databricks.com/blog/2017/02/16/processing-trillion-rows-per-second-single-machine-can-nested-loop-joins-fast.html[8] https://www.slideshare.net/SparkSummit/costbased-optimizer-framework-for-spark-sql-spark-summit-east-talk-by-ron-hu-and-zhenhua-wang[9] https://medium.com/@copyconstruct/monitoring-in-the-time-of-cloud-native-c87c7a5bfa3e[10] https://www.lightbend.com/blog/lightbend-monitoring-now-integrates-with-datadog-for-monitoring-akka-based-reactive-applications[11] https://www.lightbend.com/blog/how-to-get-monitoring-right-for-streaming-and-fast-data-systems-built-with-spark-mesos-akka-cassandra-and-kafka?utm_content=buffera2917&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer[12] http://zipkin.io/[13] https://www.theverge.com/2016/7/21/12246258/google-deepmind-ai-data-center-cooling[14] http://www.npr.org/sections/thetwo-way/2017/03/03/518322734/amazon-and-the-150-million-typo.[15] http://money.cnn.com/2016/09/07/technology/delta-computer-outage-cost/[16] https://www.slideshare.net/SparkSummit/ernest-efficient-performance-prediction-for-advanced-analytics-on-apache-spark-spark-summit-east-talk-by-shivaram-venkataraman[17] https://www.slideshare.net/SparkSummit/auto-scaling-systems-with-elastic-spark-streaming-spark-summit-east-talk-by-phuduc-nguyen[18] https://cloud.google.com/blog/big-data/2016/03/comparing-cloud-dataflow-autoscaling-to-spark-and-hadoop[19] https://www.youtube.com/watch?v=URWXG5jRB-A[20] https://en.wikipedia.org/wiki/Reinforcement_learning[21] https://stats.stackexchange.com/questions/234891/difference-between-convolution-neural-network-and-deep-learning[22] https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-0-q-learning-with-tables-and-neural-networks-d195264329d0[23] https://github.com/awjuliani/DeepRL-Agents/blob/master/Q-Network.ipynb[24] https://www.youtube.com/watch?v=22g14GtVhXk[25] https://www.youtube.com/watch?v=C-BY3JhXTiE