realtime insight in your application usage with nodejs, elasticsearch and kibana onno de haan
TRANSCRIPT
![Page 1: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/1.jpg)
Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana
Onno de Haan
![Page 2: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/2.jpg)
Hier uw naam
Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana
Onno de Haan
![Page 3: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/3.jpg)
Agenda
• About VanMeijel
• Why we need insight in Metacom (our ERP-system)• Requirements for monitoring tool• Developing the tool• Final solution
• Demo !
Please feel free to ask questions at any time
![Page 4: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/4.jpg)
About VanMeijel
• Founded in 1987• Based in The Netherlands, Emmeloord• Satellite office in Belgium, Sint-Niklaas• Around 110 employees• Software is targeted at construction companies• 300+ customers, 8000 - 9000 users
![Page 5: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/5.jpg)
Metacom – Our ERP system
• Contains modules for
- Estimation - Reporting - Planning - Invoicing - Cost Control - Procurement - Relation Management - Resource Planning - and many more
![Page 6: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/6.jpg)
Metacom – Our ERP system
• Build in OpenEdge, currently version 11.5
- Millions of code lines - Around 20 – 25 years of age - Lots of different techniques (DB-access on client, client-server, procedural, OO) - Deployed via Round Table, recently moved to Subversion / Env / Jenkins
![Page 7: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/7.jpg)
• About VanMeijel
• Why we need insight in Metacom• Requirements for monitoring tool• Developing the tool• Final solution
• Demo
Please feel free to ask questions at any time
![Page 8: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/8.jpg)
• Application usage is changing
- People are working mobile - People want / expect Apps - Customers don’t want big server parks: web interface
• Changing world• Integrations • Maintenance• Different license models• Move to webinterface
Why we need insight in Metacom
![Page 9: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/9.jpg)
• Customers are integrating 3th party applications
• One webservice often replaces many “real” users
• Integrations weaken our license model
• Changing world• Integrations • Maintenance• Different license models• Move to webinterface
Why we need insight in Metacom
![Page 10: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/10.jpg)
• What methods are called
• How often are they called
• How many business objects are being created / edited per method
• How many systems are connecting to Metacom
• Changing world• Integrations • Maintenance• Different license models• Move to webinterface
Why we need insight in Metacom
![Page 11: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/11.jpg)
• System is complex
• Around 3000 – 4000 windows/dialogs
• Contains lots and lots of settings
• What is the performance of our webservice methods
• Changing world• Integrations • Maintenance• Different license models• Move to webinterface
Why we need insight in Metacom
![Page 12: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/12.jpg)
• In the future we need to move to “pay for usage”
• How many business documents are created - For example how many invoices
• Are business documents updated frequently or not?
• How many reports are being generated
• Changing world• Maintenance• Integrations• Different license models• Move to webinterface
Why we need insight in Metacom
![Page 13: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/13.jpg)
• How many users are active?
• On what time of the day / week?
• How about evenings, weekends?
• Changing world• Maintenance• Integrations• Different license models• Move to webinterface
Why we need insight in Metacom
![Page 14: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/14.jpg)
• What functions / modules are used
• What is not used anymore
• Top 100 of most used functions
• Changing world• Maintenance• Integrations• Different license models• Move to webinterface
Why we need insight in Metacom
![Page 15: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/15.jpg)
• About VanMeijel
• Why we need insight in Metacom• Requirements for monitoring tool• Developing the tool• Final solution
• Demo
Please feel free to ask questions at any time
![Page 16: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/16.jpg)
Requirements monitoring tool
• Web based
• Real time
• Easy to analyze
• Flexible, schemaless database
• Light-weight; JSON
![Page 17: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/17.jpg)
• About VanMeijel
• Why we need insight in Metacom• Requirements for monitoring tool• Developing the tool• Final solution
• Demo
Please feel free to ask questions at any time
![Page 18: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/18.jpg)
• Events are send to our AppServer API (written in OpenEdge)
- Client - Triggers - Business Logic - Webservices
• AppServer API sends events to webserver
• Webserver stores events and hosts a viewer
• Architecture• First try, january 2014• Different license models
Developing the tool
![Page 19: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/19.jpg)
• NodeJS as webserver
• MongoDB as database
• Visualization was a problem
• Architecture• First try, january 2014• Second try, march 2014
Developing the tool
![Page 20: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/20.jpg)
• We replaced MongoDB with ElasticSearch
• Visualization is super easy with Kibana
• Migration from MongoDB to ElasticSearch was difficult
• Architecture• First try, january 2014• Second try, march 2014
Developing the tool
![Page 21: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/21.jpg)
• About VanMeijel
• Why we need insight in Metacom• Requirements for monitoring tool• Developing the tool• Final solution
• Demo
Please feel free to ask questions at any time
![Page 22: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/22.jpg)
• Clients push events to AppServer
• AppServer start HTTP-session to REST-service in cloud
• Fires JSON event, simplified example: { “customerid”:”vma”, “datetime”:”2015-11-05T14:36:54”, “appid”: “mtc”, “appversion”: “9.2.2.1” “type”:”ui”, “function”: “wcustomers” }
• OpenEdge events• NodeJS Webserver• ElasticSearch database• Kibana for visualisation
Final solution
![Page 23: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/23.jpg)
• OpenEdge events• NodeJS Webserver• ElasticSearch database• Kibana for visualisation
Final solution
var restify = require('restify');var elasticsearch = require('elasticsearch');
var server = restify.createServer( { name : "stats" }); server.listen(‘1234’,’127.0.0.1’, function()
{ console.log('%s listening at %s ', server.name , server.url);});
server.use(restify.queryParser());server.use(restify.bodyParser());server.use(restify.CORS());var client = new elasticsearch.Client({ host: 'localhost:9200' });
server.post({path : '/stats', version: '0.0.1'}, postNewMtcStat);
function postNewMtcStat(req , res , next){ var body = req.body; res.setHeader('Access-Control-Allow-Origin','*'); client.create({index: 'usage', type: 'usage', body: req.body }, function(err , success){ if(success){ res.send(201 , 'saved'); return next(); }else{ return next(err); } });}
![Page 24: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/24.jpg)
• OpenSource, Apache License
• Based on Lucene
• Distributed database, very scalable
• Near real-time search
• Document oriented, schema changes not a problem!
• OpenEdge events• NodeJS Webserver• ElasticSearch database• Kibana for visualisation
Final solution
![Page 25: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/25.jpg)
• OpenEdge events• NodeJS Webserver• ElasticSearch database• Kibana for visualisation
Final solution
• Cool
• Flexible
• Fast (but not on this laptop )
• Very easy to use
![Page 26: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/26.jpg)
• OpenEdge events• NodeJS Webserver• ElasticSearch database• Kibana for visualisation
Final solution
![Page 27: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/27.jpg)
DEMO
![Page 28: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/28.jpg)
QUESTIONS
![Page 29: Realtime insight in your application usage with NodeJs, ElasticSearch and Kibana Onno de Haan](https://reader036.vdocuments.us/reader036/viewer/2022062409/5697bf941a28abf838c907c1/html5/thumbnails/29.jpg)
Thank you for your