master’s(thesis: descriptive(study(of(the(elkstack ......october)6th, 2015 uludag –...
TRANSCRIPT
Software Engineering for Business Information Systems (sebis) Department of InformaticsTechnische Universität München, Germany
wwwmatthes.in.tum.de
Master’s Thesis: Descriptive study of the ELK stack applicability for data analytics use cases in the mobility industryOctober 6th, 2015Oemer Uludag, Prof. Dr. Matthes, Thomas Reschenhofer, M.Sc.
Agenda
© sebis 2
6 Project plan§ Structured project plan for realizing the approach
Approach§ Approach for answering the research questions5
Research questions§ Overview of relevant research questions of the master’s thesis4
3 Technologies§ Big Data and search-based discovery tools§ The Elasticsearch, Logstash and Kibana (ELK) stack
October 6th, 2015 Uludag – MT Kickoff Presentation
TUM Living Lab Connected Mobility (TUM LLCM) § Research project TUM LLCM overview2
Mobility§ Society in transition and urbanization§ Effect of the global urbanization on Munich
1
Building up a framework for the master’s thesis
© sebisOctober 6th, 2015 Uludag – MT Kickoff Presentation 3
u 1. Mobility
u 4. Research questions
u 6. Project plan
u 5. Approach
?
u 2. TUM LLCM
u 3. Technologies
1. Mobility
© sebisOctober 6th, 2015 Uludag – MT Kickoff Presentation 4
u 1. Mobility
Society in transition
© sebisOctober 6th, 2015 Uludag – MT Kickoff Presentation 5
Urbanizationu Impacts on mobility
Profound changes in the coming decades for the world’s population
© sebis 6October 6th, 2015 Uludag – MT Kickoff Presentation
0
2
4
6
8
10
12
1950 1970 1990 2014 2030 2050Po
pulatio
n (billions)
The world's population development
Total world population Urban population Rural population
In 2007, for the first time in history, global urban population exceeded the global rural population.
54%of the world’s population was urban in 2014.
66%of the world’s population is expected to be urban by 2050.
1950
2014
2050
Percentage of the population residing in urban areas
Percentage urban75 or over
50 to 7525 to 50Less than 25
No data
?Implications
Sources: based on [1]
Rising urbanization impedes mobility
The increasing population in urban areas involves serious challenges for mobility in urban areas:
7October 6th, 2015 Uludag – MT Kickoff Presentation © sebis
1. Increasing vehicle miles traveleduGiven that the population increasingly resides in urban areas, it is clear that vehicle miles traveled will increasingly accrue in urban areas.
0.00.51.01.52.02.53.03.5
Trillions of m
iles
Year
Development of vehicle miles traveled in the United States
Rural Urban Total
Sources: based on [2, 3]
Rising urbanization impedes mobility
The increasing population in urban areas involves serious challenges for mobility in urban areas:
8October 6th, 2015 Uludag – MT Kickoff Presentation © sebis
2. Increasing congestion in urban areasuRising population in urban areas leads toincreasing congestion in urban areas.
Increasing delay per U.S. commuter in traffic (hours):u 1982: 14.4u 2000: 34.8u 2006: 39.1u 2020: ≈ 41.0
Increasing fuel waste (billionsof gallons):u 1982: 0.36u 2000: 1.63u 2006: 2.18u 2020: ≈ 3.2
$20.6 bn
$79.2 bn$100 bn
≈ $175 bn
leer
Annual cost of congestion in the United States$
How does thissituation look in Munich?
?
1982 2000 2006 2020Sources: based on [4]
Munich is also affected by the global urbanization trend
Some facts about Munich’s transitionA population growth of 15.4% is expected between 2013 and 2030 Based on historical and predicted data of Munich’s population and
employment development, a great increase of its traffic is expected
9October 6th, 2015 Uludag – MT Kickoff Presentation © sebis
4,275,000 4,579,000
2,059,0002,427,000
1,000,0002,000,0003,000,0004,000,0005,000,0006,000,0007,000,0008,000,000
Analysis 2000 Prognosis 2015
Trips per day
Development of trips per day
Trips of Munich's surrounding region's inhabitants
Trips of Munich's inhabitants
6,334,0007,006,000
+ 18%
+ 7%
Sources: based on [5, 6]
Munich is also affected by the global urbanization trend
10October 6th, 2015 Uludag – MT Kickoff Presentation © sebis
32.6 %
43.7 %
16 %
7.7 %
Overview of preferred means of transport 2015
public transportationmotorized individual transportationwalkcycle
32 %
42.3 %
17.6 %
8.1 %
Overview of preferred means of transport 2000
public transportationmotorized individual transportationwalkcycle
Intelligent use of data as an opportunity to improve the mobility situation in Munich� TUM Living Lab ConnectedMobility (TUM LLCM)
Sources: based on [6]
2. TUM LLCM
© sebisOctober 6th, 2015 Uludag – MT Kickoff Presentation 11
u 2. TUM LLCM
TUM Living Lab Connected Mobility (TUM LLCM)
October 6th, 2015 Uludag – MT Kickoff Presentation © sebis 12
The TUM Living Lab Connected Mobility (TUM LLCM)aims to make a significant contribution to an open service platform digital
mobility in Bavaria by researching, developing, and evaluating innovative platform services and applications for digital mobility platforms.
targets to promote the design and prototypical implementation of an open usable digital mobility platform.
is carried out in close collaboration with leading industrial suppliers of digital mobility services.
is used as an innovation platform for simplified and accelerated exchangewith the development of digital mobility services between university, industryand end-users.
The TUM LLCM aims to merge different mobility-relatedtechnologies into a platform that collects and processes data. Once merged, these data are then provided as a basis for an efficient, safe and comfortable mobility.
Sources: based on [7]
3. Technologies
© sebisOctober 6th, 2015 Uludag – MT Kickoff Presentation 13
u 3. Technologies
Big Data and search-based data discovery tools
October 6th, 2015 Uludag – MT Kickoff Presentation © sebis 14
Technological transformation in the area of mobilityUbiquitous computing is also in the area of mobility, promoting to new
technologies and leading to a rapid and disruptive technological transformation in this area.
Various kinds of vehicular sensors generated by the Internet of Things and a new generation of strongly networked and integrated systems contribute continuously to the expansion of huge mounds of data.
The ability to process and analyze this data and to extract insight and knowledge that enable intelligent services is a critical capability.
Sources: based on [8, 9, 10]
Big Data and search-based data discovery tools
October 6th, 2015 Uludag – MT Kickoff Presentation © sebis
Technological transformation in the area of mobilityExample of these kinds of applications in the mobility industry comprise:
15
The 5 Vs of volume, velocity, variety, veracity, and value are often used to describe the requirements of Big Data applications and the characteristics of Big Data.
Connected vehicle Autonomous driving
Smart manufacturing Industrial internet Mobility services
Sources: based on [10]
Big Data and search-based data discovery tools
October 6th, 2015 Uludag – MT Kickoff Presentation © sebis 16
5 Vs ofBig Data
Volume
▶ Terabytes▶ Records▶ Transactions▶ Tables, Files
Velocity
▶ Batch▶ Real/near-time▶ Processes▶ Streams
Variety▶ Structured▶ Unstructured▶ Semi-
structured▶ All the above
Veracity
▶ Trustworthiness▶ Authenticity▶ Availability▶ Accountability
Value
▶ Statistical▶ Events▶ Correlations▶ Hypothetical
Sources: based on [11, 12]
Big Data and search-based data discovery tools
October 6th, 2015 Uludag – MT Kickoff Presentation © sebis 17
Search-based data discovery toolsraise huge expectations and promise high benefits for organizations among
Big Data and analytics technologies.facilitate users to develop and refine views and analyses of multi-structured
data using search term and to find relationships across structured, unstructured, and semi-structured data.
feature a performance layer to lessen the need for aggregates and pre-calculations.
are vended by i.e. Attivio, IBM, Oracle, Splunk, and ThoughtSpot
The combination of the three open source projects Elasticsearch, Logstash and Kibana (ELK), also known as the ELK stack is an outstanding alternative to commercial search-based data discovery tools.
Sources: based on [13]
The Elasticsearch, Logstash and Kibana (ELK) stack
October 6th, 2015 Uludag – MT Kickoff Presentation © sebis 18
The ELK stack as an outstanding search-based data discovery toolThe ELK stack is and end-to-end stack that glean actionable insights in real
time from almost any type of structured and unstructured data source. Elasticsearch: performs deep search and data analytics.Logstash: is responsible for centralized logging, log enrichment and
parsing log files.Kibana: is used to visualize data from Elasticsearch.
Due to the fact that the ELK stack is used by many organizations for a variety of business critical functions, an evaluation of its applicability in the mobility industry seems auspicious and indispensable.
Sources: based on [14, 15]
4. Research question
© sebisOctober 6th, 2015 Uludag – MT Kickoff Presentation 19
u 4. Research questions
?
Overview of relevant research questions
© sebisOctober 6th, 2015 Uludag – MT Kickoff Presentation 20
? Research question 1:What are capabilities and key features of the ELK
stack for data analytics?
? Research question 2:What are data analytics use cases in the mobility
industry?
? Research question 3:For which type of data analytics uses cases is the ELK
stack applicable?
5. Approach
© sebisOctober 6th, 2015 Uludag – MT Kickoff Presentation 21
u 5. Approach
Approach for answering the research questions
© sebis 22October 6th, 2015 Uludag – MT Kickoff Presentation
3. Ascertainment of data analytics use cases in themobility industry� Deliverable: Collection of data analytics use cases
4. Implementation of data analytics uses cases in the ELK stack� Deliverable: Running ELK stack with real data
5. Analysis of implemented data analytics use cases� Deliverable: Derivation of ELK stack applicability for data
analytics use cases for the mobility industry
1. Implementation of the ELK stack� Deliverable: Configured ELK stack on the cluster
2. Analysis of the ELK stack� Deliverable: Derivation of capabilities and features of the ELK
stack
6. Project plan
© sebisOctober 6th, 2015 Uludag – MT Kickoff Presentation 23
u 6. Project plan
Structured project plan for realizing the approach
© sebisOctober 6th, 2015 Uludag – MT Kickoff Presentation 24
2015 2016October November December January February March
Master's Thesis kickoff presentation10/6/2015
ELK stack configuration10/27/2015
ELK stack feature derivation12/5/2015
Use case collection12/30/2015
Runnig ELK stack1/27/2016
ELK stack applicability assessment2/18/2016 Master's
Thesis submission3/31/2016
Master's Thesis final presentation
3/21/2016
10/1/2015 -10/27/2015 Installation of the ELK stack at node
10/27/2015 -12/5/2015 Analysis of the ELK stack
12/5/2015 -12/30/2015 Ascertainment of use cases
12/30/2015 -1/27/2016 Implementation of use cases in the ELK stack
1/28/2016 -2/18/2016 Analysis of implemented use cases
10/1/2015 -3/15/2016
Writing the Master's Thesis
Discussion
25
Thank you very much for your attention!Do you have any questions?
October 6th, 2015 Uludag – MT Kickoff Presentation © sebis
References
26
[1] United Nations, Department of Economic and Social Affairs, Population Division. 2015. World Urbanization Prospects: The 2014 Revision. New York, USA.[2] William J. Mitchell, Christopher E. Borroni-Bird, and Lawrence D. Burns. 2010. Reinventing the Automobile: Personal urban mobility for the 21st century. MIT Press, Cambridge, MA, USA.[3] US Department of Transportation, Federal Highway Administration, Office of Highway Policy Information. 2014. Annual Vehicle –Miles of Travel, 1980 – 2007. Washington, DC, USA. http://www.fhwa.dot.gov/policyinformation/statistics/vm02_summary.cfm. Accessed: October 6, 2015.[4] David Schrank, Tim Lomax, and Bill Eisele. 2011. The 2011 Urban Mobility Report. Texas Transportation Institute, Texas A&M University, TX, USA.[5] Landeshauptstadt München. 2015. Demografiebericht München – Teil 2, Kleinräumige Bevölkerungsprognose 2013 bis 2030 für die Stadtbezirke. Munich, Germany.[6] Landeshauptstadt München. 2006. Verkehrsentwicklungsplan. Munich, Germany.[7] TUM Living Lab Connected Mobility. 2014. Technische Universität München, Munich, Germany. http://tum-llcm.de. Accessed: October 6, 2015.[8] Michael Friedewald and Oliver Raabe. 2010. Ubiquitous computing: An overview of technology impacts. Telematics Inform, 28, 2 (2011), 55 – 65.[9] Günther Sagl, Martin Loidl, and Bernd Resch 2012. Visuelle Analyse von Mobilfunkdaten zur Charakterisierung Urbaner Mobilität. In Geoinformationssysteme, Wichmann Verlag, Berlin, Germany, 72 – 79.[10] Andre Luckow, Ken Kennedy, Fabian Manhardt, Emil Djerekarov, Bennie Vorster and Amy Apon. 2015 In Proceedings of IEEE Conference on Big Data. IEEE. Santa Clara, CA, USA.[11] Philip Russom. 2011. Big Data Analytics. TDWI Best Practices Report, Fourth Quarter. The Data Warehouse Institute, Renton, WA, USA.[12] Yuri Demchenko, Paola Grosso, Cees de Laat, and Peter Membrey. 2013. Addressing Big Data Issues in Scientific Data Infrastructure. In Proceedings of the 2013 International Conference on Collaboration Technologies and Systems. IEEE. San Diego, CA, USA, 48–55.[13] Bart De Muynck. 2015. How to Derive Value From Big Data in Transportation. Gartner Research.[14] An Introduction to the ELK stack. Elastic. http://www.elastic.co/webinars/introduction-elk-stack. Accessed: October 6, 2015.[15] The ELK Stack in a DevOps Environment. Elastic. http://www.elastic.co/webinars/elk-stack-devops-environment. Accessed: October 6, 2015.
© sebisOctober 6th, 2015 Uludag – MT Kickoff Presentation