telvent big data approach and case studies
TRANSCRIPT
Schneider Electric –Telvent Global Services – 2014
TELVENT
Telvent Global Services
TSIUC’14
Retos en Big Data en la Universidad y la Investigación
Telvent Big Data Approach and Case Studies
TELVENT
Schneider Electric –- Telvent Global Services – 2014
Introduction
Schneider Electric –- Telvent Global Services – 2014
Bringing together IT with OT to increase business performance through:
Consulting, Integration and
Outsourcing Services
Our Mission: Technology Integration & Real Time Architectures
+250 end customers
worldwide in 2013
More than 30,000 m2
of Data Centers
managed
24/7 IT Support for
more than +30,000
users
+1,500 Network
devices
+5,000 Servers
+3.5 Managed Pbyte
+250 VHost
About Telvent ….
TELVENT
Schneider Electric –- Telvent Global Services – 2014
Big Data
Schneider Electric –- Telvent Global Services – 2014
Requirements….
Today, data business needs to satisfy 4 characteristics:
Solution: BIG DATA
Schneider Electric –- Telvent Global Services – 2014
Consulting
Services
Managed
Services
Big Data Landscape
Data Collection
Communications
& Networks
Data Centers &
Critical Infrastructures
Applications &
Business Processes
IT Systems
Data Processing Data Analytics
Managed
Services
Consulting
Services
Integration
Services
TELVENT
Schneider Electric –- Telvent Global Services – 2014
Telvent Case Studies
Schneider Electric –- Telvent Global Services – 2014
Case Study I
Bottleneck
Background
Carriers must save all the information related to the phone calls and navigation (CDRs) due
to a new regulation law.
IT Architecture
Architecture was based on a relational database that manages more than 50 TB of data.
Capacity planning expects to triple the data in less than two years.
Options Analyzed
Schneider Electric –- Telvent Global Services – 2014
Case Study I
Solution
• Exadata was rejected due to its cost and performance.
• Hadoop cluster design and implementation in order to save and manage all the traffic
coming from the calls and navigation.
• Relational database is only used for reporting and dashboard. Its size is limited to 5TB
Schneider Electric –- Telvent Global Services – 2014
Case Study II
Background
• Carriers must save all the information related to an user navigation (Public IP and private
IP) due to a new regulation law.
Options Analyzed
IT Architecture
• There is a Teradata database but it has not
enough capacity, so investment is needed.
• Architecture was not implemented yet, but it has
been estimated that it would be needed more than
600TB in normal storage.
Schneider Electric –- Telvent Global Services – 2014
Case Study II
Hadoop as a Service
Front End
Routers Acceso Plataforma IaaS Telvent
Servidor Integrador
(TGC)
Back End
Cluster namenodes
Crecimiento Neto garantizado
Cluster Datanodes
Servidor Monitorización
(TGC)
Solution
• Hadoop cluster that assure a response time of 60 seconds per user.
• Web Application development that let the user make their requests.
Schneider Electric –- Telvent Global Services – 2014
Case Study III
Background
• Marketing department needs a platform that help it to know,
what products are the most suitable for each person User
Segmentation, looking for a pattern on behavior.
• High security protocols due to the level of confidentiality
needed.
IT Architecture
• Current database has not enough process power.
• BI system are not so effective in predictions but analyzing past data.
Schneider Electric –- Telvent Global Services – 2014
Case Study III
Solution
• Working with a partner for the
analytics scope.
• SPARK service use by data
analyst, working on cluster memory. .
• Firstly a dedicated platform and
once they test it, we transform the
service to “Telvent Hadoop as a
Service.”
• Volume: 5 TB
• New potential users appears once
Big data si working.
Schneider Electric –- Telvent Global Services – 2014
Case Study IVBackground
• In “Free-Flow” motorways there is a platform that takes a
picture to each user that crosses the toll. With this image and
using OCR systems, it is recognized the 'number plate' and then,
it is searched its bank account associated So, no vehicle has
to stop at toll.
Scope
• Number of images: 1.300.000.000
• Image size: From 90K to150K
• Retention: From 1 to 5 years
IT Architecture
• SAMFS solution
• Based on a hierarchical storage that only
keeps online the images for the last 6 months.
The previous images are saved in tapes
Restricted
Schneider Electric –- Telvent Global Services – 2014
Case Study IV
IT Architecture
• New platform based on Hadoop,
composed by 8 servers (namenodes and
datanodes) and 384 GB RAM.
• New platform allows the company to
keep all the images online. Current
hierarchical storage disappears
Advantages
• Efficiency in fraud management default
improved.
• Performance in image processing
improved.
• Every sattelite application can be
centralized
Restricted
Schneider Electric –- Telvent Global Services – 2014
Case Study V
Background
• New Smart Meters are managed from a centralized platform, that should be able to
receive and send information to each one. Also, all the information (measures) has to be
recorded, at least, for two years.
Scope:
•.More than 11 M smart meters 1
measure per hour 198.000 M measures
• Platform should be used also like a
dashboard
IT Architecture:
• Current IT Infrastructure is mainly Oracle
Restricted
Schneider Electric –- Telvent Global Services – 2014
Case Study V
IT Architecture
• Exadata infrastructure provisioning. This solutions was
considered the best due to its performance for:
• Data loading (it receives lots of measures)
• Requests (Ex: repeat the measure, change de
power, electricity switch off …).
Proof of Concept
• Based on smart meter remote management, switch off
about 100.000 homes in 40 minutes.
TELVENT
Schneider Electric –- Telvent Global Services – 2014
Education
Schneider Electric –- Telvent Global Services – 2014
Case Study
• Recomendador de documentación para un alumno basado en datos
de otros alumnos con el mismo perfil (Filtro colaborativo)
• Previsión y planificación de recursos internos de la universidad en
función de variables oferta/demanda (Forecast)
• Segmentación de alumnos en función de las campañas de marketing
lanzadas (Clustering)
• Alumno Alumno; cruce de búsqueda para usuarios similares que
compartan experiencias con los nuevos alumnos
• Dimensionamiento servicio y calidad del servicio. Estimación de
llamadas Call Center
• Detección de abandono /suspenso de una alumno de un curso
(Decisor)
Schneider Electric –- Telvent Global Services – 2014
Conclusions
The growth of digital information and the
need to manage and analyze the data will not
change its exponential path (both structured
and unstructured).
Usual tools, applications, or databases does
not work with this amount of data.
Business requeriments:
• Cost reductions
• Time reductions
• New product development and
optimized offerings
• Smarter business decision making
Big Data is a new way to
face IT Infrastructure
problems
IT Transformation
TELVENT
Make the most of your energy™
Marta de Mesa Rincón – Solution Consulting Director
Jesús Gironda Díaz – Open System Manager