big data platform implementation€¦ · such streamlining of data provides a unified view across...
TRANSCRIPT
Big Data
Platform Implementation
Innovation | Intelligence | Cloud
Consolidate | Automate | Predict
Big Data Platform Implementation - Objective
• InnoTx helps organizations create an Analytics Ready Data environment.
• This helps in creating a single, unified data management framework that brings appropriate data together for reporting & analysis. Such streamlining of data provides a unified view across various segments.
Manage huge volumes of data, with multiple deployment options.
Automation of data pull, aggregation for reporting and increase in productivity
Gain deep insights to take timely action
Big Data and Analytics Platform
Big Data Platform - Benefits
All data has potential value
No defined schema - stored in native format
Schema is imposed and transformations are done at query time (schema-on-read)
ELT(Extract, Load, Transform) rather than ETL processing
Designed for low cost storage
Highly agile – configure and reconfigure as required
Provide Faster Insights
Apps and users interpret the data as they see fit
ELT WORKFLOW
Big Data Platform – Data Lake Ecosystem
DESIGN
IMPLEMENT
ANALYTICS
SUPPORT
Big Data Platform – Technical Architecture
Full Stack Support
Big Data Platform – Technical Architecture
Analysis of data sources
i. Identification of data sourcesii. Analyzing the architecture of databases,
studying table structures, etc.iii. Data sizing for planning hardware
requirements
Technology setup i. Installation of Big Data platform on respective project server
ii. Installation of supporting applications such as MS office, etc.
i. Identifying data to be migrated ii. Creation of input databases and
output databasesiii. Migrating data to project server
Testing data lake architecture, UAT, validation reports, etc.
Data movement
Testing and validation
PHASE 1: Data Set-up and Movement
Activities include:• Hardware Setup and Testing• Connectivity • Big Data Platform Download and
Installation• Coherency Testing• Testing (including sample data)• Data Transfer- Final• Acceptance Testing activity
*Each process involves InnoTx team and Client support
Big Data Platform – Technical Architecture
PHASE 2: Transfer and Loading Data for End Use
Activities include:• Understanding requirements for
Metadata and final Use Cases• Document Metadata
implementation• Design/Implementation• Testing of final structured data• Regulatory Reports generation &
validation• Acceptance Testing
*Each process involves InnoTx team and Client support
i. Selection of relevant input tablesii. Marking relevance of variables in each
table
Data Model creation
i. Understand the specific report requirementsii. Reuse the existing queries to create Hadoop queriesiii. Run queries and output tables in consumable form
i. Establish connection between BI tool and Data Martii. Generation of intermediate data cubes for visualizationiii. Demonstrate generation of reports via the BI tool
Generation of reports
BI tool integration
i. Specification of output tableii. Mapping of input tables/variables to output tablesiii. Scripts for output table creationiv. Data mart creation
Creation of assets data dictionary
Delivery Process
Business
requirement
gathering and
sign offs
Specific data
sets
identification
as per mutual
discussion
Data sizing
requirements
Technology
Setup
Validation to
establish data
integrity
Analysis of
data to track
industry
trends from
the data
Reporting –
Insights,
dashboards/r
eports and
output
analysis
User review
12 3 4
56
7 8
Delivery - Success Metrics
Deliverables/Milestones Success Criteria
Infrastructure Setup Specification Documents
Infrastructure verified as per information provided in the proposal and subsequent emails
Assets Data Dictionary Excel with Tables, Variable names and their importance etc
Data Movement Pipeline (Data Pipe) Scripts that enable automated movement of data; single click run and query tests for newly moved data
Archival Data Store Basic stats comparison, eg.. row counts, sum of columns, unique entries etc
Reports Documentation Business Requirements Document: Report output values, input variables, frequency, updates, and usage (department wise)
Assets Data Mart Reports enabled with quick query respose wrt raw data
Assets Reports Tables and Graphs as per BRD
Query Performance Assessment Excel Tables and Graphs for execution time for Data ingest, Compression store, Data Cube building, Report generation etc
Real-time Dashboards via Spotfire Spotfire real-time interactivity with underlying Hadoop data
Implementation Team
• Client Engagement Manager
• Business Analyst – Requirement specification
• Data Scientist – Data & Platform architecture, Advisory & Sizing
• Data Modeler/Analyst - Design architecture and Data Model
• Big Data Engineer – Platform Implementation and ETL
• Data Analyst/s : Analytical work load
• Specialist Support if any, with vendor Involvement (If any, like Cloudera support)
• Post Go Live – Admin Roles for Maintenance & Risk
• Assumption – Delivery is on onsite basis
Big Data Platform Implementation - Impact
Case Study1: Big Data Implementation in payments & transactional intelligence
Client Context : Nodal agency for payments serving 600+ Banks/Payment product providers
Benefits:
Implementation costs 1/10th of similar implementations
Automated reporting, Complex query for analysis (for 6 products, multiple stakeholders across Banking/Govt)
High volume, High velocity, Real time, User Behaviour use cases
Big Data Platform Implementation - Impact
Case Study2: Leading Private Sector Bank
Client Context :
• Implementation of a Big Data platform (Hadoop) on premise on existing virtualized IT infrastructure
• Building a fully functional Assets Data Marts using data model, corresponding regulatory reports, and interactive visualization
• Implementation and testing of various data security and system security processes and technologies required for Banking applications and use-cases
Benefits: Archival data storage Performance DBs Real-time integration Secure environment
Analytics Services Insurance Analytics Suite
Customers
A SnapshotYour Digital Transformation Partner
Innovation Aggregators• Leverage digital technologies to disrupt business models, re-design customer experiences and
transform business processes based on
Innovation Intelligence CloudInnovation is no longer an option. It is central to digital transformation.
There is no dearth of data. Intelligence will be the differentiator
Innovation at scale is not possible without Cloud.
CloudIntelligenceInnovation
Digital Transformation Stack
InnoTx is your digital transformation partner. We help you translate your digital vision, and transform your business into a digital business. We bring disruptive technologies to
• create a first mover environment• provide a competitive edge• provide a platform to innovate
GovernmentIndustry
TransformationUtilities &
EnergyRetail
Banking & Insurance
Healthcare & Hospitality
Education
Technology-led Disruption
AI or Cognitive
IoTBig Data Analytics
eInsurance RPAPrivate, Public, Orchestration
Resourcing
Who we are. What we believe in
• Middle East arm of Teckraft InfosolutionsGroup, estd 2002.
• Team of 180+ professionals spread across India, MENA and Europe offering• Advisory
• Big Data Analytics
• Software Development
• Cloud
• Resourcing Solutions
• MENA office based out of Abu Dhabi, UAE
• We enable CIO/CTO/CDO in developing services portfolio for digital transformation.
• We challenge your vision and accelerate your digital transformation. Quantum vs Incremental changes
• Making the Innovative and disruptive technology accessible locally.
• We enable experimentation. PoCs are essential
• Rooted and aligned to Govt’s vision on Innovation and backed by UAE national’s investment
Logistics Pharmaceuticals ITES BFSI Manufacturing & Other International
Clients – Select List
19
