the convergence of data technologies and information · speculates that yale library will contain...

The Convergence of Data Technologies and Information

David Iverson, Data Architect

ABOUT “Dave from State Farm”• Army Vet (1970 – 1976)• National Blue Shield Association (Systems Analyst)• CNA Insurance Companies (Director, Systems and Admin

for Professional Liability Division)• LOMAH Consulting (Founder/President for 6 years)• IBM (Instructor and Consultant)• State Farm (14 years)• 43 years of Data and Insurance experienceAnd Outside of Work…• Married (Maureen), 5 daughters, 10 grandchildren• Co-publisher of Sportsmen Magazine• Outdoor Enthusiast• Love woodworking

ABOUT STATE FARM

6

Our Locations

HeadquartersBloomington, IL

Dallas, TXCity Line

Phoenix, AZMarina Heights

Atlanta, GAPerimeter

© Copyright, State Farm Mutual Automobile Insurance Company, 2015

State Farm Campus location @ CityLine

Let’s Drill Down…

Those Numbers• 12.8M Claims / year (35,000/day)• 82.1M Policies in Force• More Agent Locations than…

• McDonalds• Starbucks• Walmart Stores

Takes some serious processing & IT

capabilities


IT @ State Farm

The Network• One of the world’s largest networks• Network of 60,000 network nodes, and 24,000

network links installed• Connects nearly 18,000 agents and 65,000

employees • Provides support for 625,000 devices


IT @ State Farm

• That focuses on providing employees with the latest tools and hardware

• Where Data professionals able to learn at one of the largest OLAP and Analytical environments in the world

• Experience application development as part of one of the largest development communities in North America.

• Utilizing the latest technologies - Open Source, Rules Engine, Service Oriented Architecture (SOA)

The Technology


Then you have Data @ State Farm

• More data on hand than the Library of Congress• Over 8.3 PETABYTES of Storage Area Network

(SAN) – 70+ Petabytes total storage• Over 300,000 servers supported by Systems• Over 140 million lines of code in COBOL, PL/1 and

Assembler• Largest single table – 6 TB, 254 parts, 13.7 billion

rows

The Data


And then there are the people…

Data Professionals @ State FarmData Design Data Scientist

Business Intelligence (BI)

Data Movement Data EngineerDBA/DBSA

Approximately 11,000 IT professionals


Disclaimer

The content of this presentation and the following discussion reflects my own personal views and opinions and may not represent State Farm’s positions or practices.

Now A Disclaimer…

ARE YOU READY FOR CHANGE?

Data vs Information…

Source: The Internet “of course” (Dictionary.com)

2005

The term data refers to factual information, especially that used for analysis and based on reasoning or calculation. Data itself has no meaning, but becomes information when it is interpreted. Information is a collection of facts or data that is communicated.

Data and its History…

Data Technology and its History…

2005

The following timeline is sourced from Bernard Marr’s book entitled “BIG DATA”

Data Technology and its HistoryHumans use tally sticks to record data for the first time. These are used to track trading activity and record inventory.

The abacus is developed, and the first libraries are built

The Library of Alexandria is the world’s largest data storage center – unit it is

destroyed by the Romans.

The Antikythera Mechanism –the first mechanical computer – is developed in Greece

The term “business intelligence” is used by Richard Millar Devens in his Encyclopedia of Commercial and Business Anecdotes

John Graunt conduct the first recorded statistical-analysis experiments in an attempt to curb the spread of the bubonic plague in Europe

Herman Hollerith creates the Hollerith Tabulating Machine which uses punch cards to vastly reduce the workload of the US Census

Nikola Tesla predicts that in the future, a man will be able to access and analyze vast amounts of data using a device small enough to fit in his pocket.

Source: Bernard Marr “Big Data”

Data Technology and its HistoryFritz Pfleumer creates a method of storing data magnetically, which forms the basis of modern digital data storage technology

Fremont Rider speculates that Yale Library will contain 200 million books stored on 6,000 miles of shelves, by 2040

Hans Peter Luhn defines Business Intelligence as “the ability to apprehend the interrelationships of presented facts in such a way as to guide action towards a desired goal”

The US Government plans the world’s first data center to store 742 million tax returns and 175 million sets of fingerprints on magnetic tape


Data Technology and its HistoryRelational Database model developed by IBM mathematician Edgar F Codd.

Materials Requirements Planning (MRP) systems are commonly used

Early use of term Big Data

The birth of the internet.

Google launch their search engine

The price of digital storage fall to the point where it is more cost-effective than paper.

First use of the term Internet of Things

Three “Vs” of Big Data –Volume, Velocity, Variety –defined by Doug Laney

Hadoop and the birth of Web 2.0

Globally 9.57 zettabytes (9.5 trillion gigabytes) of information is processed by the world’s CPUs.

The average US company with over 1000 employees is storing more that 200 terabytes


Data Technology and its HistoryAs much data is now being created every two days, as was created from the beginning of human civilization to the year 2003

The shortfall of between 140,00 and 190,00 professional data scientist by 2018

Mobil internet use overtakes desktop for the first time.


Information (Knowledge) Growth…

• Until 1900 – Knowledge didn’t double until then

• By 1942 – Knowledge doubling every 25 years

• 2000+ – Nanotechnology doubling every two years

• 2000+ – Clinical knowledge doubling every 18 months

• 2000+ – Human knowledge doubling every 13 months

• Today - According to IBM, with the “internet of things” will

lead to the doubling of knowledge every 12 hours.

Buckminster Fuller “Knowledge Doubling Curve”

What happens with this convergence…Data Sources/Technologies

OLTP, ERP, CRM Systems

Document & emails

Weblogs & Click streams

Social Networks

Machine Generation

Sensor Data

Geo-Location

Rapid increase in Methods of

Access

Storage

Storage Technologies

Information Growth Explosion

The IMPACT of this convergence…Ec

onom

ic In

fluen

ces • Capital

Investments• Overall cost• Pricing• Labor

Opp

ortu

nity

Gro

wth • .com

• Storage advancements

• Consulting• Training• etc. M

oder

niza

tion • Changing

technologies• Coding

languages• Impact on

societies• Impact on

school curriculums

The IMPACT of this convergence…Co

mm

unic

atio

n • Speed• Transformation• Inclusiveness• Technology

Pers

onal

Dev

elop

men

t • School Curriculums

• Internal offerings

• Re-skilling and• Re-tooling• Keeping up

Org

aniza

tiona

l Im

pact

s • Changing technologies

• Centralization vs Distributed

• Training• Re-skilling and• Re-tooling

What History Tells Us…

• Times (in all aspects) are changing almost faster than we can keep up with!

• No let up it sight and in fact the pace will increase!

• We can’t sit back on our past laurels – we have to adjust and morph with the times

ALL OF THIS RESULTS IN IMPACTS AND CHALLENGES FOR

INDIVIDUAL CAREERS AND

HOW COMPANIES ADJUST TO HANDLE BOTH THE INDIVIDUALS AND ORGANIZATIONAL CHANGE

ADDRESSING CHANGES FROM 2 VIEWS…

How does this affect your role as a Data Professional?…

SESSION #1

BUT FIRST…

?

What affects the Data Professional…

• Availability and speed of collecting data• Increasing volume

• Regulations and controls• Sensitive data (Privacy)• Business Analytics and Data Science• Data Storage Technologies• Foundational Data Education

• Changing types of data – structured & unstructured

• Anything else?

What are the challenges for Data Professional?…

Sometimes the challenge is just people !

Challenges for the Data “people”…

6. Focus on analytics vs design

1. Projects go… “straight to code”

2. Fast changing technologies / tools

3. Increased scope of work for the data modeler

4. Competing or overlapping skillsets

5. Structured vs unstructured “to be or not to be”

7. What the business wants to see (next slide)

Business wants to see this…

Not this…

Challenges for the Data “people”…

6. Focus on analytics vs design

1. Projects go… “straight to code”

2. Fast changing technologies / tools

3. Increased scope of work for the data modeler

4. Competing or overlapping skillsets

5. Structured vs unstructured “to be or not to be”

7. What the business wants to see

• Anything else?

How do we meet the challenges for Data Professional?…

Hoberman nailed it…

Steve Hoberman “Data Modeler of 2020”

http://stevehoberman.com/

Meeting the challenge

Meeting the challenge

6. Broaden you own skillset

SOURCE: *Steve Hoberman’s Video “Data Modeler of 2020”

1. Become a Data Structure Scientist

2. Ride the wave caused by the perfect storm

3. Learn a Graph or Document Database

4. Master a Fact Based Modeling (FBM) dialect

5. Build and maintain an Enterprise Logical Data

Model (LDM)

Meeting the challenge…


SOURCE: *Steve Hoberman’s Data Modeler of 2020

• Taking complexity and making it simple for the world (Business) to understand

• Data modeling (design) is a process of representing real world complexities in a precise and simple form (the data model), to solve important problems (meet the business challenges)

• Using data modeling (design) in non-traditional ways (a Data Structure Scientist)




A. Agile + Cloud + Big Data = Perfect Stormo Agile – projects without any modeling is the

trendo Cloud – store data outside organization.

Circumventing data structureso Big Data – Easy to use analytical tools –

circumventing data modeling tools




A. Ride the wave (continued)o We must become more agile – don’t be

legalistic – is 3NF ERDs absolutely necessary?o Don’t explain normalization – go with the flow –

be creativeo Document your questions – using templates

such as Who, What, When Where, Why, How




B. Become a DBA (Database Architect)o Advise and recommend on data storage

technologies (RDBMS vs NoSQL; Cloud or not; Agile or not)

o Develop a decision tree for your company to make those decisions

o Adapt to technology changes as they occur




C. Sharpen your influencing skills (Soft skills)o Learn the art of influencing – attend seminars,

etc.o Learn the technologies and understand


3. Learn a graph or document database


• Learn how in forward and reverse engineer to and from the databases (i.e. a LDM to Hadoop implementation)

• Simplify and Organize

• Choose something like graph (Neo4J) or document (MongoDB) and be able to discuss




• Extract out facts from the business

• Develop semantics in structure – talking points

• Modeling by examples – be creative

• Example: Object Role Modeling (ORM)


5. Enterprise Logical Data Model


• Long history – many attempts – coming back

• Develop a single view of the business (as-is)

• Metadata, Data Governance, Data Management

HOBERMAN’S VIEW…


Project RDBMS and Some NoSQL Enterprise Project Some

NoSQL

CDM (ERD) CDM (ERD)

LDM (ERD) LDM (ERD) LDM (FBM)

PDM (ERD) IMPL


6. Broaden your skillset

• Add to your repertoire – Skills + 2 or more Most college level – entry level new hires have

some language skills like “R” or Python or some JAVA, C++, C#, or SQL, if not already known

Data Mining & Data Visualization• Requirements/Process modeling a plus

Repeating the list…

6. Broaden you own skillset



3. Learn a Graph or Document Database


5. Build and maintain an Enterprise Logical Data

Model (LDM)

• Anything else?

What career path does one take?…

POINT A

Data Modeler

POINT B

Data Structure Scientist

BEYOND THE MAP

Data ???

The Data Professional’s Map (Career)…

Let’s summarize…

1. Times are changing fast

2. As Data Professionals we must be able to adapt and morph to the changing needs

3. We must expand our skillsets

4. Think beyond the map!

How does this affect your Organization?…

SESSION #2

BUT FIRST Do We Need Another…

What affects the Organization…

• Availability and speed of collecting data• Increasing volume

• Regulations and controls• Sensitive data (Privacy)• Business Analytics and Data Science• Data Storage Technologies• Foundational Data Education

• Changing types of data – structured & unstructured

• Anything else?

What are the challenges the Organization?…

Challenges for the Organization…

1. Projects go… “straight to code”2. Fast changing technologies / tools3. When to engage “data experts” on projects4. Keeping up with changing industry titles5. Transient workforce 6. Impact on culture of company7. Educating the non-technical and non-data “folks”8. How to re-skill & re-tool your employees (training)9. How to organize – (structure your data functions and people)

• Anything else?

OPERATIONAL ANALYTICSDATA MANAGEMENT

Transactional Data Analysis Data Analytics (includes BI and Data Science)

Event drivenHands on exploration and evaluation

Structure design/data movement Stewardship

Production Converting data into intelligence Historical Trending Metadata

Runs the business Data quality Discovering patterns Governance

Reporting (dashboards, etc.) Manages the data

Ask data related questions

Diagnostics

Today’s data role environment…

Data Modelers

Data Analysts

Analytical Data Modelers

Database Administrators

By Committee

Architects

OPERATIONAL ANALYTICSDATA MANAGEMENT

Transactional Data Analysis Data Analytics BusinessIntelligence Data Science

Event drivenHands on exploration and evaluation

Predictive capabilities

Historical Trending

Create sandboxes to determine value Stewardship

Production Converting data into intelligence

For the mathematically/technically inclined

Insights in reports, dashboards,etc.

Diagnosticsanalytics Metadata

Runs the business Data quality Discovering patterns

Ask data related questions

Governance

Manages the business

Not necessarily IT Manages the data

Future state (now through near future…)

Business Intelligence Data Scientist

Data Modelers/Data Designer

Database Administrators

Data Architects/Engineers

By Committee

What are Your Suggestions for the Organization being a Pacesetter, or a Corporate Visionary when it comes to

managing the roles of Data Professionals ?…

Let’s summarize…

1. Times are changing fast

2. Plan for re-skilling and re-tooling

3. Organize for the future (depends on size)

4. No easy answers

“Good hockey players skate to where the puck is, but GREAT hockey players skate to where the puck will be”

-Wayne Gretzky

Final Thought…

the convergence of data technologies and information · speculates that yale library will contain...

Documents