Emerging Big Data Standardization and Industry Efforts

1St GdR MaDICS Workshop on Big Data for 5G November 2015November 2015

Wael William Diab – Senior Director, Huawei

(wael.william.diab@huawei.com; wael.diab@gmail.com)

Acknowledgments

• Wo Chang – JTC 1/WG 9 Convenor

• Neil Frost – TC 204 South Africa HoD

• Jennifer Garner – JTC 1/WG 9 Secretary• Jennifer Garner – JTC 1/WG 9 Secretary

• Polly Okunieff; ISO/TC 204 Expert

• Yi Zhao –Huawei; INCITS EB Member

Agenda

Overview

Motivating Fields and Applications

Introduction of ISO/IEC JTC 1/WG 9

Example Emerging Application Areas and Developing Ecosystems

Final Thoughts

Overview

Introduction• A quick Google search on “Big Data Definition” returns thousands upon

thousands of hits• From analysts to ICT influencers, virtually every organization that touches IT has its

own definition on the topic

• It would be fair to say that there is a bit of confusion in the market place on what exactly someone means when they say “Big Data”

• So what is the good news?• So what is the good news?• This is an emerging field that cuts across IT (information technology) and OT

(operational technology)• Put differently, infrastructure managers and operations managers (I&O)

• Has the potential to be transformative • from how businesses operate…

• …to the digital / physical divide

• Within the large variety of definitions, there are consistent themes

• A number of emerging ecosystems being built around a Big Data paradigm

• There is a lot of excitement around this area… and that is why we are all here!

A Working Definition of Big Data

• Big Data is a data set(s) with characteristics (e.g. volume, velocity, variety, variability, veracity, etc.) that for a particular problem domain at a given point in time cannot be efficiently processed using current/existing/established/traditional technologies and techniques in order to extract value

• Key drivers in understanding the Big Data paradigm – how this is different from traditional data storage and compute / processing applicationsfrom traditional data storage and compute / processing applications

• Volume: too big• Velocity: arrives too fast• Variability: changes too fast• Veracity: contains too much noise• Variety: too diverse

• The applications generating this data or requiring its analysis may have one more of the above aspects present

• To understand why a Big Data Reference Architecture (BDRA) is needed, and the emerging ecosystems, we need to take a step back and look at some of the motivating applications

Roles in Big Data Ecosystem

• There are a variety of actors in an ecosystem. Below is a list of the different functional roles:

• Data Provider• introduces new data or information feeds into the ecosystem

• Big Data Application Provider• executes a life cycle (collection, processing, dissemination) controlled by the system orchestrator to

implement specific vertical applications requirements and meet security and privacy requirements

• Big Data Framework Provider• Big Data Framework Provider• establishes a computing fabric (computation and storage resources, platforms, and processing

frameworks) in which to execute certain transformation applications while protecting the privacy and integrity of data

• Data Consumer• includes end users or other systems who utilize the results of the Big Data Application Provider

• System Orchestrator• defines and integrates the required data application activities into an operational vertical system

• Security and Privacy• the role of managing and auditing access to and control of the system and the underlying data including

management and tracking of data provenance

• Management• the overarching control of the execution of a system, the deployment of the system, and its operational

maintenance

Motivating Fields and Motivating Fields and Applications

Big Data Analysis and Predictions

• Strong support that Big Data adoption is happening, cross-cutting and has significant implications and potential

• IDC – Predictions from the IDC FutureScape for Big Data and Analytics1. Visual data discovery tools will be growing 2.5 times faster than rest of the business

intelligence (BI) market. By 2018, investing in this enabler of end-user self service will become a requirement for all enterprises.

2. Over the next five years spending on cloud-based Big Data and analytics (BDA) solutions will grow three times faster than spending for on-premise solutions. Hybrid on/off premise grow three times faster than spending for on-premise solutions. Hybrid on/off premise deployments will become a requirement.

3. Shortage of skilled staff will persist. In the U.S. alone there will be 181,000 deep analytics roles in 2018 and five times that many positions requiring related skills in data management and interpretation.

4. By 2017 unified data platform architecture will become the foundation of BDA strategy. The unification will occur across information management, analysis, and search technology.

5. Growth in applications incorporating advanced and predictive analytics, including machine learning, will accelerate in 2015. These apps will grow 65% faster than apps without predictive functionality.

6. 70% of large organizations already purchase external data and 100% will do so by 2019. In parallel more organizations will begin to monetize their data by selling them or providing value-added content.

Big Data Analysis and Predictions• IDC – Predictions from the IDC FutureScape for Big Data and Analytics

7. Adoption of technology to continuously analyze streams of events will accelerate in 2015 as it is applied to Internet of Things (IoT) analytics, which is expected to grow at a five-year compound annual growth rate (CAGR) of 30%.

8. Decision management platforms will expand at a CAGR of 60% through 2019 in response to the need for greater consistency in decision making and decision making process knowledge retention.

9. Rich media (video, audio, image) analytics will at least triple in 2015 and emerge as the key driver for BDA technology investment.

10. By 2018 half of all consumers will interact with services based on cognitive computing on a 10. By 2018 half of all consumers will interact with services based on cognitive computing on a regular basis.

• Gartner• On weather or not “Big data hype or substance?”

• Beyond all the discussions, adoption of big data is simply inevitable

• Within key IT trends • Identifies Big Data expertise as essential

• Identifies Big Data expertise as needed within Web-scale I, IoT and others

• Benefits of big data are not limited solely to better decision making• fewer than half of big data projects focus on direct decision making

• most big data projects are geared to generating deeper business insights and optimizing, automating or even designing new processes

Big Data Market Projections

Broad Classification of Application Areas

• Government

• Research

• Life sciences

• Transportation

• Healthcare• Healthcare

• IoT• Industrial / manufacturing• Consumer

• Social media

• Enterprise • Process optimization• Insight

• Sustainability implications

Introduction: ISO/IEC JTC 1/WG 9

Overview• ISO/IEC JTC 1/WG 9 (Big Data)

• Working Group on Big Data

• Terms of Reference• ISO/IEC JTC 1/WG 9 was established by Resolution 28 during the 29th Plenary of the ISO/IEC

JTC 1 in November 2014• JTC 1 establishes JTC 1 Working Group 9 on Big Data reporting directly to JTC 1, with the

following terms of reference: • 1. Serve as the focus of and proponent for JTC 1's Big Data standardization program.

• 2. Develop foundational standards for Big Data – including reference architecture and vocabulary standards – for guiding Big Data efforts throughout JTC 1 upon which other standards can be developed.

Standardization

Focal Point

GAP Analysis 2. Develop foundational standards for Big Data – including reference architecture and vocabulary standards – for guiding Big Data efforts throughout JTC 1 upon which other standards can be developed.

• 3. Develop other Big Data standards that build on the foundational standards when relevant JTC 1 subgroups that would address these standards do not exist or are unable to develop them.

• 4. Identify gaps in Big Data standardization.

• 5. Develop and maintain liaisons with all relevant JTC 1 entities as well as with any other JTC 1 subgroup that may propose work related to Big Data in the future.

• 6. Identify JTC 1 (and other organization) entities that are developing standards and related material that contribute to Big Data, and where appropriate, investigate ongoing and potential new work that contributes to Big Data.

• 7. Engage with the community outside of JTC 1 to grow the awareness of and encourage engagement in JTC 1 Big Data standardization efforts within JTC 1, forming liaisons as is needed.

• Structure• As this is a working group, participation is by expert• Several national bodies are actively involved in the effort through JTC 1 such as US, JP, KR, CN,

UK, IR etc.

GAP Analysis

Industry

Engagement and

Promotion

Overview• Officers

• JTC 1/WG 9 Convener – Mr. Wo Chang (NIST) – USA

• JTC 1/WG 9 Secretariat – Mrs. Jennifer Garner – INCITS

• JTC 1/WG 9 liaison to JTC 1/SC 39 & ISO/TC 204 – Mr. Wael William Diab (Huawei) – USA

• Past meetingsPast meetings

• 2nd meeting: 7-9 July, 2015; Ciudad Real, Spain

• 1st meeting: 7-9 April, 2015; Bremen, Germany

• Future Meetings

• 3rd meeting: 1-4 December 2015; Brasilia (Federal District), Brazil

• 4th meeting: 8-10 March 2016

• teleconference schedule to be discussed at December 2015 meeting

• 5th meeting: 12-14 July 2016; Beijing (Tsinghua University), China

• 6th meeting: November/December 2016; Marrakech, Morocco (TBD)

Active Projects• ISO/IEC 20546 Information technology – Big Data – Overview and Vocabulary

• Scope: This International Standard provides an overview of Big Data, along with a set of terms and definitions. It provides a terminological foundation for Big Data-related standards.

• ISO/IEC 20547 Information Technology – Big Data Reference Architecture (5 Parts) • Scope: This International Standard specifies the Big Data Reference Architecture (BDRA). The

Reference Architecture includes the Big Data roles, activities, and functional components and their relationships.

• Part 1: (TR) Framework and Application Process • Scope: This technical report describes the framework of the Big Data Reference Architecture and the

International Standard Components of BDRAStrong focus on

security and • Scope: This technical report describes the framework of the Big Data Reference Architecture and the process for how a user of the standard can apply it to their particular problem domain.

• Part 2: (TR) Use Cases and Derived Requirements • Scope: This technical report would decompose a set of contributed use cases into general Big Data

Reference Architecture requirements.

• Part 3: (IS) Reference Architecture • Scope: This International Standard specifies the Big Data Reference Architecture (BDRA). The Reference

Architecture includes the Big Data roles, activities, and functional components and their relationships.

• Part 4: (IS) Security and Privacy Fabric • Scope: This international standard specifies the underlying Security and Privacy fabric that applies to all

aspects of the BDRA including the Big Data roles, activities, and functional components.

• Part 5: (TR) Standards Roadmap • Scope: This technical report will document Big Data relevant standards, both in existence and under

development, along with priorities for future Big Data standards development based on gap analysis.

security and

privacy from

the start

ISO/IEC Big Data Standards – A Big Roadmap

July 2015 Meeting Report• Logistical information

• July 2015 Working Group• Met in Ciudad Real, Spain

• Met for 3 days 7-9th

• Participants ~20 • Due to the last minute change of venue from South Korea to Spain due to MERS, there was less than

expected participants

• Key recommendations to JTC 1 Tremendous amount of industry collaboration• Key recommendations to JTC 1• Editor Appointments

• Editors for ISO/IEC 20546 reconfirmed

• Editors recommended for 5 parts of ISO/IEC 20547

• Liaisons• ISO/TC 204, Big Data, Mr. Wael William Diab (US) liaison representative from JTC 1/WG 9 to ISO/JTC

1/SC 39 and ISO/TC 204

• Other appointments from JTC 1/WG 9 to

• JTC 1/SC 29/WG 11, JTC 1/SC 32, JTC 1/SC 38, ISO/TC 69, ITU-T SG13

• Acknowledgment of incoming appointments to JTC 1/WG 9 from

• JTC 1/SC 6/WG 7, JTC 1/SC 29/WG 11, JTC 1/SC 32, JTC 1/SC 32/WG 1, JTC 1/SC 32/WG 2, JTC 1/SC 36, JTC 1/SC 38, JTC 1/SC 39, ITU-T SG13, ISO/TC 204

July 2015 Meeting Report• Key resolutions

• Project related

• Circulation of ISO/IEC 20546 Working Draft 1 (WG 9 N0087)

• Appointment of ISO/IEC 20546 Project Editors (WG 9 N0088)

• Request for ISO/IEC 20547 Title Change and Project Split (WG 9 N0089)

• As per current active projects shown on prior slide

• Circulation of ISO/IEC 20547 Part 3 3rd Editor's Draft (WG 9 N0092)• Circulation of ISO/IEC 20547 Part 3 3rd Editor's Draft (WG 9 N0092)

• Circulation of ISO/IEC 20547 Parts 1-2 and 4-5 Editor's Drafts (WG 9 N0090, N0091, N0093, N0094)

• Request for Approval of ISO/IEC 20547 Project Editors (WG 9 N0095)

• Future meetings

• Per schedule on prior slide

WG 9 Collaboration Opportunities

• JTC 1/WG 9 is very interested in working closely with other application groups interested in Big Data

• If you are involved in any organization where there may be interest in collaboration, please contact mecollaboration, please contact me

• If you are interested in participating in WG 9, please let me know if I can be of assistance and contact your NB

Example Emerging Application Example Emerging Application Areas and Ecosystems

It is All About the Application… Ecosystem

• The complexities associated with Big Data are a product of the increasing breadth of compute applications and how the information is being used

• If we think of the last inflection point in IT as the move towards solutions that address the increasing number and diversity of solutions that address the increasing number and diversity of applications, then Big Data is about understanding the ecosystem that these applications sit in

• There are probably endless areas of application where the underlying SDO community has identified the need to look at Big Data

• …and probably just as many where they have yet to do that

• I will focus, briefly, on three example areas• ITS, Industrial IoT, IT Sustainability

Intelligent Transportation Systems• Intelligent Transportation Systems (ITS) is focused on oInformation, communication and

control systems for transportation

• Within the SDO community, ISO TC 204 is a leading organization focused on ITS• Scope: Standardization of information, communication and control systems in the field of urban

and rural surface transportation, including intermodal and multimodal aspects thereof, travellerinformation, traffic management, public transport, commercial transport, emergency services and commercial services in the intelligent transport systems (ITS) field

• Created 1992• Total number of published ISO standards >200• 27 Participating Countries and 26 Observing Countries• 27 Participating Countries and 26 Observing Countries• Decoder ring on working group and active areas

• ISO/TC 204/WG 1 Architecture

• ISO/TC 204/WG 3 ITS database technology

• ISO/TC 204/WG 4 Automatic vehicle and equipment identification

• ISO/TC 204/WG 5 Fee and toll collection

• ISO/TC 204/WG 7 General fleet management and commercial/freight

• ISO/TC 204/WG 8 Public transport/emergency

• ISO/TC 204/WG 9 Integrated transport information, management and control

• ISO/TC 204/WG 10 Traveler information systems

• ISO/TC 204/WG 14 Vehicle/roadway warning and control systems

• ISO/TC 204/WG 16 Communications

• ISO/TC 204/WG 17 Nomadic Devices in ITS Systems

• ISO/TC 204/WG 18 Cooperative systems

ISO TC 204 and Big Data• Interest in and motivation for Big Data

• Each one of the 12 active working groups of TC 204 expressed interest in Big Data

• At a high level, the interest is motivated by the notion that each one of the working groups can provide at least one use case

• Moreover, each working group has one or more applications that act as a data source for Big Data

• At the 2015 Fall Plenary in Germany, a Big Data workshop was held • Keynotes by IEEE, SAE, and others with kickoff by JTC 1/WG 9

• Heavy interest in establishing a group within TC 204

• Emphasis on use cases, privacy and security

• Term “Internet of Transportation” thrown around several times

• Interesting aspects on the 5Vs as different use cases stretch them in opposite directions• Interesting aspects on the 5Vs as different use cases stretch them in opposite directions• E.g. Semi-persistent data

• Working groups already looking at analytics

• Resulting resolutions at plenary (1131 and 1132)• 1131

• ISO/TC204 notes:

• the increasing importance on big data in the transport domain;

• ISO/TC204 resolves to set up an ad-hoc group to study the domain of Big Data and its applicability for Intelligent Transport Systems standardisation and appoints Neil Frost, Head of Delegation from South Africa, as the ad-hoc group leader. NSOs are invited to nominate experts to the ad-hoc group. A first draft report will be presented at the ISO/TC204 meeting in April 2016.

• 1132• ISO/TC 204 welcomes the offer of JTC 1/WG 9, Big Data to collaborate with ISO/TC204 on this topic within the liaison installed

between JTC1 and ISO/TC204;

• ISO/TC204 appoints Mr. Wael William Diab (ANSI) as the liaison representative from ISO/TC204 to JTC1/WG9

Example Big Data Use Cases Shown

Industrial Internet of Things (IoT)

• The nature of IoT lends itself well to Big Data

• One can argue that Big Data is an extension of IoT and vice versa

• Value of IoT is in performing analytics to take action on dynamic data

• Industrial applications are not different• Industrial applications are not different

• Tremendous sensor generated data

• Increased importance of IT for operation KPIs e.g. OEE

• Industrial Automation moving from connectivity to services model

• Emerging ecosystems

• Comprehensive communities such as IIC, AIOTI, Industry 4.0

• Standards organizations enabling the transition such as WG 10, oneM2M, P2413

Industrial Internet Consortium • Industry driven organization focusing on

• Holistic requirements and architecture for industrial IoT

• Test beds

• Use cases

• Inaugural IoT Solutions World Congress

• Membership and leadership• 209 Members (October 2015)

• Leadership

• 5 Founding Members (ATT, Cisco, GE, IBM, Intel)

• 2 Contributing Members (SAP, Schneider)

• 1 Nonprofit/Academia category (MITRE)

• 1 Small Industry category (RTI)• 1 Small Industry category (RTI)

• 2 Large Industry category (Fujitsu, Bosch)

• 1 IIC Executive Director (ex-officio)

• Example Approved Test Beds• GE: High Speed Network

• Intel: Transportation Grand Challenge-Business Collaboration

• Bosch/TechMahindra: Track and Trace

• EMC: SW Defined Cloud for IoT

• EMC: INFINITE

• RTI/Cisco/NI: Communication and Control Framework for MicroGrid Applications

• Hewlett-Packard/RTI: Edge Intelligence

• IBM/NI: Predictive Maintenance

• Infosys/PTC/Bosch/Intel: Asset Efficiency

• Fujitsu: Factory Operations Visibility and Intelligence

• Infosys/GE: Industrial Digital Thread

• Cisco/GE/Bosch/NI/KUKA/TTTech: Time Sensitive Networks for Flexible Manufacturing

IIC Reference Architecture: Functional Decomposition of Information, Application and Business Domains

Magnitude of the Industrial IoT Investment

Source: IDC, February 2014

30

IT Sustainability

• Metrics on Big Data that reflect energy cost

• Example with ISO/IEC JTC 1/SC 39

• Sustainability by and for Information Technology

• Currently has two working groups• Currently has two working groups

• WG 1 is looking at KPIs relating to datacenters and datacenter equipment

• Interest by JTC 1/SC 39

• To examine BDRA when ready to see if the KPIs developed need amending, enhancement etc

• Provide feedback on architecture for inefficiencies

Summary of Related SDOs working on BD• ISO/IEC JTC 1/SC 32

• Data management and interchange, including database languages, multimedia object management, metadata management, and e-Business

• ISO/IEC JTC 1/SC 38• Standardization for interoperable Distributed Application Platform and services including Web Services,

Service Oriented Architecture (SOA), and Cloud Computing

• ITU-T SG 13• Cloud computing for Big Data

• W3C• W3C• Web and semantic related standards for markup, structure, query, semantics and interchange

• OGC (Open Geospatial Consortium)• Geospatial related standards for the specification, structure, query, and processing of location related data

• OASIS (Organization for the Advancement of Structured Information Standards)• Information access and exchange

• TPC (Transaction Processing Performance Council)• Benchmarks for Big Data systems

• TM Forum• Enable enterprise, service providers and suppliers to contagiously transform in order to succeed in the digital

economy• Share experiences to solve critical big data issues

Final Thoughts

Final Thoughts• The BDRA is an important and necessary component for emerging and

future IT and OT infrastructures• Targeted to address the issues that stem from increasingly complex data sets

• 5Vs: Volume (too big), Velocity (arrives too fast), Variability (changes too fast), Veracity (contains too much noise), Variety (too diverse)

• Targeted to address the increasing applications and needs (analytics, insights, decision making) on the data setsdecision making) on the data sets

• From an industry perspective, seeing two aspects develop• Key organizations focused on BDRA• Ecosystems building around tradition communities / SDOs that look at the specific

use case and application space• E.g. Intelligent Transportation Systems, Industrial IoT, Smart Cities

• ISO/IEC JTC 1/WG 9• Recently formed central focal point within ISO and IEC for Big Data• Engaging communities to pull in use cases, experts and foster early adoption

• If interested, please do not hesitate to contact me

• Initial work items focused on vocabulary and BDRA

References

• ISO/IEC JTC 1 Report from WG 9 Convener

• ISO/IEC JTC 1/Big Data Study Group Final Report

• ISO/IEC JTC 1/SC 39

• IIC Reference Architecture• IIC Reference Architecture

• ISO TC 204 Big Data Workshop

• Wikibon Big Data Market Forecast

• Publically available Big Data Reports from IDC and Gartner