dataops: the collaborative framework for enterprise data ... · dataops: the collaborative...

14
Copyright © 2017 Blue Hill Research Page 1 ANALYST INSIGHT DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number: A0287 Analyst: Toph Whitmore, Principal Analyst What You Need to Know DataOps is an enterprise collaboration framework that aligns data-management objectives with data-consumption ideals to maximize data-derived value. DataOps “explodes” the information supply chain to create a data production line optimized for efficiency, speed, and monetization. Borrowing from production optimization models and DevOps theory, DataOps’ successful adoption requires adherence to three key principles: Global Enterprise Data View: Define data journeys from source to action to value delivered, and measure performance across the entire system. Collaborative Thinking: Structure organizational behavior around the ideal data-journey model to maximize data-derived value and foster collaboration between data managers and data consumers. Get in Front of Data: Decentralize, then empower self-service data services and analytics throughout the organization. The Old Way of Enterprise Data Operations: Siloed, Slow, Reactive It’s not a radical speculation to say that most modern enterprises could derive more value from their data. Too often, data operations progress is stagnant, stuck in modes of centralized data-querying point-solution-reinforced data silos, and poor data integrity. Unfortunately, this is what the modern data-driven organization has come to: an impasse characterized by a less-than-constructive friction created, fostered, and reinforced by the walls built up around functional silos. On one side sit line-of-business (LOB) stakeholders, emboldened by easy access to “newly-democratized” data, demanding more self-service power … as long as it comes with further abstraction from data administration. Pitted against them on the other side of the artificial enterprise barrier are IT leaders—CIOs, data scientists, DBAs—desperate to deliver service, but challenged to retain control over corporate data. AT A GLANCE DataOps Framework DataOps offers a collaborative approach for orchestrating enterprise data flows to maximize value delivery. The Approach Take a global enterprise view of data flows Break down silos to collaborate Get in front of data

Upload: trinhnhi

Post on 03-Jul-2018

215 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: DataOps: The Collaborative Framework for Enterprise Data ... · DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number:

Copyright © 2017 Blue Hill Research Page 1

ANALYST INSIGHT

DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration

Published: January 2017 Report Number: A0287

Analyst: Toph Whitmore, Principal Analyst

What You Need to Know DataOps is an enterprise collaboration framework that aligns data-management objectives with data-consumption ideals to maximize data-derived value. DataOps “explodes” the information supply chain to create a data production line optimized for efficiency, speed, and monetization.

Borrowing from production optimization models and DevOps theory, DataOps’ successful adoption requires adherence to three key principles:

• Global Enterprise Data View: Define data journeys from source to action to value delivered, and measure performance across the entire system.

• Collaborative Thinking: Structure organizational behavior around the ideal data-journey model to maximize data-derived value and foster collaboration between data managers and data consumers.

• Get in Front of Data: Decentralize, then empower self-service data services and analytics throughout the organization.

The Old Way of Enterprise Data Operations: Siloed, Slow, Reactive It’s not a radical speculation to say that most modern enterprises could derive more value from their data. Too often, data operations progress is stagnant, stuck in modes of centralized data-querying point-solution-reinforced data silos, and poor data integrity.

Unfortunately, this is what the modern data-driven organization has come to: an impasse characterized by a less-than-constructive friction created, fostered, and reinforced by the walls built up around functional silos. On one side sit line-of-business (LOB) stakeholders, emboldened by easy access to “newly-democratized” data, demanding more self-service power … as long as it comes with further abstraction from data administration.

Pitted against them on the other side of the artificial enterprise barrier are IT leaders—CIOs, data scientists, DBAs—desperate to deliver service, but challenged to retain control over corporate data.

AT A GLANCE

DataOps Framework DataOps offers a collaborative approach for orchestrating enterprise data flows to maximize value delivery.

The Approach • Take a global enterprise

view of data flows

• Break down silos to collaborate

• Get in front of data

Page 2: DataOps: The Collaborative Framework for Enterprise Data ... · DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number:

Copyright © 2017 Blue Hill Research Page 2

ANALYST INSIGHT

Meanwhile, thanks to new technologies and new operations (e.g., IoT and commodity clusters), enterprise data grows exponentially, even while the organization’s ability to manage and then ultimately consume said data doesn’t scale to accommodate those new volumes.

With apologies to Abraham Lincoln, an enterprise divided against itself cannot stand. Or, more importantly, compete in the marketplace.

It wasn’t supposed to be like this. Cheap storage (in particular, easy-to-procure-or-spin-up-on-AWS Hadoop storage) would make it easier to manage more data. Data lakes enable different data types to commingle (“SQL. And you?” “Flat-file. Come here often?”), but rely upon downstream interpretation and integration for data consumption. Self-service data tools offer a seductive empowerment for LOB managers eager to get at the data, but that easy access comes with the high operational price of new risks.

Even progressive, data-committed enterprises often take too narrow a view. Functional silos (marketing, finance, R&D, customer service) are comfortable, but they can be obstacles when it comes to deriving value from enterprise data. Antiquated obstacles and obstreperous functional prejudices are preventing progress in some really stupid ways. Ever work in marketing? How good was your data? How good was your marketing-automation system? Did it integrate well(-enough!) with your CRM? Or your ERP? Accounting system? Or third-party data? Ever work in finance? Did your team have enough time to analyze your corporate data and prepare all your reports? What about bandwidth to adapt to change, or to consume new data and dashboards?

Data—and the value derived from it—dictates success in the modern enterprise. Enterprises that can exploit data to derive value will recognize new revenue, see new efficiencies, and enjoy intangible benefits like strengthened customer relationships and greater marketing efficiency.

Introducing DataOps: Collaborative, Cross-Functional, Creative We’ve been going about this all wrong. We need an operational model that rewards data growth, that offers data consumption capability that scales as easily as an Azure-based Hadoop cluster. We need a management model that starts with cross-functional alignment (or, even better, eliminates functional silos completely) instead of one that adds bricks to divisive organizational walls.

It’s time for a new approach. It’s time for DataOps. DataOps is a way to get enterprise IT and enterprise LOB stakeholders—this is big—to work together toward common objectives. More specifically,

DataOps is an enterprise collaboration framework that aligns data-management objectives with data-consumption ideals to maximize data-derived value. DataOps “explodes” the information supply chain to create a data production line optimized for efficiency, speed, and monetization.

Deriving Functional Value from Enterprise Data For more information on data monetization, read my recent Analyst Insight: Data Monetization: Who’s Doing It Right (Now).

Page 3: DataOps: The Collaborative Framework for Enterprise Data ... · DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number:

Copyright © 2017 Blue Hill Research Page 3

ANALYST INSIGHT

DataOps isn’t a new term:

• Gartner sought to define “data ops” as a centralized hub for IT leaders to parcel out data, with the overarching mission to “control” consumer access to systems of record.

• In 2014, consultant and InformationWeek contributor Lenny Liebman more constructively defined DataOps as “the set of best practices that improve coordination between data science and operations.”

• More recently, Tamr CEO Andy Palmer called for a DataOps approach to foster collaboration between Data Engineering, Data Integration, Data Quality, and Data Security/Privacy functional delivery. (Read his 2015 introduction to DataOps. And read the Blue Hill Research August 2016 interview with Palmer. )

(Read the Blue Hill Research blog post on the genesis, brief history, and new definition of DataOps here. This work builds upon the definitions proposed by Palmer and Liebman.)

Gene Kim, Kevin Behr, and George Spafford applied advanced management theory to software development and IT operations to define DevOps in their seminal 2013 work The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win. And The Phoenix Project was itself influenced by Eliyahu Goldratt’s 1984 manufacturing-optimization novel (pretty specific genre, but yes, there is such a thing) The Goal: A Process of Ongoing Improvement.

In the same way that The Goal evangelized the radical idea of production and management working together to improve throughput and reduce costs, and in the same way that The Phoenix Project posited the unheard-of notion that software development and IT management were actually on the same team, DataOps aims to align data-management goals with data-consumption ideals.

The Road to Effective DataOps There are three stages to deploying a DataOps approach in the modern enterprise. The phases analogously mirror the three “ways” of DevOps (systems thinking, feedback-loop amplification, and fostering experimentation culture) popularized by Kim, Spafford, and Behr in The Phoenix Project.

In a DataOps context, incremental adoption begins with a big picture view, breaks down silos to support delivery, and then empowers consumers with data democratization. The three steps to DataOps adoption are:

1. Global Enterprise Data View: Define data journeys from source to action to value delivered, and measure performance across the entire system.

2. Collaborative Thinking: Structure organizational behavior around the ideal data-journey model to maximize data-derived value and foster collaboration between data managers and data consumers.

3. Get in Front of Data: Decentralize, then empower self-service data discovery and analytics throughout the organization.

Let’s look at each and examine some case examples of enterprise innovators using DataOps principles.

Page 4: DataOps: The Collaborative Framework for Enterprise Data ... · DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number:

Copyright © 2017 Blue Hill Research Page 4

ANALYST INSIGHT

1. Global Enterprise Data View: Defining the Data Journey in Its Entirety Traditional assembly-line optimization theory begins with an “exploded” view of the production process. Students of manufacturing know the schematic: inputs on the right, feeding into process steps, with work-in-process (WIP) stations, cycle times, etc. The production steps narrow to final assembly, and the left-side-of-the-sideways-Christmas-tree-like-illustration output is the completed widget (or commonly in manufacturing class examples, a bicycle).

In The Goal, Goldratt applied the theory of constraints to production-line process management. Kim, Behr, and Spafford then related bottleneck-optimization modeling to integrated software development and IT operations.

In the enterprise data world, too often we manage and process data in isolation: the marketing analyst blocked from access to financial reporting, the software developer with no visibility to consumer purchase behavior, the customer support lead without the full view of multi-channel engagement. Enterprise data leaders that maximize data-derived value take a comprehensive view, and understand how data—unified data from a wealth of sources—informs action.

In the modern enterprise, data-informed action must deliver value, in whatever tangible or intangible form that takes. In DataOps theory, the data flow serves as the enterprise nervous system around which operational workflows are structured. Much like the “exploded” bicycle view (“Station #2: Attach left pedal”), data flow models map from output—maximized value, however it is defined or measured—backwards to source or data creation point.

So What Does DataOps Look Like?

Picture a “series of tubes, ” or perhaps “something out of the movie ‘Brazil’.” Whatever metaphoric form it takes, the exploded data flow network defines the journey of a single piece of data from source (or creation) to value-driving action.

The exploded view of a DataOps assembly line looks a bit like a factory-floor layout schematic. (Refer to Figure 1 below.)

Page 5: DataOps: The Collaborative Framework for Enterprise Data ... · DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number:

Copyright © 2017 Blue Hill Research Page 5

ANALYST INSIGHT

Figure 1. DataOps "Exploded-View" Conceptual Assembly-Line Context Diagram

Source: Blue Hill Research, January 2017

The conceptual schematic design starts with action—“the right thing to do” triggered by the data consumer’s insight—and works back to the contributing data sources. Each phase of the journey emphasizes its own value delivery: input to curation to transformation to consumption to insight to action. It’s the new responsibility of DataOps leaders to define, then accelerate that journey.

After designing the ideal data journey, the next steps are to establish effective performance measurement techniques, identify bottlenecks, optimize—then repeat. That starts with the enterprise data value chain in Figure 2 below.

Figure 2. Enterprise Data Value Chain

Source: Blue Hill Research, January 2017

Flipping that around, the enterprise data value chain aligns with each stage of the DataOps assembly-line model. (See Figure 3 below.)

Input Curation Transformation Consumption Insight Action

Page 6: DataOps: The Collaborative Framework for Enterprise Data ... · DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number:

Copyright © 2017 Blue Hill Research Page 6

ANALYST INSIGHT

Figure 3. DataOps "Exploded-View" Conceptual Assembly-Line Context Diagram With Enterprise Data Value Chain Overlay

Source: Blue Hill Research, January 2017

In practice, data integrity, security, machine-learning, and process improvement are horizontal functions that form a vertical stack. (See Figure 4 below.)

Figure 4. DataOps Data-Governance Stack

Source: Blue Hill Research, January 2017

That data-governance stack touches each phase of the idealized DataOps workflow:

Page 7: DataOps: The Collaborative Framework for Enterprise Data ... · DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number:

Copyright © 2017 Blue Hill Research Page 7

ANALYST INSIGHT

Figure 5. DataOps "Exploded-View" Conceptual Assembly-Line Context Diagram With Enterprise Data Value Chain and Horizontal Data-Governance-Stack Overlays

Source: Blue Hill Research, January 2017

Taking an enterprise view of data flows requires more than just a data-journey map. To achieve DataOps journey-mapping success, IT leadership must:

• Establish progress and performance measurements at every stage of the data flow. Where possible, benchmark data-flow cycle times.

• Define rules for an abstracted semantic layer. Ensure everyone is “speaking the same language” and agrees upon what the data (and metadata) is and is not.

• Validate with the “eyeball test”: Include continuous-improvement-oriented human feedback loops. Consumers must be able to trust the data, and that can only come with incremental validation. (Done right, the scope and scale of those validation checks declines as accuracy improves and the entire system gets smarter.)

• Automate as many stages—even BI, data science, and analytics, when possible—of the data flow.

Page 8: DataOps: The Collaborative Framework for Enterprise Data ... · DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number:

Copyright © 2017 Blue Hill Research Page 8

ANALYST INSIGHT

• Using benchmarked performance information, identify bottlenecks and then optimize for them. This may require investment in commodity hardware, or automation of a formerly-human-delivered data-science step in the process.

• Establish governance discipline, with a particular focus on two-way data control, data ownership, transparency, and comprehensive data-lineage tracking through the entire workflow.

• Design process for growth and extensibility. The data flow model must be designed to accommodate volume and variety of data. Ensure enabling technologies are priced affordably to scale with that enterprise data growth.

2. Collaborative Thinking: Align Goals, Unify Data Across Silos At an admittedly basic level, DataOps segments enterprise data principals into two groups: those who manage enterprise data, and those who consume it. Data managers are tasked with storing, curating, and serving data to line-of-business (LOB) stakeholders who then analyze it and take action.

DataOps data-journey mapping deconstructs the data flow from source to action, with purposely little regard for corporate silos. In DataOps, silos can obstruct—process and data flow must be as transparent as possible. For example, an enterprise marketing organization can work effectively with a sophisticated marketing-automation solution (say, Hubspot or Marketo). But that team won’t recognize revenue impact or gain a “customer 360” view if that marketing system is not integrated with the corporate CRM, financial-reporting, and/or customer-support systems.

DataOps collaboration must bridge corporate silos and break down organizational prejudices. DataOps collaboration depends upon:

• Open channels of communication between functional stakeholders

• C-suite commitment to process, accountabilities, and objectives

• Shared priorities

• Shared trust in the data

• Shared reward systems based on data-derived value maximization

There are multiple ways to achieve DataOps collaboration in the modern enterprise. But good management leads to good data management, and C-suite buy-in is therefore a must. Two organizational behavior management approaches to consider: cross-functional team assignments and Centers of Excellence (CoE).

Some progressive organizations have restructured into cross-functional product and data teams. (Holacracy being a rather extreme example.) Putting an IT lead, marketer, and software developer on the same product team (or even in the same co-located, Ikea-furnished open workspace) isn’t antithetical to productivity, at least when accountability is established, decision-making is defined, and data-oriented tasks are aligned to each stage of the data flow where they add the most value.

Page 9: DataOps: The Collaborative Framework for Enterprise Data ... · DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number:

Copyright © 2017 Blue Hill Research Page 9

ANALYST INSIGHT

Short of pulling employees out of their functional roles and into product groups, the Center-of-Excellence (CoE) model “borrows” stakeholders from their functional teams and places them on a data-focused leadership council of sorts. Assignment is typically rotated.

The CoE model can have a positive indirect effect on adoption. Done right, it turns participants into sponsors, and that can foster program evangelism through the organization.

3. Getting in Front of Data: The Decentralized, Democratized, Experimental DataOps Culture DataOps builds on traditional digital transformation models. Following the DataOps roadmap, we’ve identified value opportunity and designed data process to maximize it. Now it’s time to tweak the organization to support the idealized data flow.

The “old” way of managing and delivering data will limit data-derived value. Centralized data access management is slow and unresponsive. (What analyst can wait weeks to get data answers from a queued query?) And “old” reporting models are inflexible and static: “Here’s a spreadsheet, make of it what you will, and enjoy the same one next month.”

Setting up a data-innovation enterprise for DataOps success demands a new way of work, with new responsibilities for data managers and for data consumers. The new way of work means DataOps enterprises must decentralize, then empower self-service data discovery and analytics throughout the organization. There are three steps to get to that new way of DataOps work:

1. Establish and evangelize data governance

2. Automate

3. Develop a culture of data exploration

Establish and Evangelize Data Governance

The “old” way of managing data work? Resource-intensive manual data-prep. Relying on manual, repetitive “data-munging” limits an enterprise’s ability to scale data operations (including data consumption) as data grows and data evolves.

Data governance is essential to the new way of DataOps work. In a best-practices DataOps environment, data management stakeholders have the responsibility to ensure governance and security principles are in place, and more importantly, that adherence to them is evangelized throughout the organization. Similarly, data consumers share

Spreadsheets Aren’t A Solution In his February 2016 report “Quantifying the Case for Enhanced Data Preparation,” Blue Hill Research analyst James Haight surveyed data analysts and found that more than three quarters of those surveyed relied on spreadsheets for analytics. Spreadsheets introduce data-immediacy risk—as soon as spreadsheet data is emailed, there’s no longer any way to preserve control over the accuracy or immediacy of that very un-dynamic data. As Haight put it, “Using spreadsheets to prepare data often involves moving data outside of governance controls put in place by IT and can present a challenge for data quality and consistency.”

Page 10: DataOps: The Collaborative Framework for Enterprise Data ... · DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number:

Copyright © 2017 Blue Hill Research Page 10

ANALYST INSIGHT

the responsibility to observe those governance and security principles. That mandate appears straightforward, but it’s not that simple: Data governance initiatives are often weakened by poor communication and disparate (sometimes selfish) objectives. Without clear governance and security guidelines, data management cedes control over data, impacting transparency, data integrity, and consumer trust.

Automate

The Phoenix Project co-authors Kim, Spafford, and Behr outlined four types of DevOps work—business projects, internal projects, operational change, and unplanned work—and preached a philosophy of maximizing time spent on value-adding activities (and reducing time spent on unplanned work).

That philosophical approach applies to DataOps. In a DataOps environment, LOB data consumers must be able to access the right information, interpret data, and then act upon it. Similarly, data managers must be able to extend the system to accommodate new data sources, establish new data flows, measure/monitor data-flow performance across the system, develop new data applications, govern the data, curate data for internal customers, and plan for continuous improvement. DataOps isn’t a robotic assembly line, but any resource time spent on activities that do not contribute to maximizing data-derived value is inefficient. It’s imperative to refine and prioritize the “right” types of work for roles at every stage of the data flow map value chain.

The steady state seems attractive. But in too many enterprises, administrative tasks like data curation, unification, and preparation are performed manually by data consumers (or their agents) in a tedious, linear, ungoverned, ad-hoc fashion. (An ad-hoc, experimental approach is desirable, but only when it is efficiently governed.) In his report, “Quantifying the Case for Enhanced Data Preparation,” Blue Hill Research’s James Haight found that most data analysts’ productivity suffers because of time spent on the onerous (and too often repeated) task of normalizing data. All that manual data-munging work takes away from analysts’ bandwidth to perform actual analysis, not to mention the extent to which it introduces new data-integrity and data-immediacy risks.

Data technologies—in particular, data-preparation, data-unification, and data-curation solutions—enable DataOps automation. They can help data stakeholders (both data managers and data consumers) maximize time spent on critical-path work and minimize work spent on administration and firefighting. (Even some data science functions can be automated—a valuable convenience when data scientists are hard to come by.)

Keys to democratized-data success:

• Identify, prioritize, and measure value-added work. (Are analysts analyzing? Or spending time inefficiently manipulating spreadsheets, or worse, awaiting indirectly-submitted query responses from someone else?)

• Deploy self-service technologies throughout the organization, making sure to empower stakeholders with the level of self-service appropriate to their work. (Yes, the data scientist power user should be able to spin up virtual infrastructure, but the marketing analyst may only need self-service data-prep and data-unification access.)

Page 11: DataOps: The Collaborative Framework for Enterprise Data ... · DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number:

Copyright © 2017 Blue Hill Research Page 11

ANALYST INSIGHT

• Enable logic preservation and best-practice sharing. The right data-prep exercise should not have to be repeated. When data-prep logic is established, it shouldn’t have to be re-established the next time similar analysis work is performed.

• Require human “eyeball-test” validation to ensure data integrity.

The efficient DataOps work model makes for a pretty picture: IT data managers focus on planning, monitoring, and improving the process. Analysts—instead of spending their time translating vague data requests into SQL—are business leaders on the front lines, doing actual analysis on readily-available dynamic data. And then they’re taking action to maximize value to the enterprise.

Develop a Culture of Data Exploration

Data democratization is powerful, but only when the enterprise can take advantage of it. Self-service technologies enable DataOps, but work best within a corporate culture that emphasizes experimentation as the path to insight (and value-delivering action).

The forward-thinking, data-driven enterprise is made up of inquisitive employees seeking new ways to make the most of enterprise data. Some ways to foster that:

• Assess skills—what do you have in-house and what are you missing? What can you hire, what can you outsource, and what can you automate? (A suggestion: Pursue hires who have taken statistics.)

• Evangelize BI as the enterprise path to value, both with end data consumers and leadership. Believe it or not, there are still enterprises where this isn’t accepted gospel—beware the C-level exec who eschews data-driven decision-making for instinctual action because “like, I’m really smart.”

• Train LOB stakeholders to adopt a data-discovery experimentation mindset (“What if … ?”) and a performance-monitoring outlook. (“I see this immediately and I can act on it immediately.”)

• Position IT management as the corporate data broker shepherding data through the data journey and serving as a service bureau for data consumption throughout the enterprise. Data management must evolve as a data-curation service for internal customers. (Learn more about “data broker” and “data barista” concepts. ) Set SLA-level delivery expectations for those internal customers, whether they are analysts engaging with data via self-service channels or developers directly accessing data via open APIs.

• Ensure data consumption can scale with data growth. Data democratization is great, but only if data consumption can accommodate it.

Recommendations for Technology Vendors: Embrace the Horizontal This report emphasizes a technology stack aligned with the DataOps data flow: the DataOps enterprise must identify, evaluate, and adopt technologies to support value delivery at each stage of the managed value chain. Specifically, data curation, unification, preparation, and analytics technologies are essential to enabling the DataOps workflow to maximize data-derived value in the enterprise.

Page 12: DataOps: The Collaborative Framework for Enterprise Data ... · DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number:

Copyright © 2017 Blue Hill Research Page 12

ANALYST INSIGHT

Technology vendors can take several steps to drive DataOps adoption.

First, position point solutions within a broader enterprise data value-chain context. Enterprise IT leaders can no longer afford to think in terms of siloed point solutions, and enterprise data technology vendors can’t either: A point solution promoted without context won’t resonate with an enterprise IT lead concerned with breaking down silos. If your technology serves only one part of the DataOps value chain (e.g., only analytics, only preparation, only unification, etc.), promote it within part of a broader enterprise solution. To compete against full-stack, full-data-flow “horizontal” competition, consider partnerships with other complementary technology point-solution providers, if only for co-op marketing.

Second, get horizontal (if you’re not there already). “Horizontalness” isn’t just about marketing messaging. Enterprise DataOps stakeholders seek solutions to cover the entire enterprise data value chain. If your technology offering covers only most of the data flow spectrum, look at M&A opportunities to fill in gaps, and to present a cohesive whole solution to enterprise customers.

Third, innovate to create data-flow measurement solutions. Many data-prep and data-unification technologies offer easy-to-use, drag-and-drop interfaces that map data integration and preparation. Some even offer two-way data lineage tracking, which is essential for data governance and auditing, and a requirement for effective DataOps deployment. But ensuring DataOps success requires detailed cycle-time measurement, and vendors who can couple a full-data-flow stack with cycle-time-based performance optimization will win big. Measuring performance across only part of the enterprise data flow map isn’t enough, but that’s where we are right now. There’s a great opportunity for vendors who recognize (and seize upon) the chance to deliver performance measurement spanning the entire data journey.

Finally, solve the looming data-consumption bottleneck. Thanks to agile cloud deployment options, inexpensive commodity hardware, and flexible cluster management, our ability to store data scales. (Those are some of many technical advances fostering IoT rollout.) Unfortunately, our ability to consume it doesn’t. (Read more about dashboard evolution. ) Some vendors seek to address the data-consumption challenge by integrating anomaly-detection technology into downstream BI and analytics processes, in effect strengthening BI analysis with operations-performance-monitoring-like capabilities.

Conclusion Traditionally, data innovation has started on one of the ends of the enterprise data value chain, and then worked its way inward towards the middle. BI technologies begat self-service data-prep and data-unification tools, granting more power to data consumers, enabling them to move up the value chain—even virtualizing infrastructure. On the other end of the spectrum, data-governance protocols and storage innovations like data lakes (with their ability to store dissimilar data in the same "reservoir") have advanced storage technology, but also reinforced centralized data-management models, sometimes even to the point of controlling query access. The competing interests create a philosophical and operational chasm between data management and data consumption: How can enterprise leaders deliver "all-you-can-eat", empowering, self-service data consumption while at the same time ensuring security and maintaining data-governance mandated compliance? The DataOps collaborative framework gives enterprises seeking to maximize data-derived value a way to bridge that chasm.

Page 13: DataOps: The Collaborative Framework for Enterprise Data ... · DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number:

Copyright © 2017 Blue Hill Research Page 13

ANALYST INSIGHT

DataOps defines a deconstructed, measurable, bidirectional, silo-spanning data-flow map of the journey from source to insight to action; establishes collaboration and goal alignment between data consumers and data managers; and builds upon a commitment to a corporate culture of experimentation and data service delivery.

There is an old television commercial for a shipping company in which it is suggested that somewhere in the company’s logistics chain is the coveted “golden package,” the most important package of all. But no employee knows which package is the golden one, so each is treated as if it is. That romantic customer service message is analogous to successful DataOps modeling: DataOps leaders must establish data flows to optimize delivery not just of all data, but of any potential data. Any piece of data—now, or in some differently-defined future—could be the “golden package” that delivers the most value to the enterprise.

The business case for DataOps starts with benefits and ends with value. In a DataOps enterprise, data consumers reap the rewards of flexible, seemingly-unfettered self-service empowerment, from analytics to data preparation to data unification. And in that same DataOps environment, data management enjoys greater control, with better data governance, improved transparency, and auditable accountability. More importantly, those benefits will ultimately translate into data-derived value for the DataOps enterprise.

Data without DataOps is just numbers. In the end, a DataOps deployment will succeed only to the extent which data consumers and data managers exploit the tools and technology solutions at their disposal. Too often, data management and data consumption have been at odds. DataOps brokers the peace (and for that matter, the data): When data management and data consumption workflows are aligned around idealized value-delivery-maximizing data flows, the enterprise sky is the limit.

Page 14: DataOps: The Collaborative Framework for Enterprise Data ... · DataOps: The Collaborative Framework for Enterprise Data-Flow Orchestration Published: January 2017 Report Number:

 

About the Author

Toph Whitmore Principal Analyst, Big Data and Analytics

Toph Whitmore is a Blue Hill Research principal analyst covering the Big

Data, analytics, marketing automation, and business operations

technology spaces. His research interests include technology adoption

criteria, data-driven decision-making in the enterprise, customer-journey

analytics, and enterprise data-integration models. Before joining Blue

Hill Research, Toph spent four years providing management consulting

services to Microsoft, delivering strategic project management leadership.

More recently, he served as a marketing executive with cloud

infrastructure and Big Data software technology firms. A former

journalist, Toph's writing has appeared in GigaOM, DevOps Angle, and

The Huffington Post, among other media. Toph resides in North

Vancouver, British Columbia, Canada, where he is active in the local tech

startup community as an angel investor and corporate advisor.

CONNECT WITH TOPH

@TophW47

tophwhitmore

Blue Hill Author Page

For further information or questions, please contact us:

Phone: +1 (617) 624-3600

Fax: +1 (617) 367-4210

Twitter: @BlueHillBoston

LinkedIn: linkedin.com/company/blue-hill-research

Contact Research: [email protected]

Blue Hill Research offers independent research and advisory services for the enterprise technology market. Our domain expertise helps end

users procure the right technologies to optimize business outcomes, technology vendors design product and marketing strategy to achieve

greater client value, and private investors to conduct due diligence and make better informed investments.

Unless otherwise noted, the contents of this publication are copyrighted by Blue Hill Research and may not be hosted, archived, transmitted,

or reproduced in any form or by any means without prior permission from Blue Hill Research.