the enterprise integration cloud - cdm media · the enterprise integration cloud: avid . inthicum...

The Enterprise Integration Cloud:David S. LinthicumMarch 2017

This report is underwritten by SnapLogic.

What to look for in a new era of integration

Linthicum Research 2The Enterprise Integration Cloud

EXECUTIVE SUMMARY

Enterprise applications and the data landscape are undergoing dramatic change. The growth of the cloud, the dramatic rise in the value of machine-generated data, more empowered business users, and the increas-ing pace of business are all placing pressure on data and application integration.

New data from the Synergy Research Group shows that across six key cloud services and infrastructure market segments, operator and vendor revenues for the four quarters ending September 2016 reached $148 billion.

This means that cloud has grown by 25 percent on an annualized basis. IaaS and PaaS services had the highest growth rate at 53 percent, followed by hosted private cloud infrastructure services at 35 percent, and enter-prise SaaS at 34 percent. What was most notable is that 2016 was a year in which spending on cloud services overtook spending on cloud infrastructure hardware and software. Aggregate cloud service markets are now growing three times more quickly than cloud infrastruc-ture hardware and software (see Figure 1).

Linthicum Research

Figure 1: Over the period Q4 2015 to Q3 2016, total spending on hardware and software to build cloud infrastructure exceeded $65 billion, with spending on private clouds accounting for over half of the total, but spending on public cloud grew much more rapidly.

The Enterprise Integration Cloud


There are many solutions used to approach these issues. One is emerging as the “go to” approach and technology that’s changing the game. An Enterprise Integration Cloud (EIC) provides a single plat-form for organizations in transition, and covers a variety of features, everything from incorporating a growing range of cloud assets, to meeting the needs of empowered users, while ensuring strong governance, scaling, and security — without cre-ating a fragmented array of integrations, tools, and hand-coding.

Emerging patterns include:

• Rapid shift from on-premises data to public cloud-based data

• Rapid expansion of data usage in support of: Predictive analytics, machine learning, IoT, etc.

• Rapid increase in both data volumes and data velocity required by enterprises to obtain data as needed, when needed, and in good quality

• Lacking application integration infrastructure, including the ability to support real-time and near-time data delivery

• Renewed focus on data security and data governance

Conclusions made in this paper include:

• IT is changing business, and enterprises must understand how to adapt or die.

• Data is the game changer for business. Failure to understand current and historical data is a business killer, while complete understanding of data (in context of the business) becomes a critical advantage.

• The role of application integration is becoming strategic to all that matters, in terms of distri-bution of data within and between enterprises. Data integration is not enough, but the ability to sync application and data silos becomes a neces-sary set of services that need to be native within public clouds, as well as within the enterprise.

• The value of the application integration invest-ment will generate 30 times the initial invest-ment in terms of cost savings, and 100 times initial investment in terms of strategic advantag-es, including agility, working with near perfect information, and time-to-market.

• An enterprise integration cloud is the best path to achieve these objectives, and this technology is available now.


To explain EIC a bit better let’s use a fictitious com-pany case study: Vital Supplies, Inc. Let’s say these guys have a production business where they build, sell, and service medical equipment for 2,000 plus hospitals around the world. Vital Supplies is over 20 years old with a mostly traditional system infra-structure in place, including:

• mainframe-based computers

• EMC storage

• older Unix servers

• Windows Servers running MySQL

The Windows Servers are for tactical applications that exist in a local data center with about a third of their data residing in an Oracle database that they leverage for data marts with high license costs and maintenance costs. Finally, they leverage on-prem-ises integration technology, in largely ad-hoc and efficient ways, such as the use of batch processing and traditional data mapping technology.

As a matter of strategy, Vital Supplies has the IT objectives below to help the company move to over $20 billion dollars in sales within the next two years. This list includes the following objectives set by the board of directors:

1. Move 70 percent of the applications and data to a public cloud-based provider, such as Amazon Web Services, Microsoft, or Google. This should reduce operational costs by 40 percent through

the reduction in data center space required, and the need to purchase more hardware and software.

2. Make better use of data, including the abili-ty to predict sales demand, which will allow Vital Supplies to produce only the supplies and devices that they can sell, reducing inventory expenses and out-of-date products that they must discount.

3. Become 70 percent more agile. For instance, the ability to launch another device product line in 12 week versus 6 months, as it takes today.

4. Combine 80 percent of the business processes with real-time data, allowing decisions to be made using automation, with perfect up-to-the-second data. For example, the ability to auto-matically communicate with the department that manufactures supplies and devices, if they are depleted to a certain level that’s set by the production team.

THE KEY FACETS OF THE ENTERPRISE INTEGRATION CLOUD


Using Vital Supplies as a point of reference, let’s look at the value of EIC, and how you can leverage these concepts within your own enterprise.

Enable Cloudification and the Hybrid Enterprise

The cloudification of the enterprise continues, with organizations taking a cloud-first approach, yet still struggling with a fragmented hybrid landscape. Integration tools are often on-premises or cloud-geared, but modern integration tools must effective-ly unify both old and new worlds.

This is certainly the case with Vital Supplies, where tools were largely installed around the use of Oracle. Moving forward to the cloud, Vital Supplies is looking to dump Oracle, move to AWS Redshift as their primary data-warehousing database, and use AWS Aurora (a MySQL-compatible relational database engine) as their Windows database analog. In doing so, they will also open their integration solution to change as well. This includes leveraging integration technology on-demand out of the cloud, which will provide a better path to integration, and support more integration patterns. Moreover, Vital Supplies is considering the use of a non-relational database as well for their net-new applications go-ing forward, including object-based databases and purpose-built databases, such as in-memory.

Key points:

• Move to more heterogeneity of data, such as Vital Supplies’ movement to non-relational databases, as well as cloud-based database ser-vices.

• Accelerate expansion of the use of public cloud platforms. While Vital Supplies is a large busi-ness, large and small businesses are quickly moving to public cloud-based platforms (see Figure 1).

• Be more provocative, provide more self-ser-vices, and be disruptive. Within the market-ing department at Vital Supplies, they can’t wait for IT to provide the data that they need. On-demand and self-service use of integration is the best path for organizations that need to empower their key players to make a difference for their business.

• Ubiquity of integration. Integration should be everywhere, and it should be systemic.


Support an Increasing Volume and Variety of Data Sources

With a dramatic growth in data volume from users, machine-generated data, cloud data sources, and existing on-premises investments, traditional inte-gration tools are struggling to keep pace. It’s com-pounded by business users themselves deploying business applications, which increases the fragmen-tation of data across multiple clouds and on-prem-ises silos. Hand-coded integrations just simply can’t keep pace — it’s about connecting, not coding.

In the case of Vital Supplies, they had to leverage their integration technology as if it were a develop-ment platform, and this limited their ability to both leverage data and allow applications to exchange data to support business processes. That architec-ture simply could not keep up with the demand for integration, and perhaps cost Vital Supplies billions in lost opportunities over the last 15 years in busi-ness. In other words, the silos were built and could not share data easily.

Key points:

• Exposition of diverse and purpose-built appli-cations. As with Vital Supplies which had to build 80 percent of their business applications, such as inventory management, and their data marts, applications these days often don’t have standard interfaces, but instead use complex

interfaces that need to be abstracted from the integration technology users.

• Exposition of data fragmentation and lack of data standards. As data becomes more complex, distributed, and heterogeneous, we’re quick to point out the lack of standards that exist. While once we could assume mostly relational database technology, these days that’s no longer the case.

• Use of purpose-built database technology (e.g., in-memory databases) means that the types of databases we’ll find in the enterprise are also getting more complex. In the past, Vital Supplies avoided dealing with new types of databases, even though they better met its requirements. This can no longer be the direction if Vital Sup-plies wants to reach $20 billion dollars in sales.

One Integration Platform: Apps, Data, Things

Machine-generated data is massively outpacing user-generated data. The growth of point cloud tools is creating the need for application integration, and data analytics is a constantly growing need. Yet integrating machine-generated data and appli-cations is still under the preserve of scripting and IT-intensive tools. And integrating data for analyt-ics often require an entirely separate toolset.


Key points:

• Place complexity into a single domain, if possi-ble. This is typically the application integration technology, but the technology must have the capabilities of abstraction, the ability to deal with heterogeneous systems, as well as exter-nalize only what’s important to the application integration user. In the case of Vital Supplies, they needed to abstract both the IT and non-IT integration users from having to deal with the details behind the APIs (application program-ming interfaces) and other native database interfaces. Keep in mind that we’re dealing with APIs as to their functions. They are interfaces that applications can leverage to extract data or behavior. This means that the application inte-gration solution must be able to remove the user and the target system from having to deal with the complexities of the APIs. They are leveraged at the point of interaction.

• Abstract integration flows and data manipu-lation. Again, integration is not about forcing the end-users to deal with the dirty little details behind the processes and database interfaces. This needs to be an easy and intuitive process that can be carried out by anyone.

• Create de facto standards within enterprises, and consolidate legacy integration solutions. A single integration platform, such as the EIC, means that we can stop leveraging on-premises integration technology that’s neither pervasive nor helpful.

A Single Platform: IT and the Em-powered Business user

Increasingly, organizations are shifting from Mode 1 (IT-driven development, which means that all things must move through IT and are develop-ment-oriented) to Mode 2 (business user-led, which means the application integration task is placed with the end or business user who can directly benefit from the technology), from applications to analytics. And now it’s the turn of application inte-gration. Yet, organizations risk multiple integration tools, one set geared for IT, another for business users, which compound integration issues. It’s time for a single platform for both.

Key points:

• Platform consolidation is a requirement. In the case of Vital Supplies, they had highly technical integration solutions that could not be driven by non-experts who really needed access to that technology to do their jobs. Now we’re looking at one-stop shopping for integration technology that solves integration for both business users and expert users using the same tools.

• Merging of ease-of-use and highly capable tools means that we can provide true power to any-one who needs to leverage the technology, for any reason needed to support the business.


• Address the rise of the citizen application integration role. Non-IT departments at Vital Supplies can integrate the data they need to in-tegrate, when and where they need it, no matter what skill set they have. This frees up the busi-ness user to leverage the data in ways that can provide the most value to the business.

No-Code Design

With the increasing need for integrations, and with business users driving integrations, productivity and maintenance is key. Modern integration tools must provide visual development of workflows and integration to minimize training, and simplify ongoing maintenance. We need to get over the “old school” notion of coding any and all integration flows between the source and target systems, and get over it quickly, if we want to achieve any sort of business agility. In the case of Vital Supplies, they just avoided the integration altogether, and thus avoided opportunities, such as the ability to find and win lucrative government contracts that would have provided considerable improvements to the business. This can’t be allowed to be a limitation.

Key points:

• No-code means configuration-driven. Business users are not developers, and thus they are look-ing for solutions that operate using an intuitive

approach that matches their skill level.

• Leverage abstraction as the mechanism to re-move complexity, and the more abstraction you have, the more effective the integration users will be with the tool. The more effective, the more agility.

• Focus on productivity. Continuous improve-ment needs to be in the minds of the EIC user, who must understand that the integration platform needs to be leveraged so things consis-tently improve.

Maximize Reuse

With an increasing volume of integrations, and a move from a few users to perhaps hundreds of users performing integrations in large organizations, reuse is key for both productivity and consistency. Desktop tools, and hand-coded integrations all limit reuse. Centralized metadata, workflows, and inte-gration patterns all are critical to improve produc-tivity through reuse, and to ensure consistency.

Within the Vital Supplies’ integration technology, there was no reuse in the past. Most of the integra-tion flows were one-off ad-hoc solutions that Vital Supplies created to solve a very tactical problem. After years of doing this, the integration flows appeared to be a dysfunctional highway system that nobody could navigate. In other words, Vital Sup-plies dug themselves into a hole of complexity, and


did not reuse anything. This will be unacceptable going forward, considering the new objectives set by the company’s leadership.

Key points:

• Focus on building for reuse, not on building once. This is a change from the ad-hoc/fix-a-problem-quickly culture that we’re coming from. However, the ability to reuse existing work is on the critical path for agile integration.

• Continuous improvement of work products. We must constantly improve on the integration ar-tifacts. All flows that leverage those artifacts will benefit from those improvements as well.

• Well-understood interdependencies. When companies such as Vital Supplies understand how things are being reused within their inte-gration infrastructure, they understand what changes and what’s affected by the changes.

Provide Security, Governance, and Data Management

While empowered integration is an important need within the enterprise, if it’s unmanaged, it can easily jeopardize data quality, lead to duplicate silos of data, or risk security through ungoverned data. IT also requires an effective way to promote user-led integrations to IT-managed ones, or enable users to

integrate data in a managed environment. Enter-prises require a single platform that provides flex-ibility for users, while also providing management for IT.

Security needs to be systemic, and in the case of Vital Supplies, this could not be truer. Thus, when considering integration security, there is no line that can be drawn between the data and the integration platform, and as a rule, identities and encryption must be managed across all platforms, including integration.

Key points:

• Security is now systemic, and touches every part of the architecture.

• Governance is now systemic, and touches every part of the architecture.

• Focus on identity and roles. This needs to be the battle cry. It’s about identifying all parts, includ-ing users, devices, data, platforms, etc., and then setting rules for access.

• Integration is a common denominator of security and really should be the way that most security systems interoperate.


Enable Collaborative Development

With data such as customer information being spread across multiple systems, from marketing to sales to service, organizations must enable business users to collaborate around analytics initiatives. Separate desktop integration tools stand in the way of collaboration. Web-based design of workflows and centralized metadata enable teams to work more effectively together.

In the case of Vital Supplies, there was little coordi-nation between employees, even when working to-ward the same objective. Data becomes a common need, and the way in which we can share and work to common goals as groups within the business.

Key points:

• The rise of DevOps and cloud leads to more automation and collaboration. This will be a need at Vital Supplies and other businesses that are moving in the same direction. The only way to get value from these notions is to collaborate effectively using the right technology, such as Enterprise Integration Cloud.

• Focus on reuse, and continuous improvement of base layer components. Build things once and use them many times. This is a divergence from the way that organizations traditionally approached integration, or other technologies.

Integrate with Enterprise IaaS Strategies

As organizations build their cloud infrastructure strategies, whether on AWS or Azure, benefits from elasticity and economies of scale using those plat-forms won’t materialize until integration tools are designed to maximize benefits and performance, improve cost benefits, and reduce IT overhead. Vital Supplies is all-in for the public cloud. By using IaaS platforms, along with integration platforms, they should be able to reap these benefits.

Key points:

• Synergy with IaaS and on-premises integration. There is no way to move to the cloud unless you have an integration platform in place, prefer-ably in the cloud and on demand. You need to exchange information on-premises to cloud and cloud to cloud. All of the attributes that we’ve mentioned apply to the integration technology.

• Focus on the seamless use of data. Sharing data should never be a hindrance. The ability to leverage data, when and where you need it, should be as easy as loading web pages.

• Focus on performance. Data needs to arrive at the “speed of need.” In the case of Vital Supplies, data needs to come from the supply at a rate of 100 KB-size records per second.


Ensure Scale Across Multiple Dimensions

When integrating applications, data, and things, there are many dimensions of scale, from the num-ber of messages, to API calls, to the sheer flow of transactions. In many cases, integration tools are only optimized for one of these elements. However, shifting to a single platform requires an integration platform that can manage all three.

Key points:

• New integration requirements that ask for high-speed integration patterns are now the norm. Vital Supplies receives data from some of their medical devices to deal with IoT data that clinicians require. If the data can’t be culled and made sense of on demand, again, the value of the devices won’t be there.

• Integration patterns and technology must cover all dimensions, including applications such as SAP and custom applications, and data such as that contained within cloud-based databases, and devices that produce data as a matter of function. We must deal with it all, not just spe-cific domains. Thus, there can be nothing now, nor in the future, that can’t be integrated to the requirements of the business.

Analytics and Predictive Intelligence

Innovations in analytics and predictive intelligence aren’t just for management dashboards. They can enable smarter integration strategies, too. Analytics on integration activity can provide insights into the flow of data across the organizations, the endpoints, and lines of business. And with increases in user-led integration, predictive intelligence can be used to suggest integration patterns based upon usage across the organization.

Vital Supplies’ core concern is the need to leverage data in a much more valuable way. What could be more valuable than predicting the future of things? This could include future demand for products, future production productivity, the use of outside influencers, such as the economy, and its effects on the business.

Key points:

• Capitalize on the rise of data-in-flight analysis, or the ability to see the future using current and historical data, and to do so instantly. This provide a significant and obvious business advantage.

• Incorporate automated binding into business processes and workflows. We can automate business processing based on predictive data.


Keep in mind that when considering the previous concepts, we need to consider each equally. How-ever, for few enterprises some of these concepts will be more important than others. For instance, when Vital Supplies considered these concepts, it assigned a relative importance as you can see in Table 1. Or, figure 2 shows what it looks like when placed in a spider graph.

CONCLUSION

Disruption Vectors Relative importance to Vital Supplies

Enable Cloudification and the Hybrid Enterprise 15%Support an Increasing Volume and Variety of Data Sources 5%One Integration Platform: Apps, Data, Things 10%A Single Platform: IT and the Empowered Business user 15%No-Code Design 7%Maximize Reuse 15%Provide Security, Governance and Data Management 10%Enable Collaborative Development 5%Integrate with Enterprise IaaS Strategies 7%Ensure Scale Across Multiple Dimensions 4%Analytics and Predictive Intelligence 7%Sum 100%

EnableCloudifica/onandtheHybridEnterprise

SupportanIncreasingVolumeandVarietyofDataSources

OneIntegra/onPla@orm:Apps,Data,Things

ASinglePla@orm:ITandtheEmpoweredBusinessuserNo-CodeDesign

MaximizeReuse

ProvideSecurity,GovernanceandDataManagement

Table 1: Relative Importance of concepts (Disruption Vectors) to Vital Supplies.

Figure 2: Looking at the relative importance of the concepts or Disruption Vectors.






Then, using these concepts as the core criteria for application integration tool selection, provide rank-ings of each tool and then consider those ranking around the weighting of the importance of the con-cepts, or disruption vectors. Figure 3 looks at what a ranking may look like for Vital Supplies, given the criteria on the previous page.

Figure 3: Analyzing a application integration provider given the disruption vectors, and the weighting of each vector.


There are a few conclusions that can be reached using EIC, including:

Enterprises that don’t take advantage of this tech-nology now are likely to go away. Information is a game changer. If leveraged properly, the business will be strong and growing. If not, the business is likely to fail.

The use of cloud is now the norm. As we move to cloud and make the needed transformation, chang-ing our approach to application integration is some-thing that can’t be avoided. Consider the aspects of EIC as presented in this paper as a foundation to make the move, and to pick the right application integration technology solution.

Those who don’t follow the concepts outlined in this paper are likely to miss key opportunities at best, and go out of business at worst. Neither should be acceptable to you, so it’s time to get busy.

As we move to cloud and make the needed transformation, changing our approach to application integration is something that can’t be avoided.

“


About the Author

About SnapLogic

David S. Linthicum is an internationally recognized industry expert and thought leader in the world of cloud computing, and the author or co-au-thor of 15 books on computing, including the best-selling Enterprise Application Integration, and his latest book, Cloud Computing and SOA Convergence. He is a blogger for InfoWorld, Intelligent Enterprise, eBizq.net, and Forbes, and he conducts his own podcast, the Cloud Computing Podcast. His industry experience includes tenure as the CTO and CEO of several successful software companies, and upper-level management po-sitions in Fortune 100 companies. In addition, Linthicum was an associ-ate professor of computer science for eight years and continues to lecture at major technical colleges and universities.

SnapLogic is the global leader in self-service integration. The company’s Enterprise Integration Cloud makes it fast and easy to connect applica-tions, data, and things. Hundreds of customers across the Global 2000 — including Adobe, AstraZeneca, Box, Capital One, GameStop, Verizon, and Wendy’s — rely on SnapLogic to automate business processes, accel-erate analytics and drive digital transformation. SnapLogic was founded by data industry veteran Gaurav Dhillon and is backed by blue-chip in-vestors including Andreessen Horowitz, Capital One, Ignition Partners, Microsoft, Triangle Peak Partners, and Vitruvian Partners. Learn more at www.snaplogic.com.

the enterprise integration cloud - cdm media · the enterprise integration cloud: avid . inthicum...

Documents