how to save millions on legacy mainframe …...5 how to save millions on legacy mainframe operations...

How to Save Millions on Legacy Mainframe Operations

2


CONTENTSOverview � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 3

Legacy Mainframe System Challenges � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 4

Including Mainframes as Part of a Data Modernization Strategy � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 6

Introducing DataStax Enterprise (DSE): The Always-On, Distributed Database Designed for Hybrid Cloud � � � � � � � � � � � � � � � � � � � 7

The DataStax Solution Approach—Delivering Value Through a Highly Efficient and Scalable Operational Data Layer � � � � � � � � � 9

The Result � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 11

Conclusion � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 12

About DataStax � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 13

3


OVERVIEWWhile organizations continue to use mainframes as an integral part of their legacy IT infrastructure and business model, mainframe technology is struggling to meet new data needs� Organizations today ingest, analyze, and quickly act upon larger and larger volumes of data to extract the maximum business value� Mainframes were not designed for these new and modern kinds of workloads�

Increasingly, data is fueling new products and services and disrupting entire industries—transportation, financial services, entertainment, and hospitality� Organizations need a modern, agile solution that scales quickly and cost-effectively to handle such massive amounts of data and delivers the insights needed to compete in today’s data-driven, customer-centric economy�

Such a solution can replace or augment mainframe solutions with on-premises, hybrid cloud, or multi-cloud integrations far more quickly and at a lower price point than is possible with mainframes alone� These “new” solutions can grow as the data needs grow, enabling organizations to “pay as they go” for increased capacity or critical system changes and to implement those changes in real time�

What’s driving the need to modernize data environments are a variety of factors. New organizations that are “born in the cloud” have the luxury of starting with highly agile and scalable systems to meet data and analytics needs� This allows them to move more quickly to address changes in the market or adjust faster to meet new compliance requirements� In older organizations with heavily entrenched and rigid data solutions, these data infrastructures hamper their ability to quickly change and adapt�

Many traditional organizations have successfully modernized their data environments, including Walgreens, Capital One, Condé Nast, Macy’s, and others. In each case, the organization redesigned its data solution to be able to store and analyze rapidly increasing volumes and types of data�

Many of these organizations chose to keep their data environments on-premises and use commodity, x86-based solutions to meet their needs� Others chose hybrid cloud or full public-cloud solutions� Each of these deployment models has benefits and drawbacks that need to be evaluated based on their merits and potential value. The good news is there are solutions that can address a wide range of requirements for organizations that want to transform their data environments�

This document helps technology leaders understand the challenges of relying heavily on mainframes to meet their data needs and offers an approach to implement continuously available and highly scalable data and analytics solutions.

4


LEGACY MAINFRAME SYSTEM CHALLENGESMainframes have a long and illustrious history in global commerce� They have evolved to handle more transactions and store and analyze more and more data. At one point, most of the world’s transactions were processed on mainframes. Today, these older systems continue to serve an important purpose but face difficult challenges.

These challenges include:

1. CostMainframe pricing is extremely complex and arcane, and is based on “MIPS” (million instructions per second) consumption�

Once the organization exceeds its contracted MIPS, which is common during peak events, each MIP over the threshold gets significantly more expensive. Organizations that attempt to modernize existing mainframe hardware and software generally confront a high-cost financial step function—every increment of additional cost for compute, storage, and software licensing requires a significant new investment.

In the realm of big data, where the requirements can include extremely high volume and velocity data ingestion; indexing and storage of data; plus newer analytics and graph processing capabilities; the cost considerations on mainframes are staggering� As such, even a small reduction (15-20%) in cost can save organizations millions of dollars in mainframe operating expenses�

2. Inability to handle bursting loads and limited scalabilityIn terms of computing architecture, distributed systems with massive, dynamic scalability and near real-time responsiveness to bursts of load are becoming commonplace�

While this type of architecture is easily achieved with commodity hardware, cloud, or multi-cloud solutions, it is not practical with mainframe systems. IBM’s Parallel Sysplex is good at clustering small numbers of nodes (up to 32), but modern, global solutions can require hundreds to thousands of nodes spread across the globe�

This type of solution is simply not feasible with mainframe technology from both a cost and a technology perspective� Modern applications, like web and mobile apps, generate a massive amount of usage telemetry and click path data� The volume and velocity of this data can fluctuate wildly from one moment to the next. Mainframes must be sized for maximum throughput, creating wasted monies at times of minimal load� Unforeseen peak times like product launches or sudden changes from large marketing and sales campaigns can overwhelm the mainframe, while distributed systems can easily scale by adding inexpensive nodes when needed�

Additionally, mainframe platforms can falter when there are simultaneous requests for many different types of workloads at once� For example, the platform may struggle when it needs to handle analytics requests from many users while simultaneously ingesting and indexing massive amounts of data�

3. Staffing, skills shortageThere is currently a critical shortage of people that want to learn COBOL or Rexx, let alone understand CICS, IMS, VSAM, SMF, or other common legacy mainframe technologies. This makes it difficult to recruit and retain talent, especially considering the many hot technologies that are more appealing as a career choice�

5


4. Loss of corporate memory around key processesLegacy mainframe organizations are also losing highly trained and experienced people to retirement and general attrition. In most cases, these people have been with the organization for years and have accumulated a significant amount of critical, operational knowledge� They are often the people that are the most familiar with how to set up and run the mainframe� When they leave, valuable details about the system go with them� Such a loss can significantly hinder ongoing operations and prevent organizations from moving to more modern systems or integrating with new systems�

5. Slow time to marketMainframes have always had separate and specialized teams of people across the various mainframe technologies and application solutions� For example, the networking people know very little about the Db2 environment, or the CICS people know very little about Linux and the applications it runs on the mainframe� This creates a very monolithic operating environment that requires many meetings of various people to even schedule simple changes in the mainframe architecture�

6. Poor customer experienceMost mainframe applications suffer increased latency of accessing and are hence not suitable for demanding, real-time, customer-facing applications� Also, these applications often contend for limited mainframe resources with other users and with core backend business system processing� In addition, mainframe downtime also negatively impacts customer experience, especially if serving the new digital applications discussed earlier, creating a big risk of churn�

Because of the rigid, fragile, and high-cost step functions of employing any new mainframe-only solutions, there are excessive architectural reviews, operations impact studies, and financial impact reviews that add to the turbulence created when trying to implement new solutions on an expensive mainframe. All of this affects time to market of new capabilities�

Mainframe environments are generally expensive and not easily adaptable� This has real and consequential competitive implications that, historically, may not have been as large a factor because of the general pace of industry� But today, as competitors become nimbler by adopting newer solutions, mainframe organizations run the risk of becoming out-innovated by companies that accomplish business goals with newer technology�

6


INCLUDING MAINFRAMES AS PART OF A DATA MODERNIZATION STRATEGYMany organizations that want to move toward powerful data and analytics solutions find they cannot abandon their entrenched legacy systems� In a growing trend, mainframes continue to collect, store, and retrieve transactional data, while the overall architecture is augmented to include highly scalable and always available systems either on-premises, in the cloud, or both�

A data modernization solution that includes legacy mainframes should be able to:

1� Operate across a wide array of platforms and provide a high level of operational and data consistency between platforms� These platforms can be on-premises, single cloud providers, or multiple cloud providers�

2� Allow easy extension of the solution or migration of data to public cloud platforms, should the business need arise�

3� Automatically replicate data across all the platforms so there is high availability of data and no single point of failure without additional, costly, complex, add-on replication technology�

4� Operate at a high transactional level, plus have the data available for analytics in near real time�

5� Optimally, include tools for analytics; application development; connectors to access other systems; management and monitoring across multiple platforms; and security and encryption�

6� Deploy relatively easily and operate at a cost that is significantly below the cost of mainframe and relational database management system (RDBMS) solutions�

Until recently, it has been difficult for data solutions to have these characteristics because most were based on an RDBMS� As the volume, velocity, and variety of data has steadily increased, new types of data solutions have emerged to help organizations modernize mainframe systems� The most common category of these solutions is known as NoSQL databases.

NoSQL databases are non-relational databases that can ingest, index, store, replicate, and analyze a wide range of data types quickly—avoiding the rigidity of RDBMS-based solutions by replacing organized storage with a more flexible model� Because they can handle new types of data as well as new capabilities such as AI and machine learning, NoSQL databases are at the forefront of emerging use cases such as mobile apps, IoT, and ecommerce.

Implementing a NoSQL solution is about more than just the data. The data is just the beginning. It must be migrated, ingested, replicated, secured, and analyzed (possibly with multiple BI tools), and also integrated with and made available for many types of applications� To get there, organizations need a solution provider that delivers far more than just the NoSQL data store.

7


INTRODUCING DATASTAX ENTERPRISE (DSE): THE ALWAYS-ON, DISTRIBUTED DATABASE DESIGNED FOR HYBRID CLOUDIn the field of NoSQL solution providers, DataStax stands as a leader, especially when it comes to replacing or offloading mainframe systems. DataStax offers DataStax Enterprise (DSE), an always-on, distributed database built on Apache Cassandra™ and designed for hybrid cloud�

DSE is the foundation for real-time consumer and enterprise applications at massive scale and is the industry’s only active everywhere database platform� DSE is a distributed database where “nodes” are instances of DSE where data is stored� Individual nodes are grouped together in logical “data centers” and the database can span multiple data centers� Data is replicated transparently and seamlessly across nodes and data centers using native embedded replication technology, thereby obviating the need for additional add-on, costly, complicated replication solutions�

DSE also integrates with leading technologies such as Spark, Kafka, Kubernetes, Docker, and others to provide a comprehensive solution to customers� Also, DataStax is closely intertwined with the Cassandra ecosystem via significant contributions to various Apache and open source projects, and through free education at DataStax Academy.

DSE is the only database that provides consistent data across on-premises, hybrid cloud, and multi-cloud implementations� The ability to replicate, protect, and analyze data across a wide variety of platforms provides a significant value for organizations that need to operate at large scale and across geographies.

When augmenting or migrating mainframe data, the following DataStax capabilities are critical:

1. Performance and cost-efficiency at scaleYou achieve near linear read, write, and query performance simply by adding more nodes to the DSE cluster and without changing the application. Also, when using public cloud providers’ virtual machines, you can easily modify existing virtual machines with more compute and memory, or upgrade storage types and networking throughput to improve performance and do all of this “in-place.” This enables a much smaller financial jump to add scalability compared to on-premises solutions and certainly mainframe environments�

2. Designed to handle failure and provide continuous availabilityBecause data is replicated across clusters and multiple cloud providers operating on and backing up the same data, DSE allows for enterprise application designs without a single point of failure�

3. Data portability, distribution, and replication across clouds and on-premisesData can be moved or consolidated to any platform at any time� For example, if a cloud provider changes its pricing policy or performance suffers for some reason, or if perhaps they even enter your industry and become a competitor, new nodes can be set up on a new cloud platform, data can be automatically replicated, and the nodes you don’t need can be retired�

4. Minimal disruption to business through transitionMost mainframe organizations are looking at different ways to extend their mainframe capability to service modern digital applications. This is where DataStax can be of extreme benefit. By being able to abstract the legacy data into an operational data layer, DataStax can provide these organizations with a more reliable, real-time, scalable, and

8


multi-cloud deployable solution� DataStax can become an extension of the mainframe and thereby protect existing investments while enabling completely new and richer ways to derive value from its implementation�

5. Multi-modelDSE is a multi-model database that provides analytics and data visualization, search, graph processing, geospatial, in-memory performance, and global replication to allow architects to build a wide variety of real-time operational and analytical applications on a single data set, reliably and securely� There is no need to install multiple databases to meet the needs of different applications, nor is there a need to move data into expensive data warehouses or batch clusters to extract insight and value from data�

6. No platform lock-inBecause of its unique masterless architecture design, DSE replicates data seamlessly across clusters that can be run on-premises or in a cloud architecture of choice. This includes hybrid cloud configurations to multiple cloud involvement� Further, organizations can start the journey with a complete on-premises solution and easily extend or migrate to cloud platforms with relative ease while using the same tools for management, analytics, and application development�

7. Proven services methodologyDataStax has a proven services blueprint that enables a successful migration of mainframe data structures to NoSQL databases� Our experience suggests that the decision to move key data and processing to a database outside the mainframe creates challenges. In such a scenario, the DataStax services team works closely with the customer’s project team to prioritize and coordinate the various work streams necessary to achieve each incremental success� Bringing the effort to production requires an iterative process that includes thorough testing. For example, how the data gets partitioned on the cluster often reveals unforeseeable hotspots that will need to be evaluated and addressed by one of several known workarounds�

These are just some of the high-value capabilities brought to the table when using DataStax solutions to augment mainframe environments�

9


THE DATASTAX SOLUTION APPROACH—DELIVERING VALUE THROUGH A HIGHLY EFFICIENT AND SCALABLE OPERATIONAL DATA LAYEREvolutionary migration to DSE allows enterprises to:

) Give their customers the responsive, always-on, self-service experience they expect�

) Reduce mainframe MIPS expenses by lowering peak mainframe capacity requirements and offloading traffic better suited to scalable distributed systems�

) Reduce mainframe acceleration tool expenses (e�g�, memory grids/caches)�

) Confidently experience spikes and peak season events without compromising performance or availability�

) Provide a data platform to their development teams that allows them to innovate around core business data, today, free from the constraints of legacy system architectures�

Enterprises gain an advantage from DataStax’s experience of successful delivery and continuing support of MIPS to cloud migrated workloads� The proven DataStax methodology has helped many customers, including a Fortune 100 large wireless customer in the United States, realize value from the migration quickly and extend the practicality of existing systems into the future�

A typical mainframe migration environment usually starts with:

) An RDBMS-based application (relational data constructed into objects)

) > 1,000 tables in a normalized schema

) > 200 APIs implemented as stored procedures using a variety of languages and methodologies

) Several terabytes of data (before replication)

The relational data structure that forms the basis of mainframe application design requires joins to construct cohesive objects that may be used by the application. These joins become the architectural Achilles’ heel that prevent data distribution and scaling�

DSE is purposely architected differently from traditional relational databases. It does not support JOIN statements because to do so would subject DSE to the same design limitations that cripple the scale of relational databases� So, although it is possible to build databases and tables that look similar to those in an RDBMS using SQL-like Cassandra Query Language (CQL), DataStax doesn’t recommend it. We believe we have a better approach: migrating data access methods to microservices backed by DataStax as the always available distributed data layer�

DSE provides the tools and the integrative structures needed for an enterprise to enable replication of commonly accessed mainframe data into economical and highly available data structures, against which queries are redirected from consuming applications�

10


Here is a reference architecture for a solution similar to one DataStax implemented for a customer:

Our experience suggests that the decision to move key data and processing to a database outside of the mainframe creates challenges. In this case, the DataStax project team worked closely with the customer’s project team to prioritize and coordinate the various work streams necessary to achieve each incremental success�

In the case above, DataStax followed these steps to arrive at the desired outcome:

1. Understanding the workloadThe more complicated the business process the harder it is going to be to migrate, so it is important to understand which parts of the customer’s system would benefit most from the capabilities of DSE and prioritize those for migration. DSE works great with CRUD (Create, Read, Update, Delete) operations and some common capabilities were mixed in, including:

) Authentication

) Authorization

) Logging

) Auditing

) Transformations

) Aggregations (client-side joins)

) Throttling

) and possibly some level of concurrency management�

2. Capacity planningThe heart of this solution was hardware muscle. However, for DSE, success comes from its efficient usage of high CPU counts and large amounts of RAM. Unlike storage limitations of a typical server, DSE doesn’t impose an upper bound on RAM and customers have benefitted from as much as 1TB - 3TB RAM per server and more. The broad goal here is for each robustly equipped node to host a relatively small percentage of the total data and for 100% of the active data to be in RAM�

11


3. Data modelingThis approach required moving all or most of the existing schema as is� The main reason for retaining the existing schema in this first evolutionary step was that it greatly simplified bulk import and subsequent data synchronization with the mainframe system that will continue to be the system of record for some period of time� A secondary reason was that the current schema, its internal relationships and dependencies, were usually familiar to the customer’s application teams who would be writing APIs that would access the same data in DSE. This familiarity accelerated the overall effort. The final benefit to this approach was that actual usage patterns and workloads could be observed running in DSE to make empirical decisions about the future improvements to the data model�

4. Data synchronizationDSE was imagined as a front-end cache to data hosted in the mainframe� Writes to the main system were concurrently fed to DSE through the Change Data Capture (CDC) functionality� This delivered the following advantages:

) Real-time data – users were immediately consuming the freshest version of the data, rather than waiting for updates to propagate at a later stage�

) Reduced application complexity – read and write operations no longer needed to be segregated between different systems.

) Enhanced application agility�

5. Creative API authoringThe hottest APIs were hosted on independent groups of application servers which were equipped, scaled, and coded to meet the unique performance demands of those APIs� This included caching / expiring lookup tables and other frequently referenced data, facilitating client-side joins, dual writes to multiple sources, and write buffering. Restful interfaces were developed for these access methods, allowing for the new interfaces to be used as microservices�

THE RESULTBy working with DSE and leveraging a proven support and services model, this enterprise customer was able to achieve considerable success with its mainframe transition efforts. Some key improvements included:

) 1�7 million transactions per second peak offloaded from the mainframe and MIPS billing

) Transaction latencies of one millisecond max at peak

) Zero downtime

) Over $25M of annual savings over the MIPS-based platform by making a $4M annual investment in DataStax technology

) A new ability to bring products and services to market faster

12


CONCLUSIONOrganizations are increasingly seeking to engage customers across new digital channels, which in turn drives significant growth in both the number of consumers and the frequency with which mainframe data is accessed�

Trying to serve these new digital initiatives from an existing mainframe platform can present major challenges that drastically reduce the pace of application delivery while escalating cost and risks� Organizations know they need a scalable and flexible solution now. Choosing the wrong solution can negatively impact the business significantly.

For more information on how DataStax can help with your mainframe migration projects, please contact your DataStax account team or visit www�datastax�com�

http://www.datastax.com

13


ABOUT DATASTAXDataStax delivers the always-on, active everywhere distributed hybrid cloud database built on Apache Cassandra™� The foundation for personalized, real-time applications at scale, DataStax Enterprise makes it easy for enterprises to exploit hybrid and multi-cloud environments via a seamless data layer that eliminates the issues that typically come with deploying applications across multiple on-premises data centers and/or multiple public clouds�

Our product also gives businesses full data visibility, portability, and control, allowing them to retain strategic ownership of their most valuable asset in a hybrid/multi cloud world. We help many of the world’s leading brands across industries transform their businesses through an enterprise data layer that eliminates data silos and cloud vendor lock-in while powering modern, mission-critical applications� For more information, visit www�DataStax�com and follow us on Twitter @DataStax�

© 2019 DataStax, All Rights Reserved� DataStax, Titan, and TitanDB are registered trademarks of DataStax, Inc� and its subsidiaries in the United States and/or other countries�

Apache, Apache Cassandra, Cassandra, Apache Tomcat, Tomcat, Apache Lucene, Lucene, Apache Solr, Apache Hadoop, Hadoop, Apache Spark, Spark, Apache TinkerPop, TinkerPop, Apache Kafka, and Kafka are either registered trademarks or trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States, and/or other countries�

Last Rev: APR2019

http://www.datastax.com

https://twitter.com/DataStax

14



how to save millions on legacy mainframe …...5 how to save millions on legacy mainframe operations...

Documents