Leveraging continuousintegration, delivery, and deployment in StreamAnalytix

SOLUTION BRIEF

Build, deploy, and deliver at high velocity with StreamAnalytix

Velocity is one of the key differentiators for enterprises today. With a focus on faster time-to-market, businesses require systems that can develop, test, and deliver applications faster, and have the flexibility to accommodate changes, which is where the agile practices of CI/CD come into the picture.

Continuous Integration and Delivery (CI/CD) is a set of automated SDLC practices and methods that enable frequent and error-free releases of change in code or data, with extensive visibility and traceability. The benefits of this approach include reduced time, risk, and cost of software delivery.

Continuous integration and delivery of ETL workloadsMany enterprises are adopting an automated approach to accelerate and simplify the entire process of Extract, Transform, and Load (ETL). While designing these processes, it is crucial to ensure sanity before operationalizing in production environments.While the quality engineers aim to achieve shorter cycle times and higher service levels for each delivery, developers and QA teams work on automating the manual processes whenever possible.

ETL tools help in expediting the process by providing an easy-to-use visual interface with ease of connectivity to a wide variety of sources, ready-to-use transformation logic, and choice of multiple stores. The process typically involves the creation of automated pipelines for ETL processes and test suites that orchestrate the build and test of pipelines across staged environments, leading to production deployment.

Background

2

Fig 1: Typical Steps to CI/CD

Code Build Test Integrate Release Deploy

3

Introducing StreamAnalytixStreamAnalytix is a self-service ETL and analytics tool. The platform lets you easily create batch and streaming ETL pipelines using drag-and-drop operators on a visual IDE. StreamAnalytix has a wide array of built-in operators for data sources, transformations, machine learning, and data sinks.

Continuous integration and delivery in StreamAnalytixStreamAnalytix comprises of various features to support CI and CD. The diagram below details how to create pipelines and manage them using StreamAnalytix.

Fig 2: StreamAnalytix Interface

You can build production-grade continuous applications, which makes it easier for customers to manage out-of-sync data better, maintain greater consistency within their data streams, and join streams with static data sources more efficiently.

4

Dev 1 Dev 2

StreamAnalytixEnterpriseRepository

Version Control

ETL & Analytics Pipeline

Multi-user and multi-tenant

environment

Version control ofStreamAnalytix

pipelines

Multiple test cases foreach pipeline

Deployment to target environment

BUILD VERSION TEST PIPELINE PROMOTION

CI/CD MANAGEMENT

External Repository

Sourceenvironment

Target environment

DeployActions

Test CaseExecution

PipelineUpdate

Fig 3: CI and CD in StreamAnalytix

Step 1: StreamAnalytix ETL and analytics pipelines

Pipelines are the core deliverables in any StreamAnalytix production instance. They are the blocks delivering value by processing data as per the project requirement. It is necessary to evaluate any change to these blocks before they can be made available in production.

In a complex enterprise environment, multiple ETL developers, analysts, and data scientists work together across different locations and time zones and make changes simultaneously. Therefore, it is important to take care of the continuous integration and delivery aspect in such scenarios.

Step 2: Choose a version control system

Version control systems play a critical role in ETL processes. Many tools are available to help you deal with different types of users working on a project. A version control system helps in better coordination and collaboration among team members spread across different locations and time zones.

NumerousDatasets

VariousScenarios

History Coverage

1 2 3 4

5

5

StreamAnalytix comes with an enterprise version management system with support for external systems like GIT. Choosing the right versioning system can help your business get a competitive edge, help you do frequent releases, and enable faster time-to-market.

Step 3: Test case and test suites

In a typical enterprise setup, the data, its sources, transformations, rules, and storage could change. Pipelines need adjustments to ingest and process any such changes. Without a robust testing and integration backbone, failures are inevitable.

StreamAnalytix pipelines have an inspection process where you can load data to create a flow simulation. Businesses can use the results of the simulation to build test suites and test cases.

Step 4: Identifying source and target systems

A typical delivery process deals with promoting artifacts to higher environments. With StreamAnalytix, you can create pipelines after the test, and making QA, UAT, etc. live. You can also automate all StreamAnalytix pipelines delivered to the higher environments from development to promote changes once they are tested and signed off.

Step 5: Continuous delivery and deployment

The final step of this process is to configure and set up the continuous delivery process. StreamAnalytix provides a template for CD scripts, which businesses can use to:

• Define deploy actions like source/test/target

• Execute test cases on the test environment

• Update pipeline in the target environment after successful execution of test case

Impetus StreamAnalytix is an enterprise-grade visual platform for all your batch and stream processing and analytics needs. Ingest, blend, and process high-velocity big data streams as they arrive, run machine learning models, visualize results on real-time dashboards, and train and refresh models in real-time or in batch mode. Teams can build and operationalize modern data applications five to ten times faster using a visual drag-and-drop interface, a comprehensive set of built-in operators, full application lifecycle support, and one-click options for on-premise and cloud deployments. With support for multiple big data engines and built-in extensibility, StreamAnalytix gives you complete flexibility and control to work with the technology stack of your choice.

How can StreamAnalytix help?

Continuous integration, delivery, and deployment are crucial to organizations developing ETL workflows. StreamAnalytix helps in:

1. Easing the entire ETL process

2. Ensuring bug-free ETL pipelines

While StreamAnalytix provides a visual interface to rapidly build and run ETL and analytics pipelines, it also seamlessly manages the entire CI/CD process.

6

StreamAnalytix is an enterprise grade, visual, big data analytics platform for unified streaming and batch data processing based on best-of-breed open source technologies. It supports the end-to-end functionality of data ingestion, enrichment, machine learning, action triggers, and visualization. StreamAnalytix offers an intuitive drag-and-drop visual interface to build and operationalize big data applications five to ten times faster, across industries, data formats, and use cases.

Visit www.streamanalytix.com or write to us at inquiry@streamanalytix.com

© 2020 Impetus Technologies, Inc.All rights reserved. Product and companynames mentioned herein may be trademarksof their respective companies. Jan 2020