building a data and analytics platform to leverage the power of ai · leverage the power of ai data...

White paper

Building a data and analytics platform to leverage the power of AI

1

White paper

Building a data and analytics platform to leverage the power of AIData and analytics

You need to be capable of managing your data and analytics before you think about leveraging the transformational power of Artificial Intelligence. Here’s how to do it.

White paper


2

Transforming data and analytics

Enterprises that leverage data as a competitive differentiator, who turn data into analytics that leads to faster better decisions, are more likely to outperform competitors. Developing a data architecture in the cloud can help enterprises scale, secure and automate data flows from data sources to data lakes, data warehouses to enable advanced analytics.

Whilst data is seen as a critical asset, its value is tied to its use. The biggest barriers enterprises face in extracting value from data and analytics are organisational, with many struggling to attract the right talent and incorporating data-driven insights into operations and marketing. Data projects fail not because of technology. Instead, the primary causes of failure are difficulties integrating with existing business processes and applications, management resistance, internal politics, lack of skills, and security and governance challenges.

Enterprises are increasingly seeking to make better and faster operational or strategic decisions. This may be realised through a combination of enhanced data integration and data platforms, data governance, data presentation, data science and deep real-time insights that predict future outcomes.

White paper


3

Enterprises must consider building the data platform, tools, practices and skills to support better decision making across all employees. Leveraging the public cloud and integrating with on-premise systems, adopting a microservices architecture and embracing data through a securely governed data lake augmented with machine learning (ML) and artificial intelligence (AI), are key requirements to enable agile innovation practices to drive a culture of experimentation and learning.

The expectations of your enterprise for getting real-time access to data analytics and insights is rising. The speed of getting the right analytics into the right hands to make the right decision has become more important, challenging the way data and analytics platforms are developed and managed. Embracing the cloud for modern data management, leveraging partners like AWS and Microsoft, provides enterprises access to ML and AI with performance, scale and cost efficiency.

Advanced enterprises partner to complement internal resources with proven external data and analytics support. These partnerships provide access to highly skilled team members or well-rounded squads to integrate seamlessly into their existing teams or bringing expertise in-house to scale teams sustainably and effectively. As you develop your data and analytics platform to get ready for advanced AI technologies, elastic deployment of in-demand skills with speed and scale will rise in importance.

Embracing the cloud for modern data management, leveraging partners like AWS and Microsoft, provides enterprises access to ML and AI with performance, scale and cost efficiency.

White paper


4

Step-changing your data strategy

The volume, variety, velocity of available data has grown exponentially, more algorithms have been developed, and computational power and storage capacity have scaled dramatically. Data and analytics are changing the basis of competition with leaders using their data capabilities to improve core operations, personalise customer experiences and launch new disruptive business models.

Most enterprises have more data than they know what to do with. This data becomes an opportunity if the enterprise is organised and resourced to handle the data generated by increasingly connected devices and users. It can also become a risk, if mismanaged, in terms of security or privacy breaches and poor decisions that leave customer needs unmet, the operation inefficient and potential unrealised.

Enterprises that transform their approach to capturing, storing, unifying and analysing their data assets, will be best placed to leverage the transformative potential of AI to improve business processes and create direct personalised relationships with their customers. These new capabilities will increase the amount of autonomous decisions made by your systems that will improve the quality of your decisions that shape the future.

Every enterprise has a range of data sources that may include customer transactions, supplier interactions, employee actions, digital assets and possibly connected devices, all generating increasing volumes of data that help you measure your organisation. When effectively analysed, this data can inform decisions and even help enterprises predict the future. With higher levels of confidence, AI can be leveraged to make autonomous decisions. AI monitor patterns in system and human behaviour and create algorithms to provide solutions.

The way platforms are developed and managed is rapidly changing. Traditional analytics was characterised by expensive large capex, on premise storage, designed to process gigabytes and not exabytes, dealing with primarily relational data where the majority of data was deleted to manage capacity and cost.

White paper


5

Seven C-suite questions

1 Governance and security

How can we ensure data governance and security at each stage of the data management including ingestion, storage, preparation and ongoing analysis?

2 Raw data How can we capture raw data to help us answer future questions quickly and cost-efficiently?

3 Unstructured data

How do we capture unstructured data like images, video and social posts outside of our existing databases and make sense of them?

4 Streaming How do we manage data streaming in real-time rather than processing historical data in batches to speed up analytics?

5 Unified view How can we integrate scattered data silos across on premise, off-premise, cloud and Saas partners to deliver a unified view of our data assets?

6 User access How do we extend the analytics capability beyond our developers, data scientists and business analysts so it's accessible closest to the decision maker?

7 Personalisation How can we apply advanced technologies, like AI, to create a direct personalised relationship with our customers to drive greater engagement?

Traditional analytics operated on isolated data silos including a data warehouse appliance and SQL database across ERP, OLTP, CRM and LOB systems feeding into some sort of data warehouse to provide some level of business intelligence on the current operation.

Enterprises need a well-developed data analytics platform, including automated processes and structured analytics, before attempting to adopt advanced technologies like AI. Trying to leapfrog best practices for data and analytics platforms, heightens the risk of failure resulting in cost, waste and frustration. This inhibits the opportunity to leverage advanced technologies making the enterprise vulnerable to competitors. Enterprises with strong basic analytics foundations and capabilities are more likely to successfully deploy advanced technologies to predict future operational performance or customer behaviour.

Developing a data and analytics strategy to deliver actionable insights and real-time decisions, powered by AI, could help optimise your supply chain, enhance predictive maintenance, improve operations, support dynamic pricing, increase cross-selling, and create a more relevant and personalised relationship with your customers resulting in lower costs, faster innovation and higher customer loyalty. Leaders need to consider governance and security, managing raw data, creating a unified view of processes, deploying basic structured analytics, user access and automation to drive value.

White paper


6

Data governance and security

Data governance and security need to be a key priority at each stage of the data management cycle including ingestion, storage, preparation and ongoing analysis.

Enterprises need to be clear about their information strategy, information management roadmaps, information governance reviews, governance maturity assessments, maturity uplift, standards and processes including access controls and business change or governance rollouts. These enhance the organisational capability for evidence-based decision making, mitigate information risks, improves information quality and reduces decisions taken on erroneous data and reduces operational costs of information management and makes investment more effective.

White paper


7

Capturing raw data

Capturing and centralising raw data into a cloud-enabled data lake helps create a single view of the enterprise across silos. This unified data can be then viewed by different teams with unique needs in real-time.

Because enterprises cannot predict the opportunities that will arise from the increasingly available raw data, and the questions that will emerge, storing this data cost effectively and with limitless scale, into a data lake provides the potential for future analytics. A data lake is a centralised repository that allows enterprises to store their structured and unstructured data at almost any scale. Data lakes extend the traditional approach providing the analytical power of data warehouses with the limitless scalability of serverless computing and the distributed processing of big data systems. A data lake enables analytics and makes AI enabled ML possible.

Enterprises can capture structured and unstructured data files in their source files and store it in their data lake. Given that storage is designed to be separated from processing and analysis, this data can be analysed later when there is a business question or problem to solve. As long as the raw data is captured and centralised and can be restructured with speed and scale, the enterprise can generate new analytics to deliver business intelligence with agility.

Accessing your key business data becomes more difficult if raw data is locked away in proprietary systems like ERP and CRM. Some platforms deliberately hamper efforts to extract your raw data as a form of vendor lock-in, as the vendor wants businesses to remain dependent on their own built-in reporting and analytics tools. After unlocking your business data, the next major challenge is ensuring data quality. Businesses can own years or even decades of data which has been collected and stored in a variety of methods without a strong focus on quality or consistency. Poor data quality can not only cost the enterprise it can also limit future potential to apply advanced technologies like AI.

White paper


8

Even basic structured data such as customer details can create quality issues if birth dates are in different date formats, postcodes are recorded in the wrong field or first names and surnames have been reversed. The challenge of data quality is even greater when working with unstructured data like medical records, where spelling can vary and different medical professionals can use different words or phrases to describe the same symptoms. This is where ML can be an invaluable tool when trained on a large data set.

The challenge lies in the different sources, formats and content of data. Unstructured data like web pages, emails, images, videos, social posts, customer service transcripts and chatbot questions are growing exponentially and creating new opportunities to build rich profiles of customers. This data is increasingly available, immediate and manageable at scale. Typically, enterprises analyse only a fraction of the data files available, especially for unstructured data which is rising exponentially and underleveraged. Data leaders are faced with the challenge of delivering real-time analytics while managing increasingly complex and diverse unstructured data formats and sources.

Typically, enterprises analyse only a fraction of the data files available, especially for unstructured data which is rising exponentially and underleveraged.

8

White paper


9

Case study: National Australia Bank (NAB)The NAB Digital Insights Platform project was an innovation initiative to apply cloud-hosted analytics technologies to the problem of understanding visitor and customer behaviour on the Bank’s digital properties (web site, internet banking platforms and mobile applications).

In the early days of the project, the platform’s viability was uncertain in terms of regulatory permission, security, technical feasibility and business value, so it operated in a fail-fast mode until much of the uncertainty was reduced.

Once the benefits of the platform were confirmed, Arq Group’s focus was to increase the speed of development and industrialise the platform.

Arq delivered a substantial redesign and rebuild of the software architecture while operating the legacy platform, maintaining service continuity. We also were central to extending the platform into near real-time analytics and automated decision capabilities.

“We have our people spending less time on data, and more time on understanding insights and driving better business decisions.”Anthony Coviello General Manager of Digital Commercialisation, National Australia Bank

White paper


9

White paper


10

Creating a unified view

Data analytics has undergone massive changes over the last five years and the pace of change is expected to accelerate. The cloud, the need for scale, new data and analytical skills and the need to improve customer experiences are all influencing this change. The cloud offers limitless storage and computing power cost effectively and flexibly.

We have seen the data available explode and we expect that with the Internet of Things (IoT), the growth will accelerate. If the amount of data grows at more than ten times every five years, data platforms will need to scale 1,000 times to be operational for the next 15 years, underlying the importance of scaling the data platform with view of the future requirements. New data science skills and roles are emerging and are influencing key data investments and these roles are gaining influence, even entering the C-Suite. Further, we are seeing applications, that form the customer experience, increasingly being informed by data to make them more personalised at scale and at real-time.

Leading enterprises effectively manage data across silos to create a unified view of their processes and customers. Cloud storage allows data to be centralised in raw native formats into a data lake which enables enterprises to analyse data into the future which can integrate into a data warehouse. Structured data is stored to be processed for a specific business case.

Building good data foundations enables users like data scientists, analysts and employees to make use of the data in effective and dynamic ways. Advanced enterprises are focused on building the data platform, tools, practices and skills to support better decision making and innovation.

White paper


11

Rather than just producing a high-level overview of trends, modern data analytic tools can drill down through millions of interactions and data points to identify the buying habits of an individual shopper at any point in time.

Structured data analytics and automation

Enterprises need to automate repetitive processes that compute lots of data and do not enable real-time analytics. The automation of data feeds is required to ensure that AI systems are analysing historic and real-time data to increase predictive capabilities.

Data generated from a wide range of web, mobile and IoT applications can be streamed into data processing systems in real-time using cloud streaming analytics and integrated into a cloud based data warehouse. Enterprises need to extract relational data from databases continuously, replicate this data and consolidate databases into a petabyte-scale data warehouse for reporting or data lake. Fast, scalable data warehouses make it simple and cost-effective to analyse and report all the data across your data warehouse and data lake. Automating the collection, classification, cleansing and integration of data from a range of sources enables leaders to define the schema, set up dynamically scalable clusters and start querying using standard SQL to extract deeper insights.

Visualising your data analytics, enabling users to share, publish, search and drill down into the data, helps drive internal adoption and usability. Simple and compelling visualisation on key measures, presented with dashboards or exception reports, can help focus employees on how their role directly impacts what is important for the team and enterprise.

Rather than just producing a high-level overview of trends, modern data analytic tools can drill down through millions of interactions and data points to identify the buying habits of an individual shopper at any point in time. Alternatively, these tools might be used to drill down to identify an individual piece of machinery or single part – determining when it is likely to fail, the impact of that failure and the actions required to minimise disruption and optimise operations. Further, enterprises need to design and deliver reliable performance reporting, servicing regular information requirements, including data acquisition, data integration, data engineering, KPI and performance management definition and visualisation analysis.

White paper


12

Case study: Australia PostWith a network of 4,500 post offices and the generation of 100 million data points per day, Australia Post wanted to utilise the best of this information to make customer experience innovations and improvements. The use of virtual reality (VR) technology offered a unique opportunity to peek into the future and improve their networks, customer experience and commercial returns, in a tangible way.

“Since the project completion, we have been able to change things for customers. This work is testament to the strong working relationship between Arq Group and Australia Post.” Stuart NickolsHead of Data Science, Australia Post

White paper


13

Experimenting with Artificial Intelligence

AI is the most important technology development of our lifetime, enabling systems to make autonomous decisions. AI refers to the ability for software to make autonomous decisions without being programmed transforming data and analytics. AI monitor patterns in system and human behaviour and create algorithms to provide solutions. We are operating at the early stages where AI and automation will change the nature of work, play and the economy.

The standard structured analytics can be integrated with AI to predict operational performance and customer behaviour. Identifying when a machine requires maintenance or a customer will churn, allow leaders to develop automated actions that deliver better decisions.

AI systems can help enterprises leverage unstructured data in new ways, scanning characters, images, video and voice, and layer it with structured data at scale to create rich insights. AI can also play a role in informing decisions that are not based on historical data. ML practices can be used when there is little structured data to predict how new products and services will perform. AI can help enterprises improve their forecast accuracy leveraging diverse and disparate data. Whether it be predicting a movie’s success based upon social media engagement, a company’s supply chain savings based upon less working capital as a result of better forecasting, or the performance of stocks on the share market, AI is helping enterprises predict performance by applying AI to structured and unstructured data sets.

At present, data analytics describes the current and historical performance of an enterprise helping you make informed decisions based on historical data. Established enterprises with large volumes of operational and customer behaviour data have a unique advantage when applying AI. Rather than focussing on insights based on past performance, AI can make decisions autonomously based on a prediction of the future, providing a competitive edge previously unknown. This represents a seismic shift towards AI enabled data and analytics.

But now you are increasingly able to make decisions based upon a prediction of the future, enabled by AI.

White paper


14

Personalising customer experiences

Leading enterprises are delivering solutions, using advanced analytics, to personalise online and in-person experiences at scale and in real-time. Online touchpoints and recommendations are being personalised with relevant, timely and compelling content. Previous non-digital channels are being transformed with data tools that help staff serve customers in-store or in customer service centres with personalised real-time data-driven insights.

Seven actions to develop your data and analytics platform

1Step-change data strategy and align culture

Don't focus on the technology at the outset, begin by focusing on insights-based value creation and then work backwards to find the best data and tools for the job. Enterprises with strong basic analytics foundations and capabilities are more likely to successfully deploy advanced technologies to predict future operational performance or customer behaviour.

2 Embed governance and security

Data governance and security need to be a key priority at each stage of the data management cycle including ingestion, storage, preparation and ongoing analysis.

3 Capture raw dataCapturing and centralising raw data into a cloud-enabled data lake helps create a single view of the enterprise across silos. This unified data can be then viewed by different teams with unique needs in real-time.

4 Create a unified view

Leading enterprises effectively manage data across silos to create a unified view of their processes and customers. Cloud storage allows data to be centralised in raw native formats into a data lake which enables enterprises to analyse data into the future which can integrate into a data warehouse where data is stored to be processed for a specific business case.

5Deploy structured analytics and automation

Enterprises need to automate repetitive processes that compute lots of data and do not enable real-time analytics. The automation of data feeds is required to ensure that AI systems are analysing historic and real-time data to increase predictive capabilities. Visualise data with intuitive dashboards, empowering your people to interpret data and provide business insights. Enable them to visualise, share, publish, search and discover.

6 Experiment with AI

Data science techniques and ML can deliver new levels of descriptive, diagnostic, predictive and prescriptive data analytics. Transform and operationalise your analytics capabilities from interpreting the past to forecasting future behaviour. The standard structured analytics can be integrated with AI to predict operational performance and customer behaviour. Identifying when a machine requires maintenance or a customer will churn, allow leaders to develop automated actions that deliver better decisions.

7 Drive personalisation

Personalisation requires a unified data foundation, decisioning logic and real-time triggers along with a content and campaign management platform that delivers content, measures performance and iterates based upon customer actions.

Advanced analytics, with ML, create customer signalling and real-time content triggers. A dynamic content management system, supported by a digital asset management system, creates the content for distribution across owned, bought or earned channels. These personalisation advances are generating increased customer engagement resulting in incremental revenue increases.

White paper


15

Conclusion

What we know about the future is that the volume, variety and velocity of data will increase each year, the number of people and systems querying the data will also increase, and the number of problems to be solved with ML will increase into the future. Faced with this future, advanced enterprises are seeing data as a future asset not a current cost. They are retaining their data, focusing on developing a data platform, making analytics more accessible to more users, using better processing techniques and deploying AI to make autonomous decisions to predict the future.

AI is transforming processes across a range of sectors from manufacturing to retail to financial services. An enterprise’s ability to realise the promise of AI is dependent on the ability of capturing and analysing data before the predictive powers of AI can be implemented and leveraged.

Despite their ambition to leverage the power of AI technologies, enterprises cannot afford to skip the basics on the data and analytics platform, that will become the foundations. Enterprises need the data management foundations in place before they start more advanced AI projects. Failure to build the basic data management stack with the intention of leapfrogging to greatness with advanced technologies, will lead to extra cost, waste and failure. If your enterprise is on the early path to data maturity, deploying advanced technologies, may be a leap too far.

The core foundations for AI include cloud storage, data warehousing, integrating data into a unified view and structured data analytics. These foundations enable continuous improvement by making data available for real-time automation and optimisation.

Digital transformation has made it imperative for enterprises to invest in their data and analytics capability to effectively manage and analyse data to enable better decision making whilst adopting AI and new technologies to shape the future.

White paper


15

arq.group 1800 664 222

Connect on Linkedin

About Arq Group Arq Group, previously Melbourne IT Group, is Australia’s leading digital solutions partner. Arq Group creates unforgettable experiences, solves complex challenges, and provides seamless, end-to-end solutions – from design thinking, mobile, cloud, analytical insights, digital marketing and web design capabilities that come together to create valuable products and channels and unforgettable consumer experiences. Arq Group powers the growth of businesses, big and small. Founded in 1996, Arq Group has evolved from the leading Australian internet infrastructure business to the leading Australian digital solutions partner. Today, the company builds and manages innovative products and channels to market for many of the country’s largest enterprises, and provides digital marketing solutions to Australian small businesses.

This publication contains general information only. By means of this publication Arq Group is not rendering bespoke professional advice or services. Therefore, this publication alone should not be used as a basis for any decision or action that may affect your business. Arq Group shall not be responsible for any loss sustained by any person relying on this publication alone. Given each business is operating under unique circumstances and with individual requirements, please approach us directly for specialist advice. A

rq_G

rou

p_D

ata_

&_A

nal

ytic

s_W

hit

e_P

aper

_v1

10/1

9

Connect on LinkedIn

Justin Parcell Executive Director, Southern Region

[email protected]

https://www.linkedin.com/company/arqgroup/

https://www.linkedin.com/in/justin-parcell-661342b/

mailto:justin.parcell%40arq.group?subject=Data%20and%20Analytics%20white%20paper

building a data and analytics platform to leverage the power of ai · leverage the power of ai data...

Documents