4 aa4 3925enw

September 2012, IDC Retail Insights #GRI237012

Big Data and Analyt ics in Retai l :

Unlocking Hidden Opportunit ies

W H I T E P A P E R

Sponsored by: HP

Greg Girard September 2012

I D C R E TA IL I NS I GHT S OP I NION

The new information economy — increasingly digital and intelligent — defines the terms of reference for retail today, and within this milieu, mastering Big Data has become a cornerstone for success. Mastering the four dimensions of Big Data — social, channel/customer, market, and supplier/product — for actionable and monetized insight requires newfound technical, organizational, process, and decision management frameworks to handle three challenging Vs of Big Data and analytics — volume, velocity, and variety — in order to deliver the other V — value.

Value from Big Data and analytics can come from three sources — gaining insight to improve processes and resource allocation, personalizing and localizing offers, and creating community as the nexus for branding and customer engagement. Extracting value economically from Big Data requires a new generation of technologies and architectures — shared information ecosystems designed to enable the high-velocity capture, discovery, analysis, and application of insights. Several analytical underpinnings of value creation are clearly evident, including context-aware and pattern-based analytics, quantitative natural language processing and social network analysis, and the extension of retail data models beyond conventional enterprise entities and ontologies.

Applying these insights to control and optimize operations, tactics, and strategies may require new business processes, decision management frameworks, and roles and organizational constructs. Opportunities to apply Big Data analytical insight reach deep into digital or social marketing or customer insight — the organizations where these efforts often originate — but just as important and likely more important, these opportunities extend into brand management and marketing, merchandising, product management, localization, pricing, fulfillment, and all channels of commerce.

What do the dimensions of "big," "fast," and "varied" mean with respect to the opportunities to create value from Big Data and analytics in your business? Accurately evaluating competitive and market scenarios that impact your Big Data and analytics business case calculus requires the strategic and tactical points of view of line-of-business leadership. Don't

Glo

bal H

eadquart

ers

: 5 S

peen S

treet

Fra

min

gham

, M

A 0

1701 U

SA

P

.508.9

35.4

400 F

.508.9

88.7

881

w

ww

.idc-r

i.com

#GRI237012 ©2012 IDC Retail Insights

let Big Data and analytics become the bright shiny new object of your attention. Forgetting that value from Big Data and analytics doesn't materialize just because you have invested in these capabilities will lead to too many "dry wells" filled with misspent time, talent, and treasure.

S I T UA TI ON OVE R VI E W

T h e N e w I n f o r m a t i o n E n v i r o n m e n t o f R e t a i l

As the digital world continues to emerge, information has become the singular environment within which retailers earn customer loyalty, bring successful new products to market, collaborate through supply chains with business partners, enable associates, reduce risk, ensure compliance, and above all, burnish their brand. At least that's the potential — if only retailers master Big Data.

As more Big Data and potential insight amass within and beyond the enterprise, traditional approaches to acquiring and analyzing relevant data and applying insight gained from interrogating the data no longer suffice. Barriers to capitalizing on this potential include the technical capabilities of conventional information technologies as well as the processes, decision management frameworks, and organizational constructs by which organizations govern data, develop information from the data, and apply insight to operational, tactical, and strategic decisions.

By some estimates, up to 90% of the world's information lies hidden in forms and formats beyond the competencies of the information management technologies commonly used in retail. Retailers own some of this hidden information within their four walls or a cloud — some of it unstructured (e.g., in product descriptions, customer emails, contact center notes, promotional agreements) and some of it structured (Excel spreadsheets being the nearly ubiquitous example). More and more information hidden in Big Data, however, lies beyond a retailer's information management assets — residing in social data, customer data, market data, and supplier data.

Big Data in the Social Context of Retail

The impact of Big Data — its promise amid peril — is easiest to see in how it plays out in the inherently social context of retail. A growing and diverse number of consumers now interconnect and influence one another with their use of social networking and media — YouTube, Facebook, Twitter, LinkedIn, Google+, blogs, wikis, and ratings and reviews. The speed, scale, and viral aspects of social media and networking influences, whether they are facts, opinions, or misinformation, dwarf old modalities of "word of mouth" and "power of the pen" pressures on shaping consumers' attitudes and opinions about brands as well as consumers' buying patterns and propensities.

©2012 IDC Retail Insights #GRI237012

Conventional ways by which retailers seek to understand consumer sentiments and the sources of influence on those sentiments fall short of the mark when their customers engage one another through social media and networks. Traditional information management approaches will no longer suffice to address these emerging demands and the resulting complexity.

The disparity between the speed and strength of consumer sentiments spreading across social networks and the limits of what retailers have to track these sentiments puts retailers at risk of losing control of their brand, the loyalty of their best customers, and market share, revenue, and margins.

The imperative retailers face is deciding how they should ready themselves to understand, react, and ideally engage their consumers in this brave new world. The key lies in mastering new competencies in Big Data and analytics to successfully and automatically integrate their insight into the various retail business processes.

L a t e n t V a l u e : T h e F i r s t V i n B i g D a t a

Social data is one of four dimensions of Big Data in retail. Customer data sources, market data sources, and supply data sources round out the IDC Retail Insights typology of Big Data in retail (see Figure 1).

Insights hidden in the Big Data of each dimension can improve decisions inherent in that dimension; for example, insight from loyalty and shopper profile data can improve customer segmentation and promotional targeting, and in another example, real-time stock (distribution center [DC], in-transit, store) data can improve next-day or same-day replenishment. Increasingly, however, new insight can be drawn by mashing up data across domains. For example, contextual analysis of contact center call logs, tweets, ratings and reviews, lot tracking of shipment records, product design information, and loyalty data taken together could spot product defects from customer feedback, identify root causes from product design information, identify stores holding any defective merchandise, and find loyal customers who purchased the item.

In more general terms, Big Data and analytics hold the potential to dramatically increase three imperatives of omnichannel retail:

● Insight applied in commerce (online and in-store); marketing (brand management, advertising, and promotions); fulfillment (from private label management to customer orders); and merchandising (from curating core assortments to localizing prices and promotions)

● Personalization applied in private promotions and offers, dynamic configuration of online assortments, and digital content presentation based on knowing the customer at any touch point and anticipating his or her needs


● Community applied in physical or digital social venues to harness the influence of social networks and media to create personal engagement with the brand

F I G U R E 1

R e t a i l B i g D a t a T y p o l o g y

Source: IDC Retail Insights, 2012

V o l u m e , V e l o c i t y , V a r i e t y : T h e O t h e r V s o f

B i g D a t a O p p o r t u n i t y

Realizing the opportunities from analyzing Big Data in individual domains and mashing up Big Data across domains, in other words, harvesting its latent value, is a function of an organization's ability to

Enterprise Systems of Record

Market Data Sources• Trade• Syndicated• Demographic• Geospatial• Events• Venues• Business• Economic• Weather

Social Data Sources

• Twitter• Facebook• Games• Shutterfly• Pinterest• Google+• YouTube• LinkedIn• Blogs• Wikis• Foursquare• Shopkick• Yelp• Citysearch

Billions of Interactions

Millions of Transactions

Cu

sto

mer

Data

So

urc

es

Su

pp

ly D

ata

So

urc

es

• Market baskets• Shopping carts• Loyalty and profiles• Offer/response• Shopping list• Click stream• SMS• Associates' black books• Search terms and patterns• Ratings and reviews• Downloads• Geospatial• QR scans• App usage• Web chat• Contact center• Sensor• Survey and focus group• Email

• Purchase orders• Shipments• Returns• RFID and sensors• DC stock• Store stock• Receipts• DC issues• Trade promotions• Product information• Design specifications• Market intelligence• Compliance


master the inherent volume, variety, and velocity of the types of Big Data shown in Figure 1:

● Data volumes growing from terabytes to petabytes, exabytes, or more. IDC Retail Insights estimates a base of 9.6PB held in social media today with 10.2PB being added in the year ahead. We also estimate — conservatively — that 19.8PB of created content drives 14,000PB to 18,000PB of content consumed.

● Data varieties extending in formats from structured to semistructured and unstructured and in sources from core transactional enterprise systems — one's own or those of trading partners — through which retailers operate and control their businesses; to collaborative and communications systems (e.g., email, enterprise wikis, and contact center logs); and to external informational systems — some structured (e.g., demographic data) but the vast volume unstructured (e.g., tweets or posts to a company's Facebook page).

● Data velocities and their business process and decision management frameworks changing from those amenably serviced by batch processes to those stubbornly dependent on the streaming of data acquisition, analysis, and application of insight.

F UT UR E OUT LOOK

Market and financial imperatives present the strategic context of an urgent and critical need to master the "supply" and "apply" sides of Big Data and analytics in retail. Merchandising and marketing present an excellent example of this need in the context of the ongoing shift of power to the consumer. The forefront of insights in these two domains has long since marched past the private knowledge domains of the merchant prince who controlled product risk and is now marching past the enterprise knowledge domain of spreadsheet merchants as they control market risk to the Big Data knowledge domain of the social merchant controlling customer risk. Today's retailer is dependent on mastering all three knowledge domains.

S u p p l y S i d e a n d A p p l y S i d e o f B i g D a t a a n d

A n a l y t i c s

Mastery of Big Data's four Vs is a function of two capabilities — supply and apply. Both dimensions require agility in the solutions that supply Big Data and the processes that apply its insights. Full realization of the value hidden in Big Data in the strategic context of real-time omnichannel retail wherein the consumer holds the balance of power requires new consumption models — from real-time contextual decision management frameworks to mobile presentation of visually rendered analytics in an enterprise context and mobile personalized content in the context of instrumented, informed, and interconnected customers.


Supply Side of Extracting Value from the Volume, Variety,

and Velocity of Big Data

Extracting value economically from Big Data requires a new generation of technologies and architectures designed to enable the high-velocity capture, discovery, analysis, and application of insights. These IT assets encompass the hardware and the software that integrate, organize, manage, analyze, and present data that is characterized by the four Vs of Big Data opportunity. This set of capabilities is the "supply side" of Big Data competence.

Apply Side of Extracting Value from the Volume, Variety,

and Velocity of Big Data

A second set of capabilities — of equal importance — addresses the "apply side" of Big Data competence — the business processes, decision management frameworks, and roles and organizational constructs by which an organization consumes Big Data insight to control and optimize its operations, tactics, and strategies to achieve its business objectives.

Shared Information Ecosystems

The value of Big Data and analytics cannot be fully realized unless its insights are applied in the retail enterprise ecosystem context of other business intelligence systems, in particular, in their service of marketing and positioning, merchandising and product development, supply and fulfillment, and omnichannel customer engagement through store and ecommerce channels. The contextualization of Big Data insight in the ecosystem of enterprise systems of processes and systems of record includes:

● Aligning analytics with business concerns (e.g., product quality, brand position, customer relationships, product development, marketing campaigns)

● Managing the life cycles of Big Data — defining, harvesting, organizing, analyzing, storing, reporting, and disposing of the data at the clock speed of the business processes in which it is consumed

● Reporting and applying insights in process decision management frameworks (e.g., roles; decision rights; operational, tactical, and strategic time frames and spans of control; and plan, do, analyze cycle times)


R e t a i l B i g D a t a a n d A n a l y t i c s M a t u r i t y

Figure 2 presents the five dimensions of Big Data and analytics in the context of retail:

● Intent — context of strategic, tactical, and operational decision management

● Process — relevancy of Big Data and analytics to roles, responsibilities, and outcomes

● People — alignment of organization and cultural norms with the use of Big Data and analytics

● Data — governance and life-cycle management of voluminous, varied, and high-velocity data

● Technology — priority focus on hardware, software, and services for economical supply of relevant Big Data and analytics

F I G U R E 2

R e t a i l B i g D a t a a n d A n a l y t i c s M a t u r i t y M o d e l


Big Data Maturity

Intent

ProcessPeople

Data Technology

Clear view of business and market trends relating to the power of

Big Data analytics for insight, personalization, and community formation; the competitive

risks and rewards related to the use of Big Data analytics; and commitment to infusing the

omnichannel enterprise with the capabilities of Big Data analytics

IT priority focus on technologies and

architectures designed to extract value

economically f rom very large volumes of a

wide variety of data by enabling high-velocity

capture, discovery, analysis, and application of

insights to decision making

Governance and MDM of structured and

unstructured Big Data for consistent enterprise

and external data f rameworks for analytical

support of business processes

Big Data insight made relevant and applied in

processes that span departments and functions

with clarity of roles, responsibilities, decision

management f rameworks, and dependencies for

continuous plan, do, analyze cycles

Organizational alignment to collaborate

with Big Data analytics, def ine roles and

training for skills to apply the insights of

Big Data analytics, and align

compensation to create a Big Data

analytics retail culture


V a l u e C r e a t i o n O p p o r t u n i t i e s f r o m

B i g D a t a a n d A n a l y t i c s i n R e t a i l

Ultimately, in the business context of retail, value creation from Big Data and analytics should result in the monetization of hidden insight revealed and applied. Commercial paths to monetization can lead to product and service innovation and with it differentiation, premium pricing, and customer loyalty; process control — compliance, product quality, security, and fraud prevention; and optimization of processes and decisions — cost control, margin growth, and efficiencies.

Analytical Underpinnings of Value Creation

The range of new analytical underpinnings for monetizing Big Data insight includes the following:

● Context-aware analytics: Integrating Big Data insight with enterprise transactional data and business intelligence (e.g., predictive analytics, forecasting, and root cause analysis) such that the situational awareness of Big Data analytics complements and supports human judgment

● Broad-scope pattern-based analytics: Proactively seeking episodic, recurrent, and systemic patterns or systems of factors within process and organizational domains ("silos") and supersystems of factors spanning silos from conventional and unconventional sources that can positively or negatively impact outcomes

● Quantitative natural language processing analytics: Discovery and disambiguation of meaning primarily in digital communications (e.g., emails, tweets, blogs, and Facebook posts) and content repositories (e.g., product descriptions in product data management system or customer ratings and reviews)

● Extension of retail data models beyond conventional enterprise entities and ontologies (e.g., foundational notions of products, customers, regions, stores, promotions, channels) to include friends, followers, sentiments, and interests as extensions of customer attributes; "likes," "wants," and reviews as extensions of product attributes; and forums and topics at the nexus of people, places, and products

Opportunities for Applying Big Data and Analytics in the

Line of Business

Opportunities to monetize Big Data and analytics span core retail processes including brand management and marketing; merchandising; fulfillment; and commerce. Examples and use cases of Big Data and analytics for each of these processes are shown in Table 1.


T A B L E 1

O p p o r t u n i t i e s t o M o n e t i z e B i g D a t a a n d A n a l y t i c s i n R e t a i l

Process Process Responsibilities Sample Applications and Use Cases

Brand

management

and marketing

Brand differentiation

Support and execution of brand

management differentiation

strategy via development and

execution of media campaigns

Customer analytics, insights,

and segmentation

Creation of promotional

campaigns to shape customer

demand in collaboration with

marketing




merchandising

Identification and characterization of threats to brand values

(e.g., boycotts or campaigns reacting to unmet corporate social

responsibility commitments)

Intelligence about strengths and weaknesses of competitors by

customer segments, merchandise categories, product quality,

pricing, locations, and other categories of strategic importance

Intelligence about customer sentiment trends by customer

segments, merchandise categories, product quality, pricing,

locations, and other categories of strategic importance

Identification of networks, nodes, and personalities of viral

influence

Assessment of the impact of marketing campaigns and events

on consumer awareness, sentiments, behaviors, and intentions

Merchandising Selection, curation, localization,

allocation, distribution, and

pricing of merchandise in all

commerce channels




marketing

Intelligence about new product development, design, and

introduction — aspects that delight customers, meet their

expectations, or are of low value

Intelligence about local demand for products and services not

carried in local assortment

Fulfillment Sourcing, developing,

designing, and delivering of

merchandise into commerce

channels in collaboration with

merchandising

Fulfilling customer orders from

own network, suppliers, and

marketplace

Early warning of product defects and shortfalls of performance

and attributes against customer expectations

Intelligence about new product development, design, and

introduction — aspects that delight customers, meet their

expectations, or are of low value

Commerce Presenting and selling

merchandise and services

across all channels

Operating all channels of trade,

including stores, catalog, call

centers, and digital — mobile,

ecommerce, social, and third-

party channels

Store- and market-specific customer concerns and delights

regarding store operations, associates' engagement practices

and product knowledge, customer service, crowds, wait times,

and so forth

Real-time in-store, near-store customer tweets for customer

service, product information, and product location

Customer concerns and delights about performance and

characteristics of ecommerce, mobile, social, and other digital

channels



C HAL L E NGE S

What do the dimensions of "big," "fast," and "varied" mean with respect to the opportunities to create value from Big Data and analytics in your business? The answers will define the terms of reference for your Big Data and analytics value creation journey. Gaining clarity and consensus on these definitions will require you to address technical, cultural, and organizational challenges. Left unattended, these challenges will delay and distort identification and definition of gaps in the five dimensions of your Big Data and analytics maturity profile — intent, process, people, data, and technology.

Ignoring the dictum that "better can be the enemy of good enough" can derail your Big Data business case calculus. Accurately evaluating competitive and market scenarios that impact your Big Data and analytics business case calculus requires the strategic and tactical points of view of line-of-business leadership. Against that input, the ability to define investment tripwires — investment priorities, dependencies, and time lines for scenarios most likely to come sooner and create more risk — and using them to sequence and schedule investments under likely scenarios requires the strategic and tactical points of view of information technology leadership.

Becoming mesmerized by the allure of Big Data and analytics as a bright shiny new object of your attention presents another challenge to be managed. Opportunities to create value from Big Data and analytics projects don't materialize just because you have invested in these projects. Forgetting that will lead to too many "dry wells" filled with lost time, talent, and treasure.

OV E RV I EW OF HP B I G D A TA A ND

A NAL Y TI C S OF F ER I NG S

HP offers a wide range of information management and analytics (IM&A) capabilities drawn from its enterprise services, software products, and cloud and security platforms to address Big Data challenges and opportunities in retail.

HP IM&A services include information strategy and organization, information management and architecture, business analytics and information delivery, and social intelligence. HP provides these consulting services around its own Big Data software assets, Autonomy and Vertica, and third-party software assets, in particular SAP HANA and Microsoft SharePoint/BI platform. HP supports this with a variety of software and solution delivery models including on-premise installation, hosted, software as a service (SaaS), cloud computing, and multitenant SaaS (cloud deployment).


HP focuses its capabilities to enable retailers to proactively manage information-related business risk, enhance customer experiences, and optimize business performance to create competitive advantage.

HP's Autonomy software asset helps retailers develop connected intelligence from structured and unstructured data for actionable decisions that improve business performance.

HP's Vertica analytics database delivers scalable performance on Big Data queries enabling real-time decision making to be embedded in retailer processes in order to optimize business performance.

Vertica and Autonomy deliver a powerful combination for real-time analytics and decision making using structured and unstructured data across the enterprise.

S t r e n g t h s a n d C h a l l e n g e s

HP offers a product and services portfolio that is consistent with what IDC Retail Insights expects from a market-leading technology provider to the retail industry. In addition to the software and services capabilities noted previously, HP provides the following infrastructure component to enable Big Data application deployments:

● Cloud computing. HP can offer clients a variety of enterprise-grade cloud computing solutions, including public, hybrid, and private. The ability to offer private cloud to retailers is considered a strength for a number of reasons, not the least of which is a tighter security model.

● Security. HP offers clients a security strategy through the HP Security Framework, designed to offer end-to-end information security plans and execution road maps. Because of the sensitive nature of customer data and the requirements and penalties imposed by the regulators, security is top of mind for industry IT executives.

● Mobility. HP's approach to enabling enterprise mobility is suited for organizations that wish to reach their constituents across multiple networks and devices by delivering applications, content, and services in a scalable, secure, and reliable way. This approach leverages HP's global applications services capabilities to provide the architecture, systems engineering, development, and support services. Combined, they help an organization simplify its applications and extend them where necessary as well as build mobile business-to-business, business-to-consumer, and business-to-employee applications. This approach also leverages HP's service-oriented architecture–based integration architecture and is enabled by development and security frameworks that help create componentized building blocks from monolithic legacy applications to develop and deploy mobile applications.


HP faces several unique market challenges as well as many of the same market challenges as other enterprise vendors servicing the Big Data marketplace:

● Demonstrating leadership in the transition from information management to Big Data. HP's legacy information management solutions, including TRIM, Data Protect, and others, are not broadly implemented across the retail marketplace. HP must more effectively demonstrate leading capabilities at the component level with high-profile Autonomy and Vertica wins as well as highlight its ability to deliver end-to-end Big Data software, services, and hardware with lighthouse and marquee clients.

● Offering competitive, differentiated solutions portfolios. The retail market, as a traditional early adopter of emerging technologies, senses the value of the Big Data trend and is budgeting accordingly. HP's competitors are also focused on expanding their solution portfolios on the Big Data trend in terms of breadth and depth of product capabilities, IT infrastructure, and professional services. HP must leverage its industry position and demonstrate the value of a Big Data relationship with HP to gain Big Data market share.

● Channel network management. HP's channel strength is also a weakness. Successful execution of HP's Big Data strategy to broaden its portfolio will require that HP reinforce its position in the enterprise space without alienating the channel that has been so beneficial to the organization.

E S S E NT IA L GUI DA NC E

● Adopt an enterprisewide approach to Big Data analytics. The design, development, and deployment of Big Data analytical capabilities should be seen from the outset as an enterprisewide undertaking even as nascent initiatives of the sort incubate within marketing, in particular digital marketing, where Big Data analytics are applied to social Big Data, or in ecommerce and personalization where Big Data analytics are applied to customer Big Data. The design objectives of an enterprisewide approach to Big Data analytics should include:

○ Program governance led by a leadership team drawn from across four core omnichannel processes: marketing, commerce, merchandising, and fulfillment

○ Alignment of insights from the four quadrants of Big Data sources — social, customer, market, and supply — with the four dimensions of retail intelligence — customer intelligence, offer intelligence, channel intelligence, and merchandise intelligence


○ Delivering insights within the decision management context of the roles they inform — that is, task aligned and just in time — and the style in which and the speed at which the decisions are made

● Look for monetization opportunities across the board. Big Data analytics can inform decisions beyond marketing in merchandising, commerce, and fulfillment. Actionable insights include marketing campaigns; product design and quality; new product development and introduction; assortment localization; pricing, promotions, and personalization; crisis management, selling techniques, and store associate management; store operations and design; and competitive intelligence and differentiation.

● Maintain a readiness for rapid evolution of Big Data analytics. Big Data analytics is a new area; new capabilities for and opportunities to apply Big Data analytics are emerging quickly. Managing rapid change on the supply and apply sides of Big Data analytics requires agility in a host of technical competencies (e.g., data management and governance, commodity hardware configurations); organizational dimensions of skills, roles, and decision rights; consumption models and use cases; and more. Retailers will increasingly need to find sources of innovation and should look to Big Data information partners. Retailers should screen for partners that provide best-in-class solutions consisting of open source technologies/capabilities and proprietary technologies/capabilities, as needed, that are best suited for their unique requirements.

C o p y r i g h t N o t i c e

Copyright 2012 IDC Retail Insights. Reproduction without written permission is completely forbidden. External Publication of IDC Retail Insights Information and Data: Any IDC Retail Insights information that is to be used in advertising, press releases, or promotional materials requires prior written approval from the appropriate IDC Retail Insights Vice President. A draft of the proposed document should accompany any such request. IDC Retail Insights reserves the right to deny approval of external usage for any reason.

This document was reprinted by HP with permission from IDC Retail Insights.

4 aa4 3925enw

Documents