Creating meaningful analytics using product management principles

June 2021

Agenda

Advana 2

What is product management and why is it important?

Elevating the role of Financial Management

Introduction to Advana

Advana Product Examples

Advana Product Management Lifecycle

What is product management?

What is product management?

4Advana

Product management: the act of managing the business strategy behind a product, including specification of its functional requirements, the launch of features, and the communication to customers.

Examples of competencies that product managers should have:• Performing market assessments• Conducting customer interviews and user testing• Translating business-to-technical requirements, and vice versa• Running design sprints• Feature prioritization and roadmap planning• The art of resource allocation (it is not a science!)• Defining and tracking success metrics

Elevating the role of Financial Management

6Advana

Understanding our changing environment

7Advana

Understanding our FM customers and workforce

8Advana

How our roles evolve

Introduction to Advana

10Advana

What is Advana?

Who we areAdvana – a mash-up of the words “Advancing Analytics” – is our technology platform, which, in addition to housing a collection of enterprise data, is pushing beyond simply being a data warehouse; we are arming military and business decision makers across the Department with decision support analytics, visualizations, data tools, and associated support services.

Our visionChange decision-making behavior across the DoD enterprise through the transparency and flexibility provided by enterprise data and analytics.

Our missionSupport our National Defense Strategy and FM of the Future by making data widely accessible, understandable, and usable across the DoD enterprise. Take action upon the profound opportunity to help the DoD translate common enterprise data into actionable insights, decisions, and outcomes.

11

Foundation Built Upon Audit Improved Data Quality for Trusted Insights

Government Owned and Secured

DoD accredited system which can serve as along-term steward of business, mission, and

crisis related analytics on NIPR and SIPR

Open Architecture with No Vendor Lock-in

Best of breed open source and COTS tools are built on an open architecture, hosted on

AWS Cloud, with the ability to pull government, public, and commercial data feeds

Single Source of Truth

Thousands of users have access to over 200 DoD data sources across business domains

(e.g., Personnel, Finance, Supply Chain)

Scaled Agile Flexibility

Program resources leverage Scaled Agile (SAFe)methodology for planning and delivery, which allows

for collaboration across our team and our customers as we iterate on solutions together

Advana provides real business value to the Department of Defense

Customer-centric Design

Product design is driven by customer business and mission needs, based on customer input feedback at

all organizational levels

Something for Everyone

Range of ready-built visualizations and self-service options makes Advana relevant and impactful for ALL

DoD users – from senior and mid-level decision-makers to analysts, developers, and data scientists

Our platform began as the Universe of Transactions for purposes of financial statement audit, and grew from there. The same audit principles that instill trust in our data for purposes of better decision making are applied to everything we do as a platform. This foundation

leads to better business and mission decisions and outcomes via the ability to improve data quality, ask better questions of our data, and in turn, support decision making that creates value for the warfighter

12Advana

Advana provides an integrated data experience to inform decision makers

2. Understand and explore data

3. Query and analyze data

4. Visualize and share results

1. Discover new data sets

• Search datasets across platform• Discover trending data• Get recommendation based on your

searches

• Request new Advana project workspace• Develop data science and machine learning

algorithms against data• Run containerized models on scalable

infrastructure

•Visualize data and resulting models with drill-down dashboards

•Develop reports accessible to your community

• Role-based access• Web-based access• Horizontally (infinitely) scalable• Containerized enclaves

1. Discover data

2. Understand data

3. Query and analyze

4. Visualize Results

Advana Approach

Data Scientist

Data Analyst

• View data profile pages• Understand uses of data• Request access to data

Advana Principles

Advana provides users with a seamless experience and journey across the platform from discovering data all the way to obtaining insights from the data

Decision Maker

13Advana

Advana’s analytics offerings by portfolio

Dormant Account Review-Quarterly (DAR-Q). Provides a streamlined and centralized tool to conduct dormant balance reviews, provide auditable workflows, and collect data on reasons for dormant accounts. It allows increased insight into execution of budgetary resources and improves fiscal accountability and awareness.

Feeder Reconciliations. Offers a centralized application that reduces the time and resources required to generate reconciliation results. The application brings consistency to the 4th Estate in the way feeder reconciliations are performed across all DoD systems and agencies to support existence and completeness of financial data for audit.

Improper Payments. Focuses on identifying improper payments across financial systems in the DoD. This tool not only flags potential improper payments, but also includes implementation of a workflow and management dashboards to track, manage, and report results of improper payment detection.

EXAMPLE PRODUCTS

Advana Product Examples

Dormant Account Review-Quarterly (DAR-Q)

Advana 15

Concepts and Benefits:

• KPIs measuring number of total records reviewed and associated dollar values

• Charts depicting number of validation types, number of records reviewed over time

• Other key pieces of application include quarterly results review summary, agency analysis and drilldowns, dormancy by agency, and user growth

Feeder Reconciliations

Advana 16

Concepts and Benefits:

• Summary variance by system with ability to drill down by agency and other factors

• KPIs for current month and cumulative, tracking variance, match rate, and traceability rate

• Ability to view details for unmatched transactions and filter for transaction age, ticketing status, and more

Improper Payments

Advana 17

Concepts and Benefits:

• Conducts integrity checks across seven different FM systems to identify outliers, flagging those payments for further investigation

• Dashboard views showing results counts and potential dollar impact for resolved and unresolved integrity checks

• Ability to filter by location/site and system, helping teams monitor and manage workload

• Detail views for payment details and ticketing information

Advana’s Product Management Lifecycle

Core principles of product management

Advana 19

Product management principles:

• Focus on customer needs and business questions• Verify at every step that we’re building the right

product that answers questions and solves a real problem

• Emphasize the right product features at the right time for customers

• Be customer driven and create a delightful customer experience

• Address the full end-to-end product management lifecycle, NOT just the development phase

• Take an experimental approach and build a prototype to create a minimum viable product

• Release often and incrementally• Plan strategically but think tactically

ProductManagement

Build the product

right

Build the right

product

Advana 20

The Product Management Lifecycle ensures that every customer of every product has a consistent experience

The Advana product management methodology

ensures that every product we create is created with the customer’s needs and our

business strategy in mind. Every feature, design decision, and communication mechanism is

focused on the customer experience.

NOTE: Although this process is linear for a static set of requirements, we recognize that

things change; adapting to customer feedback is paramount to successful product management! Emerging new requirements or enhancements may require revisiting an

earlier stage, as appropriate.

Phase 0: Product Discovery & Ideation

Phase 0: Product Discovery & Ideation

• What specific problem are we being asked to solve?

• Who are the parties (or personas) that need the support? Clearly defining personas, or the roles and audiences that are involved, is critical to understand the sponsors, champions, stakeholders, and end users.

• What business questions are trying to be addressed? This helps lend insight into the value proposition of the product.

• What solutions already exist that have tried to solve this problem, or related problems, in the past? What made them successful or unsuccessful?

• What data is needed to answer these questions? Understanding the data needs and a high-level understanding of what data is available today helps scope the effort.

• Why are we using resources to build this? And is this a good idea? Resources are limited and there are lots of ideas, is this a good idea that brings value to our customers and do we have the resources to take this on at this point in time?

Advana 22

During this phase, a problem or a need is identified. This phase is focused on understanding the current problem and the need, the parties that need the support, the business questions that need to be addressed, and the data needed to support these questions.

Advana 23

How do we answer these questions today?

Your team should be asking themselves questions to explore the potential of a product idea. Use this time to ask questions, whiteboard, scribble, and be open to any ideas that are shared.

What is the

problem?

What questions are we

trying to answer?

What data is needed to answer ?

Who is affected?

Who is responsible?

What is the value? Is this a good opportunity?

Phase 0: Product Discovery & Ideation

Start by asking these questions… …end with a clear idea of what we want the product to accomplish and confirmation that it is a good investment of resources.

Question Metric Action Required

Phase 0: Personas

Advana 24

Understanding WHO needs your product and WHY they need it is critical to your team's ability to build a successful product –meaning, one that effectively meets your customers' needs. After all, the product is being created to help someone solve a problem - it's important to understand these drivers early so that you can begin designing a solution to meet that need.

Consider answering the following for each persona identified:

• What is my role and rank?

• How long have I experienced this problem and what othersolutions have I tried?

• What or who influences me?

• What or who do I influence?

• What motivates me?

• What do I need to do my job?

• What frustrates me?

• What will make me an advocate?

Phase 1: Product Strategy

Phase 1: Product Strategy

Advana 26

The Product Strategy phase is the series of activities that help set the direction and focus for the product. This phase drives teams to align on a mission, vision, and objectives and key results. This also requires teams to estimate the size of their target market, or target audience, and determine the most effective approach to reach that market.

Mission is the purpose of your product as of now (including its value to customers, inspiration, etc. ). It is the intent your team holds right now that helps you realize your vision.

Vision statement will be the north star guiding the team as they go through design, planning, development, launch. It articulates the future for the product.

Objectives and Key Results (OKRs) are simply “WHAT” is to be achieved and “HOW” we measure success.

Target Market Size requires reflecting back the personas you are serving and estimating just how large the customer base will be.

Customer Acquisition Approach is the way in which you will reach the segments of your target market, including the channels or communication mediums you will use to reach them effectively.

Phase 1: Mission and Vision

Advana 27

The Department of Defense cannot easily integrate enterprise-wide data across a variety of complex data stores, often managed insilos. This hinders analysts' ability to develop meaningful, evidence-based models and solutions to help leadership make data-driven, mission-critical decisions.

Support our National Defense Strategy by making data widely accessible, understandable, and usable across the DoD enterprise. Take action upon the profound opportunity to help the DoD translate common enterprise data into actionable insights, decisions, and outcomes.

Change decision-making behavior across the DoD enterprise through the transparency and flexibility provided by enterprise data and analytics.

Reflect back on the problem statement when determining your product’s mission and vision. This is critical to setting the direction for the product in a way that serves as a “north star” throughout planning, development, and beyond.

EXAMPLE

Phase 1: Objectives and Key Results

Advana 28

The Objectives and Key Results (OKRs) framework was created by Intel’s Andy Grove and then popularized by venture capitalist John Doerr in his New York Times best-seller “Measure What Matters.” Companies from Google to Adobe have rolled out OKRs to accelerate growth and drive innovation by helping teams see how their work fits into the overall company’s objectives. OKRs are aggressive, bold, and inspirational, helping move the something strategically important to your to your organization.

OBJECTIVES

KEY RESULTS

Objectives (Os) are simply “WHAT” is to be achieved. Significant, concrete, action-oriented, and ideally inspirational.

Key results (KRs) are quantitative, measurable targets that demonstrate the targets that achieve the objective. Effective KRs are specific, time bound, aggressive yet realistic, measurable, and verifiable. They demonstrate “HOW” we achieve success.

OKRs..• Create a bias for action• Drive prioritization and focus for teams• Help us be accountable to our mission• Help keep us focused on outcomes• Communicate what we accomplish very clearly

Phase 1: Target Market

Advana 29

Understanding WHO needs your product and WHY is critical to your team's ability to build a successful product - meaning, one that effectively meets your customers' business/mission needs. After all, the product is being created to help someone solve a problem - it's important to understand these drivers early so that you can begin designing a solution to meet that need.

• Target market – Start by calculating the total addressable market.There is no one way to calculate this, but consider starting with yourpersonas and estimating how many of each persona exists today.Then, narrow in on where you think you will have the most success bytargeting early adopters – identify segments of the market wherepossible. Estimates are fine for this exercise.

• Customer acquisition approach – Think through the most effectiveways to reach your target market above. This will help you to thinkabout upfront before product design and development, because partof this exercise requires thinking through how your customersconsume information.

Phase 2: Product Design

Phase 2: Product Design

Advana 31

The Product Design phase is focused on sketching solutions in more detail in order to obtain early user feedback. Product teams start to understand the data sources available and explore the data, define needs and themes across those needs, brainstorm and wireframe possible solutions, conduct customer interviews, and define the intended the customer journey.

• What specific features, most often metrics for analytics dashboards, is each of your personas most interested in?

• How can we organize those metrics or features based upon how each persona consumes information? What would that wireframe look like?

• How would each persona interact with the wireframe, frame by frame? Is there a journey mapped out for each persona?

• How will the data be required to behave to support this user journey? How important is a common data model to answering the business questions and features? What are all the raw and reference data sources required, what does this relationship look like, and what are all the join criteria?

• What will the application look like when clicking through each step?

• How do customers feel about the prototype early on, and how should we pivot if they request changes?

Phase 2: Metrics that Matter and Wireframing

Advana 32

Consider going back to the business questions, inventorying the metrics that matter to your target market, and build a wireframe of your product solution infrastructure to align with those metrics and the business/mission questions that started the product discussion in the first place.

• Metrics that matter - it is important to think of your product from a user standpoint in the design phase and think, “What is a user trying to get out of this product?” - On Advana, most of these products are aimed at creating holistic data views and common operating pictures for a domain or use case

that spans the entire DoD enterprise- For these data-driven products, it is important to understand the key metrics this product should be able to produce

Now your team can begin wireframing your product design infrastructure.

- Conduct data exploration: refine your initial data requirements into specific data sources, data sets, tables, and data elements required to build your product and answer your business questions

- Through sketching solutions and thinking through design, teams can understand is this feasible technically?

- Through rapid prototypes, mock-ups, and sketches, teams can gauge if these potential solutions are user-friendly

Image Source: www.usabilitygeek.com

Phase 2: Data Model Sketch

Advana 33

Data model sketching is important to understanding how all data sources and reference data required for your solution need to bearchitected.

• In coordination with data source owners, obtain data extracts, data dictionaries, representative data and explore the data sets. Understand how the data from the different sources can be linked and aggregated to provide a holistic view

• Understand the raw data in order to set up the data pipeline –different formats per system from different sources- Various delimiters in use ie , | ^ - Fixed width format – requires additional logic to parse from

the raw data

• List all data sources and reference data tables required

• List the join criteria

• Sketch a relational diagram prior to starting development and socialize with the product team, data source owners, and functional points of contact to verify that the way the data is being used or transformed is appropriate in order to solve for each business question and metric

Phase 2: Prototype and Early User Feedback

Advana 34

Bring your wireframe to life in a clickable prototype and solicit early user feedback to validate that product design meets customer needs.

• Your prototype should be a “quick and dirty” version – focus more on usability and workflow versus visual design elements in this phase

• Test your prototype with real users to gather user impressions on the proposed solution, identify areas of the prototype that are confusing or misleading, and test the intuitiveness of the task flow

Source: www.mindtools.com

Phase 3: Product Development Plans

Phase 3: Product Development Plan

Advana 36

The Product Development Plan phase is product team should define features and prioritize. Features are a portion of functionality a team delivers towards a product and provides distinct value to the customer. Features can refer to capabilities, components, user interface (UI) design, and performance upgrades.

The intent is to determine what are the necessary sets of features that are needed to get the solution right to solve the problem statement

Desirable or Launch (P1) Nice to Have (P2) Needed but less

desired (P3)

Features are launch blockers, as in the product will not launch without this feature.

P0-1 P0-2 P0-3 P1-1 P1-1

Not required for launch but fast follow in future releases

P2-1 P2-2 P2-3

Nice to have.

P3-1 P3-1

Beneficial for the future product to have incorporated but are peripheral to the overall mission (i.e., specific to certain stakeholder groups or edge cases).

Critical for Launch (PO)

Phase 3: Product Roadmap

Advana 37

Effective roadmaps provide context about “why” by including a statement about the product vision and mission, a set of outcome oriented-themes.

“A product vision should be about having an impact on the lives of the people your product serves, as well as your organization” – Products Relaunched

• Your product roadmap should:- Put your plans in a strategic context- Focus on delivering value to customers and your organization- Embrace learning as part of a successful product development

process- Rally the organization around a single set of priorities- Get customers excited about the product’s direction

• At the same time, your product should NOT:- Make promises product teams are not confident they can deliver on- Require a wasteful process of up-front design and estimation- Be conflated with a project plan or a release plan- Be a step-by-step plan of the solution, your roadmap focuses on

setting the strategic direction.

Phase 4: Product Development

Phase 4: Product Development

Advana 39

During the development phase, teams incrementally and iteratively build, test, and validate functionality of a product solution. Teams iterate through development and demo to end users and target audiences as often as possible (but at minimum) at the end of a sprint (two-week cycle). Our data engineers will build data ingestion pipelines, write business rules, create data models, architect solutions, and work jointly with front-end developers or analysts.

• Agile development provides for:- Iterative Feature development (each increment builds upon the previously accepted delivery

and current urgent customer need)- Incremental (fast delivery)- Emergent Design- Quality is Built into the Process (Validation occurs throughout the process)- Continuous improvement through assessment and discovery through all

product development steps

• Adaptive planning from high stakeholder engagement and high performing Teams is required because the steps are:- Not necessarily linear, some activities may be completed in parallel or in

an alternative order- Repeated multiple times throughout the Product Development Cycle

• Risks are reduced because small delivery cycles ensure acceptance and allows for quick pivot away from undesired results with little investment

Phase 4: Provisioning and Building Data-Driven Products

Advana 40

Data Architecture and WranglingTool and DB Provisioning

Testing

Validation

Provision• Leverage hosting Infrastructure

(Amazon Web Services, on-prem SIPRNet)

• Comply with Identify and Access Management

• Utilize internal Advana User-Facing Tools and Applications

Conduct functional testing• Script and

automated• Execute in pre-

production environment

Perform regression testing

Conduct User Acceptance Testing• Ensure

acceptance criteria has been met

Conduct data acquisition and outreach – getting data from external system and making it available as raw data to Advana• Identify data source & acquisition

method• Obtain MOA

Leverage COTS / GOTS / Open Source (OSS) software:• Addition or licensing for

• Tools (non-security tool) • Plug-ins or SW expansions• Patches (industry or project-

supplied)• Addition of a StreamSets library (e.g.

Python library)• New Qlik extension• Qlik extension upgrade, including

custom code Develop custom applications, APIs, and libraries • Advana software engineering services

provide the tools and processes necessary to support rapid, distributed development on the Advana platform.

Ingest – prepare data and automation• Prepare ingest framework / data

pipeline• Transform the data and apply the

CDM to transform the raw data to a useable format

Adhere to data architecture, quality, and governance• Attain trusted data products that

can scale, and provide consistent quality, reliability and performance

Prepare data infrastructure• Store and organize data in tables and

databases• Create metadata to facilitate searches

Phase 4: Testing and Validation

Advana 41

Building the right product

Building the

product right VerificationValidation

• Did we build the right product to meet user needs?

• Conduct validation that the product meets the users need and works from the customers point of view

• Validation can occur during A/B testing, user acceptance testing, feedback through guided small audience demos

• Externally conducted testing (not the product team) or guided to gain outside perspectives

• Did we build the product right for it to function as intended?

• Conduct verification through functional testing, regression testing, API testing, etc. that the product functions correctly

• Validation occurs continually throughout the agile development method

• Testing is conducted within the product team by all team members

Phase 5: Product Launch

Phase 5: Product Launch

Advana 43

Product launch is not just ensuring we have a technically functioning product. It includes revisiting the entire customer journey and experience. Do users know how to get to the product? Are customer communications in place to inform them about the new product? Have target market communication channels been notified? Are there user guides and training guides in place?

Phase 5: Communications and Training

Advana 44

• Build a communications plan for promoting the product to your target market to build awareness of and familiarity with your product and its capabilities- You'll also need to inform all the right internal teams; therefore, your launch communications plan will have external

(customer-facing) and internal elements

• Important elements of your communications plan:

• User guides and training must also be ready when you launch your product to:- Help users navigate to and through the product- Provide helpful tips for using the product to answer key business questions- Point customers to additional help resources if needed

WHO WHAT HOWCustomer personas within your target

market

Key messages (what you want them to know)

Communication channels (how you’ll get the right

message to the right people)

TIP: Communications is not a one-size-fits-all exercise. Carefully consider your specific

customer personas, the customer journeys you mapped (tells you WHEN they need

certain pieces of information), and how your customers consume information.

Phase 6: Product Optimization

Phase 6: Product Optimization

• Critical success factor is to create a secure, scalable, and standardized data pipeline that allows embedding data automation into each step of the data ingestion process from ingest, to processing and validation, to automated checks and refreshes

• Acquired data is programmatically validated against expected values and rejected if results are outside anticipated quality bounds

• Data quality reports are generated for each dataset, including validation at each step, profiling of every element, and signed audit documentation

Advana 46

During the Product Optimization phase, the product team heavily focuses on automation of data pipelines and any routine activities, defining key performance metrics to monitor in sustainment, working with the Advana Development and Engineering to implement telemetry (user activity tracking), and create feedback loops to gather customer insights.

Capture Process StoreBuild automated mechanisms and connections to

bring data

Run automatic processing via

Spark and StreamSets

Run automated data business rule check and compliance and store

results and data

Analyze UseMake data available to use within data, run data quality checks,

capture rejected results

Automatically refresh your dashboard

Phase 6: Monitoring and Improving Data Quality

Advana 47

Part of optimizing your data product, is to ensure that the end users have confidence in the data that is presented to them for decision-making. Product teams must automate the monitoring of data quality to ensure fidelity of product.

In order to drive data quality improvement leveraging the Advana platform, it is important we can…

• Ensure we are receiving the intended data provided in a dataset and that data isn’t missing

• Measure if we are receiving and publishing data in a timely fashion

• Make sure data, where appropriate, meets predetermined formats and rules

• Ensure data isn’t duplicated and appropriate labels and tags are routinely applied/followed

• Track any changes to datasets to ensure data hasn’t been inappropriately modified or broken

• Detect and identify erroneous data contained within datasets. (Example: a “name” listed in a “telephone” field or values outside of ranges)

47

CompletenessEnsure we are receiving the intended data provided in a dataset and that data isn’t missing

ConsistencyEnsure data isn’t duplicated and appropriate labels and tags are routinely applied/followed

ConformityMake sure data, where appropriate, meets predetermined formats and rules

AccuracyDetect and identify erroneous data contained within datasets. (Example: a “name” listed in a “telephone” field or values outside of ranges)

IntegrityTrack any changes to datasets to ensure data hasn’t been inappropriately modified or broken

TimelinessMeasure if we are receiving and publishing data in a timely fashion

• For a given group of similar data (e.g., financial accounting data), there are many systems from which data is extracted from. Each one of these systems have similar data types and elements (e.g., personal identification data), yet they use different naming convention for each column. This results in extreme inefficiencies and high level of effort when performing analytics as the end user would have to know the specific column convention for each system every time they wanted to query data across multiple data sets.

• After understanding the product’s data, product teams should work within their portfolio to understand which data elements may be useful to have mapped to the Advana domain common data model. - End users searching the catalog can review the common data models and

understand what data may be available across products without having to understand the potential name of every data column (i.e., SSN, User_SSN)

- Once elements are specified and agreed upon, the data SMEs will develop the logic and business rules for generating the content of these elements from the source data – see Data Mapping Document Process and Repositories. The developed logic for each system is then shared with the data providers to validate assumptions made in the business rule development and sheds light for the system owner on their data's downstream uses for their awareness and fidelity of data for our end users.

Phase 6: Supporting Advana Common Data Models

A common data model (CDM) is a set of well-defined and agreed upon elements/columns that are common across a group of similar data.

Common Data Model

Advana Product #1

Advana Product #2

Advana Product #3

Source #1

Source #2

Source #3

Advana 4848

Phase 7: Product Sustainment

Phase 7: Product Sustainment

Operations plan – Develop and institutionalize processes for:- Feeding knowledge gleaned from help tickets back into Phase 6 (Product Optimization)- Managing and tracking technical and communications elements of product changes - Identifying new desired features to feed into future product design efforts- Engineering to implement telemetry (user-metric tracking), logging, establish automated metric collection, and

create feedback loops to gather customer insights

Governance – Establish governance bodies to address:- Data strategy- Data quality- Infrastructure and security- Advanced modeling- Architecture – overall technology roadmap and integration

Performance monitoring- Leverage configuration management and monitoring tools that identify the data systems and feeds and are not

operating correctly and establish a daily, weekly, and monthly operational rhythm focused on adding structure around management situational awareness of the automated data management feeds

- Ensure integration with platform patches, software deployments, and other infrastructure-related activities that might impact your product’s performance

Advana 50

The Product Sustainment phase is where a team begins to shift into maintenance mode.

Thank you!