Produced by Wellesley Information Services, LLC, publisher of SAPinsider. © 2016 Wellesley Information Services. All rights reserved.

Test for Success! A deep dive into effective SAP BusinessObjects BI 4 test strategies

Alan Mayer Solid Ground Technologies, Inc.

1

In This Session

• Understand the role that testing plays in BI 4.1 environments

• Learn an overall model for testing

• Witness individual techniques via live demonstrations

• Gauge the effectiveness of each technique

• Create your own test plan from the strategies presented

2

What We’ll Cover

• Introduction

• Testing at the atomic level

• Comparing reports

• Verifying workflows

• Reconstructing scenarios

• Stressing systems

• Promotion strategies

• Wrap-up

3

Why Test?

• Catch behavior that software vendors may not have considered

• Identify and fix badly behaving reports / visualizations

• Guarantee to users reliable access to their content

• Verify that “advertised” changes are the only ones …

• Predict how your system will react under load

4

When to Test

• Before migrating to a new release

XI 3.1 to BI 4.x (Major)

BI 4.1 to BI 4.2 (Minor)

• Before applying a service pack or patch

• Before promoting content from between environments

Dev to Test

Test to Production

• Prior to changing any system that our BI systems

depend upon

Databases

Networks

5

Our Testing Model

• We’ll present several testing techniques in this session

• Each provides a measurable benefit based on your testing needs

• Knowing when to use the test is crucial

Look for the following labels in the upper right hand corner of each slide

• After presenting each technique, we’ll revisit the tests needed per event

Major / Minor

SP / Patch

Lifecycle

Other

6

When Time and Skills Matter

• Some companies opt out of testing for various reasons:

Takes too long

Don’t have the right number of people who know the technology

• We’ll help soften those excuses:

Each testing method will have an accelerated version

We’ll also post the skill level needed to perform the test

Skill levels rated from 1 – 10

Based on the application being tested

(Webi, IDT, …)

7

What We’ll Cover

• Introduction

• Testing at the atomic level

• Comparing reports

• Verifying workflows

• Reconstructing scenarios

• Stressing systems

• Promotion strategies

• Wrap-up

8

Atomic Level Testing Defined

• For each application used within your company:

Check every toolbar button

Check every menu level command

• Typical BI 4.1 applications include

Web Intelligence

Crystal Reports

Universe Designer

Information Design Tool

Major / Minor

9

Atomic Testing – Why?

• This seems to be a bit extreme

Surely a software vendor has validated their applications to this level

• The answer to the question depends on the application … and YOU

Many defects were found in BI 4.0 using this simple technique

Several were still found in BI 4.1

It also tests your environment and setup

Can you launch the Java applet required for editing?

Can you export to a shared filestore that all users

should have access to?

Can you send email notifications?

10

Atomic Testing – Documenting your Results

• Create an Excel workbook to capture the results

Add the menu-level command hierarchies

Add buttons by toolbar

This particular test revealed several interface

behaviors added “by design” that had to be

reworked

11

Atomic Testing – Accelerated

• If you don’t have time to test every command and option:

Test the most widely used buttons and menus

Use your workflows as a guide

Test the buttons / commands needed to execute the workflow

Ignore others for the sake of time

Accelerated: 1 week

Normal: 2 weeks

12

Atomic Testing – Skill Level

Execute and record the results of

invoking a toolbar button or command

Should have used the software before

to know what to expect

2

Create Workbook

4

Conduct Atomic Tests

Create the workbook of toolbar

buttons and menu commands

Have to know what is available

from the application

Reference guides help here

Skill levels range from 1 (Green – easiest)

to 10 (Red – most advanced)

13

Demonstration – Atomic Testing

• Compiling and executing

the atomic tests

14

What We’ll Cover

• Introduction

• Testing at the atomic level

• Comparing reports

• Verifying workflows

• Reconstructing scenarios

• Stressing systems

• Promotion strategies

• Wrap-up

15

Report Comparison - Defined

• Refresh the report in both versions at the same time

Scheduling the refresh to process more reports at a time

• Compare the report contents in each version

Data

Format

• Identify and resolve differences

May involve reworking the report

Defect-related differences should be resolved

by the vendor

Major / Minor

SP / Patch

16

Report Comparison – Why?

• Differences in versions can affect report content

Calculation engine changes can easily be caught

• If this testing step is skipped, users might catch the difference

Usually occurs after go-live

Lowers confidence in your newly upgraded software

17

Report Comparison – Manual Inspection

• Compile a list of inspection points that must match between versions

Queries

Headers / footers

Tables

Graphs

Fonts

Alignment

Sections

Number of pages

…

18

Report Comparison – Accelerated

• Focus on the mission-critical reports first

• Third party tools can greatly accelerate the spotting of problems

Schedule reports to both Excel and PDF formats

Use an Excel comparison tool to spot any data-related problems

Us a PDF comparison tool to spot format-related issues

These tools may not accelerate the resolution of problem reports

They will separate the good one from the bad, however

Accelerated: < 1 week

Normal: 4 weeks

19

Report Compare – Skill Level

Skill level drops if

additional tools used

2

Collect critical content Compare

Should know what is important

Made easier by looking for

critical event-driven schedules

5 2

Manual Automated

Discover what is wrong

Correct content -or-

contact vendor

Diagnose

7

20

Report Comparison – Additional Notes 1

• DO NOT use the Administrator account for testing

This is a common (but bad) practice at many companies

Avoids the time needed to apply proper security

Administrator accounts will not correctly test all environment features

Example: Universe restrictions

Not applied when using the Administrator account

Not applied to members of the Administrators group

• Gather a list of typical users by role for each business area

• Use their accounts for more realistic testing

21

Report Comparison – Additional Notes 2

• DO use production data to test production reports

Testing production reports against test data has its own issues

Dev and test databases may not be able to refresh production reports

Query conditions within those reports must be rewritten to reflect

the available data

The final result may not match reality

• A production break-fix environment comes in very handy here

A production environment that has access to the same data users have

Not widely available to all users

Usually just administrators

22

• Comparing data and format between reports

Demonstration – Report Comparisons

23

What We’ll Cover

• Introduction

• Testing at the atomic level

• Comparing reports

• Verifying workflows

• Reconstructing scenarios

• Stressing systems

• Promotion strategies

• Wrap-up

24

Workflow Definition

• Test the common actions a user conducts with this application

Log in / out

Create a document

Create a data provider (query)

Refresh all data providers

Schedule a report

Add breaks to a report

Fold / unfold the breaks

Format a report (shading / fonts)

Create a document link

…

Major / Minor

SP / Patch

Other

25

Workflows – Why?

• Workflows test series of actions

Unlike atomic testing which tests an individual action

Unlike report comparisons which test overall data / formatting accuracy

• Workflow testing will uncover other problems

• Particularly suited for quick tests

All workflows can usually be tested within

an hour or two

26

Workflows – Accelerated

• Testing workflows is fairly quick as it is

• Narrow the workflows even further to the most popular for your company

Don’t test element linking between blocks if this feature is not used

Accelerated: < 2 hours

Normal: 1 day

27

Workflows – Skill Level

Tester should have the skills of an

average users for that application

5

Document Workflows

5

Execute Workflows

28

Workflows – Additional Notes

• The same notes mentioned for report comparisons apply to workflows

• DO NOT use the Administrator account for testing

• Gather a list of typical users by role for each business area

29

• Documenting and testing workflows

Demonstration – Workflows

30

What We’ll Cover

• Introduction

• Testing at the atomic level

• Comparing reports

• Verifying workflows

• Reconstructing scenarios

• Stressing systems

• Promotion strategies

• Wrap-up

31

Scenario Testing - Definition

• Rebuilding a complicated document or visualization from scratch

• The goal is to document any trouble spots while building

It is NOT to create a picture-perfect duplicate report

Major / Minor

SP / Patch

32

Scenario Testing – Why?

• Trying to reconstruct a finished document will uncover

other problems

We are providing a comprehensive testing strategy

but are still human

Might have missed an important workflow that will be

uncovered by this step

33

Scenario Testing – Accelerated

• Try this testing step with at least ONE complicated report

• Do not skimp on the level of complexity

That is the entire reason this step was included

Accelerated: < 1 week

Normal: 2 weeks

34

Scenario Testing – Skill Level

Recreate a complicated report or

visualization from scratch

9

Recreate Content

35

• Constructing a complicated report from scratch

Demonstration – Scenarios

36

What We’ll Cover

• Introduction

• Testing at the atomic level

• Comparing reports

• Verifying workflows

• Reconstructing scenarios

• Stressing systems

• Promotion strategies

• Wrap-up

37

Stress Testing - Definition

• Simulate a very busy processing day for your BI system

This is done by subjecting to high levels of activity

Adhoc queries and reports

Schedules

Publications

Visualizations

• Measure how well your system responds under load

• Use those results to reconfigure the system and retest

• Use free performance testing tools like jmeter to

configure and run these simulations

Major / Minor

SP / Patch

38

Stress Testing – Why?

• Stress testing allows you to see how your system will respond under high load

• This event will occur whether you conduct this test or not

It’s called GOING LIVE

39

• Rather than trying to simulate a complete BI environment, focus on a

simple workflow

Login and refresh a report

• Script that stress test and simulate hundreds of users running it

• Batches of schedules / publications can be created and launched via custom events

This will simulate high periods of scheduled activity

Accelerated: 2 – 4 days

Normal: 2 weeks

Stress Testing – Accelerated

Accelerated: 2 – 4 days

Normal: 2 weeks

40

Stress Testing – Skill Level

Batch schedules and trigger by custom

event

9

Create Volume Test Script

6

Create Scheduling Payloads

Simulates high levels of

concurrent activity

41

• Create and run a simple stress test

Demonstration – Stress Test

42

What We’ll Cover

• Introduction

• Testing at the atomic level

• Comparing reports

• Verifying workflows

• Reconstructing scenarios

• Stressing systems

• Promotion strategies

• Wrap-up

43

Promotion Testing - Definition

• Test changes to reports, universes, and visualizations before

• promoting them

Typical promotions include:

Dev to Test

Test to Production

• Know what changes were made in the current

environment (Dev, Test)

• Compare that against the current copy in the

next higher level (Test, Prod)

Lifecycle

44

Promotion Testing – Why?

• Without evaluating changes to reports and universes, other changes may

slip through

• These may or may not be intentional

• Those additional changes can have unintended consequences

• Other great reasons besides validating the number of changes

Acts as a kind of “code review”

Evaluate changes:

For reports:

Does the report refresh / calculate correctly with those changes in place?

For universes:

Is the correct SQL being generated?

Are query contexts still being generated correctly?

45

Promotion Testing – Accelerated

• Metadata tools can help identify which changes were made

SAP Information Steward

Other third party metadata tools

• Need the “deep scan” of the report or universe in question

Deep metadata scans will dissect the object into

component pieces

Reports: Formulas, variables, tables,

charts, …

Universes: Folders, objects, tables, joins,

contexts, …

• Once deep scanned, the component pieces can be

verified before / after the change Accelerated: 1 hour

Normal: 1 day

46

Promotion Testing – Skill Level

Provide sample reports that

can be run before / after the

change

Report comparison

techniques can be used to

evaluate the results

2

Identify changes Compare sample content

List all components that

have changed

5 7

Manual Automated

47

• Test a universe-based promotion

Demonstration – Promotions

48

What We’ll Cover

• Introduction

• Testing at the atomic level

• Comparing reports

• Verifying workflows

• Reconstructing scenarios

• Stressing systems

• Promotion strategies

• Wrap-up

49

Testing by Activity

• Here’s the summarized list of tests that should be run by activity:

Major / Minor Releases SP / Patch Lifecycle Other (Network / DB)

• Atomic

• Report Comparison

• Workflow

• Scenario

• Stress

• Report Comparison

• Workflow

• Scenario

• Stress

• Promotion • Workflow

50

Where to Find More Information

• Alan Mayer, “SAP BusinessObjects Web Intelligence 4.x Calculation Engine Changes”,

(Reporting and Analytics INTERACTIVE, November 2015)

• Carly Thomas, “Calculation Engine Changes for Web Intelligence BI4 and XI31”, (SCN

http://tinyurl.com/lq5g7vw)

• Brown and Rapp, “Crank Your BI Performance up to 11 – Sizing, Tuning, & Performance

Testing”, (ASUG SABOUG 2015, September 2015)

• Brown, Mayer, and NS, “BI 4.1 Admin Myths and Legends”, (ASUG SABOUG 2015,

September 2015)

• James Rapp, “Creating Your First Test Plan”, (SCN, http://tinyurl.com/gnvn7lc)

• James Rapp, “Performance Testing in BI 4.1”, (SCN http://tinyurl.com/h48ya8l)

51

7 Key Points to Take Home

• Realize the importance testing plays to maintain or upgrade BI 4.1 systems

• Understand the various testing techniques available

• Justify the use of each technique and the ramifications if testing is avoided

• Know when to test

• Use accelerated techniques when necessary to reduce overall time involved

• Involve the right personnel based on skill level needed

• Create personalized test plans based on your own requirements

52

Your Turn!

How to contact me:

Alan Mayer

Email: alan.mayer@solidgrounded.com

Twitter: @solidgrounded

Please remember to complete your session evaluation

53

SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other

countries. All other product and service names mentioned are the trademarks of their respective companies. Wellesley Information Services is neither owned nor controlled by SAP SE.

Disclaimer