2015 mastering sap tech - enterprise mobility - testing lessons learned
TRANSCRIPT
SPEAKER NAMEJob Title, Organisation
Presentation Title
Testing Enterprise MobilityEneko Jon BilbaoSAP Innovation & Emerging Technology, Accenture
• Accenture ANZ SAP Mobility Lead – also get to dabble in our Innovation Centre (gadgets!)
• Recently been working with a number of large clients on their mobility journeys.
• In particular:– Two recent major implementations 2013-2015
– including SAP Mobile Platform, Fiori, Click Mobile, Device roll out
– Mostly packaged Hybrid Applications, a few Native
• This story is based on my thoughts from the journey’s we’ve been on with our customers.
Introduction & Overview
2
It is no longer enough to have a large team of in-office testers who run through “the usual” tests (unit, integration, user acceptance and performance).
Consider broader testing to gain better confidence of the benefits from your investment in mobile applications.
Enterprise Mobile Solutions are different.
3
• Lessons Learned
• Mobile Applications are Different
• New(ish) Objectives for Testing
• Skills to Learn
• Tools & Techniques
• Key Takeaways
What I’ll Cover
4
Obligatory Wor(l)d Cloud
5
*Provided by Tagul.com
Lessons LearnedMobile Applications are DifferentNew(ish) Objectives for TestingSkills to LearnTools & TechniquesKey Takeaways
1. Development complete ≠ Go Live
2. Device emulators are not enough.
3. Network latency is generally a bigger factor on mobile application performance than bandwidth.
4. Make network and communication decisions early on in your mobility journey.
5. Design for security everywhere (encryption, SSO)
Lessons Learned
7
Mobile apps are differentNew(ish) Objectives for TestingSkills to LearnTools & TechniquesKey Takeaways
Lessons Learned
There are high expectations to meet!
Enterprise Mobile is different...
9
“everywhere and easy”versus
“only in the office and it’s OK…”
I can use it anywhere!It’s simple and easy to use so I won’t need training.
I can find my data easily.It’s fast!
User Experience doesn’t stop with User Interface design, it encompasses the whole experience of the mobilised business process.
• Locations (work anywhere)• Flexibility (work on anything)• Productivity (work fast)
These expectations drive new testing requirements
10
Anatomy of a Mobile Application:
There is much to test…
11
Your Backend Code /
Server & DB Performance
Generally well understood and
tested
Here be dragons…
External Services / End Points
Network(Latency,
Bandwidth)
Browser / Client Code
Device Feel / Response
Your Total Mobile
Experience
• A reliable and consistent wired network is difficult.
• A reliable and consistent wireless network whilst being mobile is near impossible.– Realise that your users will be moving!
– Network conditions will change, bad latency will occur and drop outs (whilst getting more infrequent) will happen.
– Be curious how your application behaves in these circumstances.
Add 100-200 milliseconds for 3G latency
Add 60-120 milliseconds for 4G latency
Diff 1: Networking Killed the Mobile Star
12
• Using APIs and others’ services
• Mash ups and extensions to standard SAP Apps are common user requirements.
• For example: show me my company’s Geospatial data, grab me some information from a document management system)– Remember to test these!
– Do the API owners know the extra users are coming? Have they agreed to tell you if something changes in their service?
– Are they optimised for Mobile?
– Can they be accessed from both the intranet and internet?
Diff 2: But I want to use yours!
13
Diff 3: Not all clients are created equal
14
• Chrome on iOS behaves differently to Chrome on Android(s) etc.:
– Security Certificates are handled differently
– JavaScript engines are different – what works in one might not work in the other.
– Different # of parallel TCP connections affects speed
– You may or may not get notification of errors! (they may only be in the console)
Diff 3: Not all clients are created equal
15
• Not all touch screens are the same.
• Factors that may make your users complain:
– Speed of response whilst tapping or dragging
– Accurate “touch points” – supporting fingers in gloves?
– Brightness / Glare of screen
– Ruggedness and how easily it breaks!
Diff 4: You’re not on a PC anymore Dorothy
16
New(ish) objectives for testingSkills to LearnTools & TechniquesKey Takeaways
Lessons LearnedMobile Applications are Different
Yay! Success!
Testing the new Mobile Expectations
18
Traditional Testing, PLUS:
• Test the varying network impacts are acceptable
• Test our multitude of devices work & perform
• Test app desirability and thus our productivity
Work on Anything
Work Anywhere
Work Fast
Consider the following questions:
1. How do your applications perform in slow, medium, fast network conditions? What is the likely responsiveness?
2. How do your applications perform when network conditions change? Do they fail gracefully?
3. Will your applications be offline for periods of time without connectivity? Can users transact without a network connection?
4. Where will your applications physically be used the most? City (basements?), Rural (cell towers?)
Working Anywhere – Network Behaviour Testing
19
Consider the following questions:1. What devices will your users be on? How do they
differ?
2. What screen real-estate will they have? What will be the main mode of interaction be? E.g. Touch, pen, mouse & keyboard?
3. What is the battery life expectation for your devices? How long will your users be away from power?
4. Are your devices likely to get dirty? Be dropped? Need to work in harsh sunlight?
5. Will hybrid applications deploy the same way on all your devices?
Work on Anything – Device Testing
20
Consider the following questions:
• What do your users find difficult? What don’t they like?
• How many interactions does it take to do what your users need to do?
• Have we minimised the inputs our users need to make?
• What are the impediments to users completing their activities quickly?
Work Fast – Productivity Testing
21
As soon as you can! Generally before User Acceptance Testing.
When to plan your new tests?
22
User Acceptance Testing
Performance Testing
Integration TestingUnit Testing
New Effort
New Test Aims
2-5 days per device
Basic Device Functions:
• Baseline Battery
• Screen Brightness
• Ruggedness
• Touch responsiveness
• Cases/Peripherals?
2-3 days per App
Productivity Testing:
• Use Prototypes if available
• # of Clicks to fulfill
• # of Data Entry Fields
• What is difficult?
• What would make you love it?
2-3 Weeks
Network Behaviour Tests
• Best case performance
• Worst case performance
• “Real world” performance
• Network failure / offline behaviour
Traditional Phases
Iterate and feedback to developers
Skills to Learn
Tools & TechniquesKey Takeaways
Lessons LearnedMobile Applications are DifferentNew(ish) Objectives for Testing
Typical SAP Mobility Communication Paths
24
• SAP is getting more complicated.
• Test all your major service interactions both over external cellular and internal Wi-Fi (the arrows on this diagram)
– Connection Tests (firewalls)– Load Tests (performance)– Session Stickiness (moving locations)– TLS/SSL (security)– How does your code perform in these
scenarios?
• Developers, you’re going to need to better understand the Technology Architectures and Operational landscape your app will be deployed in. Understand your application END to END.
• Technology Operators, you’re going to need to get your hands dirty with some debugging (particularly in browsers) now and then.
Successful Mobility Testing involves DevOps skills
25
Don’t let this be you…
26
Successful Testing
27
Testing / Quality Assurance
DevelopersIT
Operations
FunctionalTesting
Technology Testing
MobilitySuccess
Good Design
What new Skills?
28
Developers Technology Operators / Architects
Familiarise yourself with your network design, understand where the services you are calling are hosted.
Understand what web services the developers are going to call, where they reside, what they do, and how many times they will call them.
Understand how variable latency affects the user experience you’re creating. Build with this in mind always.
Work towards minimising latency for time sensitive application purposes. Network acceleration / caching specific application patterns.
Understand communication security - Website Certificates, TLS/SSL, Cross Site Scripting problems you may encounter due to sharing resources
Provide developers with root certificates to establish trust, know what must be encrypted and where.
Understand the load balancing design and any Virtual IPs / Server Names and how they proxy your calls in case they adversely affect your application.
Design for BOTH intranet and internet enabled services and allow applications to have sessions cross intranet/internet boundaries. Don’t use different domains!
Understand browser limitations, differences in device behaviours - developing in your local IDE is not enough.
Familiarise yourself with JavaScript and browser development tools, provide developers with physical devices.
Tools and Techniques
Lessons LearnedMobile apps are DifferentNew(ish) Objectives for TestingSkills to Learn
Key Takeaways
You’re going to need a Lab.
• All mobile device types
• A network access point you control
• New tools/software
So what do I need?
30
Essential Mobility Testing Tools for Everyone
31
HTTP Debugging / Proxy
Text Editor Screen Capture Message / Web service testing
Why?Understand what’s happening “under the covers” of your apps.
Format Messages, HTML, XML, JSON etc.. Make it easier to read and understand.
Share problems with other teams, a picture is worth a thousand words.
Replay, investigate, debug discrete web services. Load test services.
Options
Wireshark Sublime Text Windows Snipping SOAPUI
Charles Proxy Notepad++ Snag IT JMeter
Fiddler TextMate LICEcap POSTMAN
(Browser Developer Tools – Chrome, IE,
Firefox)Komodo WinSnap
HTTP Watch VIM
• Testing your main Physical Devices is a must.
– Give devices to your developers and your normal test users.
– Emulators are OK for development, but many user experience problems will only manifest in the physical device. For example:
• Touch screen “hot spot” inaccuracy
• Resource loading in a wireless / high latency environment
• Bugs in app packaging / UI frameworks (Apache Cordova, JQuery etc.)
• Proxy configurations
• Certificate trust / handling TLS and encrypted traffic
• Not pixel-perfect
Device Testing Overview
32
• Recommendations for working through Device-specific problems:
– Run a Proxy to view the actual traffic coming and going to your device.
– If it’s a hybrid app (Fiori/UI5) check it in the native device browser as well as the Cordova package / app format.
Device Testing in Your Lab
33
• Do some real-world, manual testing on top of the normal test phases:– Run through a “10 minute day-in-the-life” for your
users • Key everyday functions (e.g. Report some graffiti)• Use all the native device functionality that they will
(e.g. Camera, GPS)
– Measure:• Battery drain (Enable % remaining mode and note it
down)• Ease of use – do you find yourself multiple tapping?
Do you miss touch hotspots?• Responsiveness (stop watch!) – confirm this with
automated testing later.
– Iterate and average-out your findings to discover problem areas
Device Testing in the Field
34
• Know your Application Service End Points – understand what your application talks to and when – are these end points available both on your internal network and over cellular internet?
• Understand Quirks of different Device behaviour before your users report problems.
• Understand likely battery life for your organisational use cases.
Device Testing Outcomes
35
How will my app perform in changed network conditions?
Network Testing in Your Lab
36
Options Pros Cons
In-Browser Tools(e.g. Chrome Development Mode)
• Free• Generally only a simple
bandwidth throttle• Chrome recently added in
Throttling based on average network characteristics
• Hybrid/Web Only• Physical Device may behave differently.• No Latency control• Cannot throttle non browser traffic
Network Emulation,Proxies
• Test from Device• Can change network during
transactions• Control latency, bandwidth AND
packet loss• All Device traffic
• Complex Set-up• Licensing cost depending on tool
Chrome Development Tools
37
• Try it yourself on any web app.
• Just hit CTRL+SHIFT+I in Chrome
• Then Click the device icon.
Set up your Network Virtualization on your Lab Access Point:• Test your key transactions.
Aim to test varying scenarios:• App Behaviour – Pass/Fail:
– Disconnected mid-way through a transaction (unplanned offline)– Coming back online whilst being offline for a period of time
• App Performance – Response Time:– Slowest Speed (3G Poor)– Average Speed (4G Normal)– Best Speed (Office Wi-Fi? Depends!)
Network virtualization options:• We have used Shunra (now HP) in the past – they provided global network profiles, ability to record
your own network conditions, replay transactions.• Alternatives: Charles (for single apps), Packetstorm, Dummynet.
Network Emulation in Your Lab
38
- Examples extrapolated from multiple tests and profiling of network conditions in Melbourne city and Accenture office.- Benchmark your own network data!
Example Network Conditions
39
Network
Emulated
Latency (ms) Data Rate
(Down/Up Kbps)
Packet Loss
Wi-Fi 35 13,000/9,000 0%
3G Good 104 2,500/750 0%
3G Fair 125 1,590/400 1%
3G Poor 160 700/100 1.5%
• Check your network coverage map!– Telstra, Optus, Vodafone all
publish fairly granular maps.
– Should I expect 4G Performance?
• (Optional) Actual location sampling – Provides for strange locations
(like a particular known blackspot)
– Assures business users that you’ve been where they will be.
Network Testing in The Field
40
Network Testing Outcomes - Example
41
Key Takeaways
Lessons LearnedMobile Applications are DifferentNew(ish) Objectives for TestingSkills to LearnTools and techniques
• Be curious about your application! - Testing Mobility applications is different, there are a broader scope of factors to cover than traditional enterprise roll outs.
• Set yourself up a Lab, and grow a mobile testing capability. This will only become more important as “Mobile” grows to encompass the Internet of Things and becomes more than phones and tablets.
• Know as much about your user as possible, and test out their Total User Experience before they do. Ensure they are delighted, and you will get the highest ROI from your implementation.
Mobility Testing - Key Takeaways
43
• Smartphone Penetration Australia http://www.statista.com/statistics/257041/smartphone-user-penetration-in-australia/
• Mobile Wireless Performance Solutions and Problems http://www.gartner.com/document/2681417
• Internet Backbone Latency https://www.dotcom-tools.com/internet-backbone-latency.aspx
• ENIAC computer - http://en.wikipedia.org/wiki/Computer
• Telstra Network Coverage Map: https://www.telstra.com.au/coverage-networks/our-coverage
• Shunra User Guide: http://media.shunra.com/datasheets/ (recently replaced with HP documentation)
• Word Clouds from http://tagul.com
• Charles Proxy can be found at http://www.charlesproxy.com/ (no affiliation)
• Device Fragmentation http://www.businessinsider.com.au/android-fragmentation-2014-8
References
46