Slide 1

Software Talks Are you listening? Julian Harty Nowhere near ready edition (26 Nov 2014) Slide 2 Value Why bother? Slide 3 www.eurostarconferences.com Are you working too hard, on the wrong things? We struggle to decide what to test, how much testing is enough, etc. 110% effort doesnt cut it www.eurostarconferences.com Slide 4 Most of our work is wasted effort How much of our testing is on target? Slide 5 www.eurostarconferences.com Most of our work is wasted effort How much of our testing is on target? Automated Tests 80% of reported bugs not addressed Slide 6 www.eurostarconferences.com Know your users Parallel Kingdom [2] Regular users generate 2.5x daily revenues Custom drink feature removed [1] => 1 star feedback ratings Logos respective owners [1] example from App Quality book [2] example from Tale of Two Apps Slide 7 www.eurostarconferences.com Understand the effects Battery drain varied by 3x for similar hardware specifications Custom code added for Kindle Fire to reduce brightness 40% less battery drain Higher network latencies reduced interactivity by 40% Users preferred Wi-Fi 69% for Parallel Kingdom, 58% for StudyBlue Tablets 2x usage Pull-out keyboard also increased usage Slide 8 www.eurostarconferences.com Analytics can augment our work Help us to correct and improve what we do Reduce waste, reduce latency, Increase value How Insights into the apps behaviour in-the-wild Feedback loops Slide 9 www.eurostarconferences.com Network profiling Volume Time (of day) Volume Time (of day) DiscoverUse Slide 10 www.eurostarconferences.com Network profiling Volume Time (of day) Volume Time (of day) Transform Slide 11 Raw ingredients / context Hows it all work anyway? Why do I care? Whats involved? Whats special about mobile apps? Slide 12 www.eurostarconferences.com One ecosystem Web Server Some history Web server logs & analysis Web App Web Logs App Logs W3C format Graphs Slide 13 www.eurostarconferences.com Whats different about mobile? Logs isolated on the device Connection not guaranteed Many more sensors Much more variation Mobile App App Logs Common System Log Slide 14 www.eurostarconferences.com TOPOLOGY Overview of Mobile Analytics Each step may be delayed Data Collector Database Analytics WebServer Business view Filter(s) Mobile Apps sending Analytics data Slide 15 www.eurostarconferences.com Types of Events Mobile app Analytics Library Analytics Collector 1:1 App-initiated m:1 App-initiated Library-initiated E1E1 E1E1 E E E4E4 E4E4 L L E E EaEa EaEa L L E E EaEa EaEa Analytics Database Internet connection Events Slide 16 www.eurostarconferences.com Analytical Questions: Past Trends, Defect Reports Trends, Defect Reports Software quality models, bottleneck analysis Whats Happened? (Reporting) How & why did it happen? (Factor analysis) How & why did it happen? (Factor analysis) Information Insight Slide 17 www.eurostarconferences.com Analytical Questions: Present Engineering Activity, Benchmarking, Testing Specification refinement, asset reallocation Whats Happened? (Alerts) Whats the best / worst that can happen?(Modeling / Simulation) Information Insight Slide 18 www.eurostarconferences.com Analytical Questions: Future Extrapolation Failure prediction models What will Happen? (Forecasting) Whats the best / worst that can happen?(Modeling / Simulation) Information Insight Slide 19 www.eurostarconferences.com Analytics for Software Development Engineering Activity, Benchmarking, Testing Trends, Defect Reports Trends, Defect Reports Extrapolation Software quality models, bottleneck analysis Software quality models, bottleneck analysis Specification refinement, asset reallocation Specification refinement, asset reallocation Failure prediction models Whats Happened? (Reporting) Whats Happened? (Alerts) What will Happen? (Forecasting) How and why did it happen? (Factor analysis) How and why did it happen? (Factor analysis) What is the next best action? (Recommendation) Whats the best / worst that can happen?(Modeling / Simulation) Information Insight Past Present Future http://research.microsoft.com/pubs/136974/foser-2010-buse.pdf Slide 20 www.eurostarconferences.com Implementation Flow Feasible Practical Useful Design events Try various libraries Propose mappings Implement events Field tests Evaluate libraries Commercial evaluations Minimalist Acceptable? Slide 21 Gaining confidence Accuracy & precision Slide 22 www.eurostarconferences.com Precision: repeatabilityAccuracy: on target Precision & accuracy https://en.wikipedia.org/wiki/Accuracy_and_precision Slide 23 www.eurostarconferences.com Add images here: precision And here: accuracy Precision & accuracy Slide 24 www.eurostarconferences.com Divergent answers increase doubt A tale of two three mobile analytics libraries (and what happens when bonuses are on the line) Where were the testers (part one)? Slide 25 The Dark Side Of Mobile Analytics Slide 26 www.eurostarconferences.com DO NO HARM TO A NEIGHBOUR Bad stuff happened; Location data collected Excessive traffic Where were the testers (part two)? http://www.vam.ac.uk/users/node/1777 A Study of Third-Party Tracking by Mobile Apps in the Wild ftp://ftp.cs.washington.edu/tr/2012/03/UW-CSE-12-03-01.PDF Slide 27 www.eurostarconferences.com Reducing precision to protect privacy 10km x 10km squares From: Capturing Mobile Experience in the Wild: A Tale of Two Apps Figure ACM Slide 28 www.eurostarconferences.com Beware the automation bias Automation bias occurs in decision-making because humans have a tendency to disregard or not search for contradictory information in light of a computer- generated solution that is accepted as correct and can be exacerbated in time critical domains. Automation Bias in Intelligent Time Critical Decision Support Systems http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.2634&rep=rep1&type=pdf When presented with an automated solution 40% of pilots reasoned less or none at all Slide 29 Necessary but not sufficient Listening is a means to an end Slide 30 www.eurostarconferences.com Complementary Feedback System logs Crash logs App Store ratings In-App feedback Speaking to humans. Oh, and software testing Polychrome, richer tones while avoiding cacophony Slide 31 www.eurostarconferences.com Developer Console (Google Play) Slide 32 www.eurostarconferences.com Instrument the Ecosystem ApplicationAnalytics Ecosystem Change Me! Slide 33 www.eurostarconferences.com Observe the Behaviours Application Ecosystem Keys & GesturesNetwork Traffic Slide 34 www.eurostarconferences.com Correlation? Causation? Appropriate? From: User Interaction-based Profiling System for Android Application Tuning Figure ACM Slide 35 Breaking-up is hard to do From one thing, to another Slide 36 www.eurostarconferences.com Divorce can be messy, even for software Changing the code is the easy part What about the data? And the systems & processes that rely on the data? Act in Haste, Repent at Leisure Slide 37 Two ears to listen Are you willing to try? Slide 38 www.eurostarconferences.com Willing to try? Become one with the data [Rob Lambert] Instrumenting their code, need to learn how to understand and use the data [Rob Lambert] Bringing DevOps to Mobile Apps? Internet-of-Things is coming [Andy Stanford-Clark] Be aware of what can go wrong [Isabel Evans] Slide 39 www.eurostarconferences.com Further reading [4]http://www.howtomeasureanything.com/ [1,2]http://wip.org/ Email me for academic references These books available at: [3]http://www.appqualitybook.com/ Slide 40 Now? Later: julianharty@gmail.com Q&A Now? Later: julianharty@gmail.com