paris qa meetup #6 kevin roulleau
TRANSCRIPT
Zenly
Testing analytics
Let’s start with a bit of context
Webedia Mobile department, 30 ppl, provides native apps for all the brands (Allociné, JeuxVideo, 750g, Purepeople ..)
Each app uses several analytics SDK (GA, Loca, Krux) and advertisement SDK (SmartAddServer, Yahoo, Facebook, Mopub ..)
Apps are rather big in terms of number of screens and features
Analytics implementation is causing pain and time loss for many people
Estimated 7d on dev side150 * 2 * 2 = 600 tags to implement and verifyAbout 20-25% of tests were marked as fail on 1st verification
This means following impact on ressources1.5 man-days implementing buggy code3 man-days spent detecting errors2 to 3 man-days to fix and verify
Google Analytics full implementation for one of the major apps
Furthermore, product managers do not trust their analytics and often ask for investigations
Product managers do not have ways to verify analytics implementation in the appWhen data looks bad or inconsistent, they ask QA / Project managers / Devs to investigatePast fails made them not fully trustworthy of the data
To tackle this problem, we developed a new tool « analytics fwd »
We chose this solution because it simply does a much better job than the previous ones
Back Office Proxy Analytics FWD
+ Easily accessible to project & product managers
+ Good precision. You see exactly what’s being sent or not
+ Easily accessible to everyone + Good precision
+ No delay + Clear, easy to understand display
+ Opens up a test automation opportunity
- Very inaccurate. You do some actions in the app and hope to see them appear in
the BO. It can take up to 24h.
- Setup hard, fastidious and error prone - We don’t know when the SDK will send
the data. There is often a delay - 5 to 10 clics required to extract the data - Data is hard to read, especially for non
tech people
- Only available on dev builds- Uses code that we don’t ship
- Doesn’t offer a guarantee that data really ends up in the BO
More importantly it copes with the two main aspects of this problem
Human aspect Technical aspect
Project and product managers do not have the time and skills to go deep and read logs in the console, or intercept in a proxy
We don’t have control over the various analytics providers and don’t have easy ways to stream back the data from a set of specific devices
And opens up a great automation opportunity
Implementation
Stack
Techno: nodejsFrameworks & libs
expresssocket.ioreact
Other toolsnpmapidocPM2 / keymetrics
Express.js - Routes
Express.js - Routes - Middlewares
Front end
Serving files is easily done in express
Front end Structure
Front end Structure
Front end Structure
Front end Structure
Front end
index.html
Front end
main.jsx
Front end
main.jsx
Front end
css: FlexBox
Tools - Apidoc Doc generated from comments in code
Tools - Apidoc Doc generated from comments in code
Tools - NPM scripts
PM2 / keymetrics
Deploy in prod with 1 command line: « pm2 deploy prod »Full monitoring dashboard with keymetricsCustom probes to enhance monitoring
PM2 / keymetricsecosystem.json
Deploy in prod with 1 command line: « pm2 deploy prod »Full monitoring dashboard with keymetricsCustom probes to enhance monitoring
PM2 / keymetrics Deploy in prod with 1 command line: « pm2 deploy prod »Full monitoring dashboard with keymetricsCustom probes to enhance monitoring
PM2 / keymetrics Deploy in prod with 1 command line: « pm2 deploy prod »Full monitoring dashboard with keymetricsCustom probes to enhance monitoring
Thanks
Demo + questions