Yehuda Miller, QA Engineer

Test Automation in a Continuous Deployment Environment

• Got into QA by accident…just like everyone else

• Worked on many different projects• I’m an avid reader (28 books this year)• Moved to Israel July 1, 2014• I’m colorblind (a challenge for a tester)• My favorite color is teal

Hello, who are you?

“How long does it take you to get one line of code into production?” - Wisdom of the Internet ;)

Where we were• All manual QA

• Repetitive tasks• Time consuming

• Two service packs a week• Holding back development• SPC size limits error handling

• Number of tests • Desktop: 587 (469) (493 IE)• Mobile-web: 69 phone, 23 tablet (0)

• Runtime of full suite: 37 min (51 min)• RND size

• Number of testers: 12• Developers: 50

• Grids: 4 (1)• Tests run on prod and semi staging env.• Continuous Deployment

Where we are today:

• We moved to CD in the last 8 months• On average we distribute code to

production 30 times per day.• Comes with challenges• Automated test on every code release• Only minimal tests on each release• Dev is responsible to run appropriate tests

PRE-RELEASE in staging

The CD mindset

• Risk reduction to production• Small incremental changes are easier to

monitor and revert if necessary• Has lower impact on the system

• Increasing R&D velocity• Avoiding wasted time on merges and complex

coordination before dist.• Allow R&D and product to experiment and

innovate more frequently.

RND Goals

• Provide a safety net• Avoid the bottleneck of manual sanity

testing• Reliability

• No flaky tests• No false failures

• QA E2E test should not be the new bottleneck

QA Goals

• QA are responsible for E2E tests only.• Dev write and maintain Unit tests• E2E tests are written during development

process • All QA members monitor CD suite results.• CD E2E tests are NOT acceptance tests

Responsibilities

• CD flow from commit to prod is ~25 minutes

• Current QA CD suite takes 1.5 minute in best case scenario.

• Runs in parallel to some of the jobs in the CD flow• Limited by time frame of parallel jobs• Still have room for further growth

Timing

• Minimal suite – No full regression on commit

• Occasional false failures due to env/site performance/human error

Risks

“A challenge only becomes an obstacle when you bow to it.” - Ray A. Davis

• Reliability• Velocity of tests• Pinpointing the failure• CD halt on failure.• Large scale tests refactoring due to

changes.

Challenges of CD

• Maintaining production data integrity• Reliance on pre-existing data• Unable to run on local staging env• Fast adaptation to feature flags• Identifying failure reasons• Need a true BDD/TDD mindset

Challenges QA faces

• Testing on more browsers• Edge• Firefox• Safari?

• Suites for our native apps (in progress)• iOS• Android

• Upgrading the infra• Gems, grids, Ruby…

• Adapt the tests for feature flags• Turn tests on/off based on prod flags• Test “flagged-off” features

The road ahead…

Questions?

Come join us for our next meetup!Topic:

Continuous Deployment Applied @MyHeritage

When: Tuesday, December 1, 2015 at 17:00

Where: MyHeritage office, 3 Ariel Sharon 4th

floor, Or Yehuda

Shameless Plug!

AA Milne

Thanks!We are hiring