T13  Test  Management  5/5/16  13:30  

Testing  at  Startup  Companies:  What,  When,  Where,  and  How  

Presented  by:  

Alice    Till-­‐Carty  

Lyst    

Brought  to  you  by:    

350  Corporate  Way,  Suite  400,  Orange  Park,  FL  32073    888-­‐-­‐-­‐268-­‐-­‐-­‐8770  ·∙·∙  904-­‐-­‐-­‐278-­‐-­‐-­‐0524  -­‐  info@techwell.com  -­‐  http://www.stareast.techwell.com/  

Alice    Till-­‐Carty  Lyst  

A  senior  QA  engineer  for  Lyst,  one  of  London's  fastest-growing  startups,  Alice  Till-Carty  landed  her  career  in  testing  sideways.  Having  studied  film  at  the  university,  Alice's  obsession  with  technology  and  problem  solving  gave  her  a  head  start  in  testing.  With  six  years  of  experience  across  two  countries,  four  companies,  and  more  than  one hundred  projects,  Alice  has  worn  many  different  testing  hats: consultant,  analyst,  engineer,  and  quality  evangelist.  She  considers  herself  a  fierce  advocate  for  the  end  user.  Alice's  determination  for  always  making  things  better  for  the  customer  has  given  her  a  unique  perspective  on  test  process  improvement.  Follow  her  on  Twitter  @aliceness.    

Alice Till-Carty (@aliceness)

what, when, where and how

or, how to show value when none is perceived

testing & startups:

1

• originally from New Zealand• background in manual testing, with a focus on end user & usability testing• worked in a number of small companies, mostly eCommerce-related• but you don’t really care about any of this, you’re here to learn about startups andtesting!

hi, I’m Alice

2

so, what is a startup?

3

• fashion aggregation website• continuous deployment• heavily data-based• half-tech, half-fashion. engineering/product team of 58, the remaining 69made up of marketing, comms, operations & sales

the startup

4

… and, what is value?

5

noun. the regard that something is held to deserve; the importance, worth, or usefulness of something.

A lot of companies are hiring for testers.

(and this was just when I looked at jobs in London!)

‘QA Engineer’

7

lyst is founded in 2010 first dedicated QA (QA lead) hired in 2013

second dedicated QA (me!) hired in march 2015 (total count: 2) third dedicated QA hired in may 2015 (total count: 3)

… third dedicated QA leaves in october 2015 (total count: 2) the second third dedicated (mobile) QA joins in december 2015 (total count: 3)

The originally hired QA (the QA lead, and my boss!) leaves in april 2016 (total count: 2)

1 tester 20 developers

then, 2 testers to 20 developers

then 25, then 30

now:

2 testers to 36 developers

the situation

8

to improve quality across the website and the iOS application to encourage teams to think about quality

to come up with and implement a process that addresses each individual team’s needs in the best way possible

to educate and improve developer’s testing skills

the challenge

9

there were a few!

challenges

10

always needing a justification for fixing a bug

attitude towards test

test as a separate team from the rest of engineering

responsibility of automation - who, and why?

working primarily with developers who had never worked with testers before

lack of communication between the business and engineering, and even between teams within

engineering

17

- Test, learn, adapt - one size fits all doesn’t work - but we tried it anyway! - a shift in responsibility: quality up front instead of at the end - finding ways to show ‘value’ - automation responsibility now belongs to the developers

process changes

18

- the one size fits all approach: didn’t work. - communication is the lifeblood of the development

process. - we definitely still hadn’t solved all of the problems.

So, what impact did these changes have?

- quality upfront is important as quality at the end - the smaller the change, the smaller the amount of testing that needs to

be done - encouraging a routine when it comes to QA/test helps developers to

think about quality regardless of how much (or how little) risk there is in a change

- get to know your developers - having a good working relationship with them will always make your job easier!

Lessons I wish I had known from the start

20

21

What do I do now?

then now

Test responsible for automation Developers responsible for UI automation tests

Primarily automated tests, with some manual testing

Extensive unit test coverage, a suite of automated

tests that are maintained and kept up to date, as well

as exploratory manual testing of all front-end features

Head of product + developers would decide on and

implement new features

All developers, tester, designers, and product owner

size and define new features together

No focus / monitoring of performance, security and

other non-functional issues like usability and UX

experience

Have a dedicated in-house UX Researcher,

tools/resources to measure performance and security

issues

Test in a separate team Test as an integrated part of the respective team they

are responsible for

now vs. then

22

Some things to remember!

an inevitable evil.

context switching

24

a test role at a startup is NOT going to be the same from one company to the next.

being flexible is important!

25

for startups, the most important thing is growth.

the relationship between startups & testing

26

A startup is a company designed to grow fast. Being newly founded does not in itself make a

company a startup. Nor is it necessary for a startup to work on technology, or take venture funding,

or have some sort of "exit." The only essential thing is growth. Everything else we associate with

startups follows from growth.

Paul Graham on startups

27

the most important thing is that everyone is on the same page about what QA is there to achieve.

so, what should test do at a startup?

28

we haven’t solved all the problems yet!

- grow the test team! - find out and define the role that automation plays within our code base.

is it a priority? Why? - continue to encourage a culture where everyone believes quality is

important.

the future

29

- iteration is important when improving process. A complete overhaul once every quarter will just confuse people.

- if you’re going to use a metric to define quality: it’s important to decide what you’re going to measure, AND start to measure it before you decide to use it as a metric. This is so you have a yardstick to define

whether the process has improved the metric or not. - having allies will make everything a lot easier! - use EVERY opportunity to have to show value.

lessons learned

30

Should startups have test?

value

32

thank you

questions?