scaling r&d org while maintaining quality
DESCRIPTION
As a fast growing company Wix R&D doubles every year. In this talk I will describe how we structured our R&D division, what we are doing to build and keep an "A" team of developers and our dev centric and quality based culture that supports innovation.TRANSCRIPT
![Page 1: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/1.jpg)
Scaling R&D Organization While Maintaining Quality
Aviran MordoHead Of Back-End Engineering @ Wix
@aviranmhttp://www.linkedin.com/in/aviran
http://www.aviransplace.com
![Page 2: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/2.jpg)
Wix In Numbers
• 44M users from 190 countries
• Static storage is >800TB of data
• 3 Data centers + 2 Clouds (Google, Amazon)
• 700M HTTP requests/day
• ~600 people work at Wix
• Of which ~ 200 in R&D
![Page 3: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/3.jpg)
Double The People Every Year
From 15 to 200 in 3 yearsPlanned growth 400 -> 800 in the next 2 years
![Page 4: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/4.jpg)
Jan 2014
1250Deployments (production changes) per month
Double the velocity from last year
![Page 5: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/5.jpg)
Structures for Scalability
• There are 2 key common structures in the industry
• Functional unit model• Business unit model
![Page 6: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/6.jpg)
Functional Model
• Disadvantages• Lack of product ownership • Lack of product level expertise • Hard to predict and plan product
roadmap• Cross-function communication is
hard• Less focus on delivery and TTM
![Page 7: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/7.jpg)
Business Unit Model
• Disadvantages• Product integration is hard• Resource and work duplication• Architecture alignment is hard• Technology knowledge sharing is
hard• Limited opportunity for
professional development
![Page 8: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/8.jpg)
• There is no perfect model
• It depends on the company’s current challenges, life cycle phase and culture
• Every model should be tuned constantly and evolve with the company
Our Assumptions
![Page 9: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/9.jpg)
![Page 10: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/10.jpg)
• Guild – a group of people that share expertise, knowledge, tools, code, and practices
• Gang – a group of people that work in a related products• Compose of all required
resources from the different disciplines (Guild)
Wix’s Gangs & Guilds Model – How?
Guild
Gang
Guild
Gang Gang
![Page 11: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/11.jpg)
Wix’s Gangs & Guilds Model – Why?• Technical power of the Guild• Independence of the Gang• Healthy balance between product and tech• Product features and technical equal in priority
Guild LeaderWhat
How
Guild Leader
How
Team member
Team member
![Page 12: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/12.jpg)
How Does It Work
![Page 13: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/13.jpg)
Dev-Centric Culture
![Page 14: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/14.jpg)
Traditional Dev Pipeline
Product Dev QA Operations
23:50
![Page 15: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/15.jpg)
![Page 16: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/16.jpg)
Product Dev QA Operations
23:50
![Page 17: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/17.jpg)
What Is The Common Denominator?
• Product manager• Project manager• QA• Operations• DBA
Developers can do these jobs
![Page 18: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/18.jpg)
Dev Centric Culture – Involve The Developer – How?
• Product definition (with product) • Development (with architect)• Testing (with QA developers)• Deployment / Rollback(with operations) • Monitoring / BI (with BI team)• DevOps – to enable deployment and
rollback, fully automated
Developer
Product
QA
ManagementOperation
BI
![Page 19: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/19.jpg)
Because It Makes You ROQ
•Responsibility•Ownership•Quality
Dev Centric Culture – Why?
![Page 20: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/20.jpg)
Project Rotation – How?
• Team usually has more than one project• Team members rotate tasks between projects• Developers can switch teams and Gangs
![Page 21: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/21.jpg)
Project Rotation – Why?
• There is no single person with a knowledge on a specific component• Ongoing review• Keep it interesting• It is everyone's responsibility• Everybody owns everything
![Page 22: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/22.jpg)
![Page 23: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/23.jpg)
Quality Thursday – How?• A software engineer works 4 days with his Gang• Thursday is a Guild day.• Developers stop working on their designated
projects and conduct quality enhancing activities with the Guild.
![Page 24: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/24.jpg)
Quality Thursday – Why?• Assimilates people into the Guild• Builds cross-team relationships• Shares knowledge • Enhances the culture• Promotes innovation
![Page 25: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/25.jpg)
Bug Hunt – How?
• Service owner picks a service running on production• Explains the service to all Guild members• What the service does• Every exception thrown in production• Performance matrix
• Get a list of AI from group feedback (sometimes for dependent services)
![Page 26: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/26.jpg)
Bug Hunt - Why?
• Teach developers about services they don’t own• Understand how your application behaves on
production• Open discussion and ideas from group members• Discover unexpected use patterns• Find bugs on production
![Page 27: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/27.jpg)
Retrospective – How ?
• Whenever developers encounter issues, they are encouraged to post them on a board for public discussion• Anything can be discussed• Topics posted on the board during the week
constitute the agenda of the retrospective
![Page 28: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/28.jpg)
Retrospective – Why ?
• Solve problems • Share lesson learned.• Suggestions on how to improve:• Our team• Quality of products• Process • Effectiveness of the R&D organization.
![Page 29: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/29.jpg)
Tech Talks – How?
• Every developer can give a talk about anything• People from other departments in the company• Guest speakers• Open to anyone from Wix• Using Meetup http://www.meetup.com/at-wix/ to
invite outsiders to our internal talks (if appropriate)• Talks videos http://goo.gl/IDqXTi
![Page 30: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/30.jpg)
Tech Talks – Why ?
• Training new and old employees• Educating about other activities in the R&D• Educating about other departments activities and
concerns in the company
![Page 31: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/31.jpg)
Thursday Tasks – How ?
• Pay technical debt (preferably not your own)• Work on a task / feature NOT in your project• Read and learn something new• Open source a project• 1 on 1 mentoring
![Page 32: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/32.jpg)
Thursday Tasks – Why?
• Learn other projects (easier to step in if necessary)• Unbiased review of other projects• Learn about problems and solutions other teams
faced and solved that you may also encounter.• Find repeating patterns across projects and
generalize a solution• Find interesting projects to switch to.• Leave the code in a better shape than they got it
![Page 33: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/33.jpg)
![Page 34: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/34.jpg)
Test Driven Development – How?
• No new code is pushed to Git without being fully tested• We currently have around 10,000 automated tests
• Before fixing a bug first write a test to reproduce the bug
• Cover legacy (untested) systems with Integration tests
23:50
![Page 35: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/35.jpg)
Test Driven Development – Why?
• Developers are responsible for their own code
• Less bugs
• Easier to enter someone else’s project
![Page 36: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/36.jpg)
It Is All About The People
![Page 37: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/37.jpg)
Do Not Compromise On Hiring
• Hire only good people• Fit the culture• Excellent technically
• Candidates can be dropped• By anyone• At any time
• If there is any doubt, then there is no doubt
![Page 38: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/38.jpg)
Quality Trust Independence Growth
The Path To Growth
![Page 39: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/39.jpg)
Independence
![Page 40: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/40.jpg)
Dedicated Resources = Organization Structure
• Have dedicated teams for each product group• Avoid product priority collisions• A team should have all the resources it needs
![Page 41: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/41.jpg)
Minimize Architectural Dependencies
• Service oriented architecture (SOA)• Separation of client (UI) and server projects• Dedicated data stores for each service• Stateless services
Lesson learned: System architecture that decouples concerns
![Page 42: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/42.jpg)
Communication Channels• To company wide activities• To knowledge centers• To key personnel
![Page 43: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/43.jpg)
Quality Trust Independence Growth
![Page 44: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/44.jpg)
Lessons Learned: Growing Teams
![Page 45: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/45.jpg)
Seeds Of Ambassadors
• Train a group of ambassadors that practice dev-centric culture from the Guild• Build new teams with at least one dev-centric
ambassador to assimilate new teams into the culture.• Beware of hiring more people than you can train /
assimilate successfully
![Page 46: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/46.jpg)
Quality
TrustIndependence
Growth
It’s Not a Path, It’s a Cycle
![Page 47: Scaling r&d org while maintaining quality](https://reader038.vdocuments.us/reader038/viewer/2022110306/554f0aa0b4c90547648b53ff/html5/thumbnails/47.jpg)
Aviran MordoHead of Back-End Engineering @ Wix
@aviranm
linkedin.com/in/aviran
aviransplace.com
Q&A
Read some more about maintaining quality R&D: http://goo.gl/c3WLsz