1. What Do You Need to Know About Creating and Running a Scalable Web Site but Were Afraid to Ask?Session Code: S304347 Chris Webster chris.webster@sun.com Girish Balachandran girish.balachandran@sun.com

2. Learn some of the techniques weused to create and buildzembly.com2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 2 3. AgendaIntroduction Development process Architecture & deployment Availability & Performance 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 3 4. AgendaIntroduction Development process Architecture & deployment Availability & Performance 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 4 5. Introduction - zembly.com Browser based social application development environment (wiki for applications) Content Management System for executing code Executable API repository many outgoing HTTP web API calls 15% writes vs. 85% readsAudience App Developers zembly App User variable (one app has 30K+) API Provider Major players (Flickr, Twitter) but extensible and expanding Concept > Stealth > Private Beta > Open Beta launch only constant is change2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 5 6. Introduction software supporting zemblyDevelopment Tools NetBeans Subversion kenai.com Hudson Automated Testing (JUnit, Selinium)Infrastructure Solaris Glassfish (Java EE 5, JPA) MySQL + Enterprise Monitor Apache Memcached AdventNET Application Manager zenoss 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 6 7. Introduction we are zemblyEngineering Team Chris Platform Girish user interaction Site engineering - Girish Chris monitoring, performance, DBA, etc. Operations data center operators - external system admin - external2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 7 8. AgendaIntroduction Development process Architecture & deployment Availability & Performance 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 8 9. Development process - overview Agile development - 3 week sprintsEveryone can nominate work for the upcoming sprintPlanning meeting selects sprint deliverablesEngineers complete workBuild vettingWeekly production deployment2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 9 10. Development process - planning Strategic directionEngineering discussions before nominationsTasks must have Clear description, time estimate, and dependencies Account for demos, blogging, bugs, testsNo features added during planningLoad balancing for peopleRetrospectives are important 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 10 11. Development process - code Pair programmingContinuous Build compiler errors unit test failures JSLint for java script errors archive successful builds2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 11 12. Development process testsDeploy successful build for functional and automated UI tests functional tests (blackbox testing) UI tests: JUnit and SeleniumNightly build code coverage FindBugsNightly performance tests Repeatability needs historical data to detect trends model production traffic2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 12 13. Development Process code lines3 code lines trunk must pass all automated tests staging created from specific trunk version once a week. Deployed in clone of production exercise it for a week blockers fixed asap hackathon one hour holistic testing week keeps extending till this codelineis perfected. production vetted production builds future P1 fixes go to production branch and pushed to production fast 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 13 14. Development process - experience No nominationsNominations without detailsGrowing backlogLooooooonnnnnngggg planning meetingsAdding work in the middle of the sprintNo accounting for bug fixing, demos, tests ...Regressions where we don't have testsUrgent fixes needed in productionUsers finding problems before monitoring 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 14 15. AgendaIntroduction Development process Architecture & deployment Availability & Performance 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 15 16. w ik ET ze i /bN m loy.bl g/ bly.fomCru zeOm MLoadBalancer zembly.COM zembly.NET forum Static Static wiki ContentContentblog z.COM z.COM z.NET z.NET .COM/.NET Dynamic Dynamic Dynamic Dynamic Dynamic SiteSiteExecution Execution * Content Content Content Content Content Dynamic Data Cache Memcached Master Write Read SlaveRead SlaveRead SlaveDatabase 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 16 17. zembly database setup Master DB Local All writes go hereLocalHDHDRAIDRAIDPrimary Replication StandbyMasterMasterLocal LocalHDHD nioat lic ep R Replication 1 Backup/10mins MySQL Dump &/ ZFS snapshotBackup Slave ThumperThumper Tape ZFS Slave DB Slave DB Slave DBSlave DB 123 n Local Local LocalLocal RAID1 RAID1 RAID1RAID1HDHDHD HD 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 17 18. Architecture - experiencesPerpetual architecture start simple and grow organic developmentand maintenance inline with site traffic monitor site traffic always keep game plan for the next level readyHave exact same setup in staging could be a subset of production experimentation, debug, test, etc. changes run in staging for at least 2weeks before moving to production Monitoring built in to each layer of the stack 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 18 19. AgendaIntroduction Development process Architecture & deployment Availability & Performance 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 19 20. w ik ET ze i /bN m loy.bl g/ bly.fomCru zeOm MLoadBalancer zembly.COM zembly.NET forum Static Static wiki ContentContentblog z.COM z.COM z.NET z.NET .COM/.NET Dynamic Dynamic Dynamic Dynamic Dynamic SiteSiteExecution Execution * Content Content Content Content Content Dynamic Data Cache Memcached Master Write Read SlaveRead SlaveRead SlaveDatabase 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 20 21. w ik ET ze i /bN m loy.bl g/ bly.fomCru zeOm MLoadBalancer zembly.COM zembly.NET forum Static Static wiki ContentContentblog z.COM z.COM z.NET z.NET .COM/.NET Dynamic Dynamic Dynamic Dynamic Dynamic SiteSiteExecution Execution * Content Content Content Content Content Dynamic Data Cache Memcached Master Write Read SlaveRead SlaveRead SlaveDatabase 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 21 22. wikETzei /b Nmlo y. blg/bl y. fo m C ruze O mM Load Balancerzembly.COM zembly.NET forumStatic Static wikiContentContentblog deploy!z.COM z.COM z.NET z.NET .COM/.NETDynamic Dynamic Dynamic Dynamic DynamicSiteSiteExecution Execution *Content Content Content Content ContentDynamic Data Cache MemcachedMasterWrite Read SlaveRead SlaveRead Slave Database2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 22 23. w ik ET ze i /bN m loy.bl g/ bly.fomCru zeOm MLoadBalancer zembly.COM zembly.NET forum Static Static wiki ContentContentblog z.COM z.COM z.NET z.NET .COM/.NET Dynamic Dynamic Dynamic Dynamic Dynamic SiteSiteExecution Execution * Content Content Content Content Content Dynamic Data Cache Memcached Master Write Read SlaveRead SlaveRead SlaveDatabase 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 23 24. wikETzei /b Nmlo y. blg/bl y. fo m C ruze O mM Load Balancer zembly.COMzembly.NET forum StaticStatic wiki Content Contentblogdepl oy ! z.COMz.COM z.NET z.NET .COM/.NET DynamicDynamic Dynamic Dynamic Dynamic Site SiteExecution Execution * ContentContent Content Content ContentDynamic Data Cache MemcachedMasterWrite Read SlaveRead SlaveRead Slave Database2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 24 25. Architecture & Deployment - deploymentRolling deployments automated homegrown script turn off monitoring no user impact during deployment firewall block of port localhost URL still accessible deployment verified thru localhost URL versioning of webserver static content special care for db structural updates changes are versioned existing table definition changes need more than 1 deployment reason?Glassfish restarts takes the longest of the deployment time parallelize deployment after first server 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 25 26. AgendaIntroduction Development process Architecture & deployment Availability & Performance 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 26 27. Availability experiences differences between staging and productionhardware failureunmonitored servicelate night changeshuman errorsdata center problemsusing services without SLAlack of documentationlack of backup (human, infrastructure)2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 27 28. Availability - failoverFailover reduce the affect on users unknown SLA -- have a workaround ready (hopefullyautomated) redundancy everywhere (including humans)! helps also horizontal scaling document and test the doc must be solid and dependable gives you a good night sleep!get professional help mysql support 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 28 29. Availability - backups Backup, backup and double backup users are forgiving about site outages, but not data loss consider data loss from software defects & human errors DB - mysql dump, mysql ZFS snapshot htdocs - filesystem replication ZFS snapshot & replication rsync works just fine also be aware of your restoration time mysqldumps are slower but gives you consistent data always watch out for mysql settings that might prolong crash recovery innodb_log_file_sizepractice your maintenance mode 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 29 30. Availability monitoring Monitor everything appserver, DB, firewall, apache, even monitor the monitor! mysql monitor homegrown scripts log file monitors AdventnetUse datacenter monitoring if available advantages call home for issues first level support for hardware should be 24x7 live and online human to trouble shoot your problem as soon as it occurs2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 30 31. Availability problem preventionChange review avoid late night changes Trust, but verify document changes -- pay it forward build up an failure proof recovery manual Watch your traffic collect all sorts of stats to start with, don't worry about coherency of data Monitor & penalize users for malicious activities Keep all infrastructure software security patches current specifically the web facing ones wiki, blog, forum, etc Run a security audit on your site2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 31 32. AgendaIntroduction Development process Architecture & deployment Availability & Performance 2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 32 33. Performance run forest run...fast Monitor your site traffic for performance numbersRead from slave DB alwaysMemcached is your friendDB query optimizations takes you really farStart with best practice configurations -- adjust systematically based on live dataKnow your OS!Know your network!Have an automated performance test suiteExpert help [blog, support, ...]2009 CommunityOne WEST Conference | san francisco, ca | developers.sun.com/events/communityone 33 34. What Do You Need to Know AboutCreating and Running a Scalable WebSite but Were Afraid to Ask?Session Code: S304347 Chris Webster chris.webster@sun.com Girish Balachandran girish.balachandran@sun.com