ZendCon09 How to run an enterprise PHP shop.

By Jim Plush - Technical Lead, Panasonic

Writing Code is the Easy PartA closer look at some of the process that goes along with being a technical lead.

Balancing process and productivity

The balance between process and productivity

The main objective is to find the least amount of process required to get clean, stable, well tested, well documented, maintainable code.

If there is too much process, productivity and creativity are stifled. Too little process and you run the risk of delays, instability, maintability concerns and customer dissatisfaction. 

This balance is something that needs to be revisited often with feedback from the teams.

The importance of balance

When process is so cumbersome that it hampers productivity, it will slow product delivery. Longer product delivery cycles create openings which startups and competitors will take advantage of.

This is how companies not only lose market share but also high performing employees due to the barriers put in place that prevent them from doing their jobs efficiently.

Why Process?

The larger your application grows, the more important foundation becomes.

When process fails

The larger your application grows the more important foundation becomes.

The Building Blocks For Your Team

• Team Management• Development• Testing• Maintenance• Security

Team Management: Building your team

• Where to find developers• How to interview them

 

Where to find developers

Don't always rely on HR, Monster and Dice• IRC PHP Channels• Craigslist jobs/gigs section• PHPClasses.com• Blogs of active developers• Previous jobs These are all places I've found developers on our team.  People also tend to feel more confortable knowing a developer is contacting them vs a recruiter.

How to interview developers

• Determine fluff vs fact on a resume• Set up phone interviews• Determine when it's worth bringing

someone in• Set up in person interviews with team• ZCE's

In Person Interview• Determine the order of the team members

ahead of time.  • Have a brief technical test ready that covers

a wide area of what you need.• Have the candidate take the test first with

HR, then start the interview process. • Ask the right questions.• The test should not be the determining

factor but merely part of the process.

Wrong Questions

Why I don't favor these types of "thinking" questions: 1. Most people who ask this aren't qualified to determine the

thought process for the answer given.2. It generally has no relevant bearing on the position

How would you load an elephant on a 787?

Better Question

Why I think this type of question is better: 1. There is no right answer but it becomes clear exactly how

much experience someone has. Do they start at the network level, load balancers, clusters, caching, dbload, hardware failures, browser level, user error, profile the code, check the logs, run performance test suite, work with the user on reproducing the issue

2. You're qualified to understand their thought process

A user reported the website is running slow. What steps do you take to address

the issue?

Other good phone questions I like• What features in PHP5 do you like the most that weren't

available in PHP4?• When a user types in a web address in their browser and hits

go, what happens behind the scenes?• Explain a design pattern you've used in the past that's helped

your architecture• What's Unit testing? What's your approach to testing• Explain the differences between GET and POST, why would

you choose one over the other in your application?• What is XSS (cross site scripting)?• What is SQL Injection?• What steps do you take in your coding to minimize

vulnerabilities?

DevelopmentSeveral fundamental concepts required to create a maintainable development environment. 1.Setting Developer Expectations2.Documentation Standards 3.Source Control Management4.Coding Standards5.Unit Testing Standards6.Release Process 7.Making developers more valuable

Development: Setting Developer ExpectationsWhen new developers come on board you need to define what you expect from them. 1.Follow the standards laid out by the team

(source, docs, unit testing, etc...) 2.Automate and have backup procedures 3.Be accountable for their work4.Be able to create documentation / use the

wiki5.Communicate openly6.Core hours

Development: Documentation StandardsIf you find yourself in charge of requirements or you need to work with your internal groups to define requirements you really only need three documents. 

1.Requirements Document2.Design Document3.Test Plan Document

Documentation: What's the difference?A Requirements Document is a contract between the client (even internal users) and the application. It should use terminology that is easily read by management and non-technical stakeholders. Even in an agile process a requirements document should be flushed out at a high level before starting development. The requirements document is how you know when something is done.

Requirements DocumentHere are some examples of what would typically appear in a requirements document.    • REQ123 - A User shall be able to search for

a  producto REQ123.1 A user can use a wildcard

search such as "*" or define a particular field such as "year:2003"

 What is not a requirement:• REQ124 - System shall be easy to use (not

testable) 

Design DocumentThe design document is aimed at the technical and testing audience. It is used to define how to actually implement the system you're designing. UML, schema designs, API lists can all go into the design doc.

The design document should outline how the system makes decisions and what constraints are placed on them.

The Quality Assurance team should be able to create a test plan based on this document.

Design Document Contents• REQ123 - A User shall be able to search for

a  producto year, vintage, winery are valid search

fields o The search is implemented using Zend

Lucence, each field is created as a tokenized element. 

o The search index is stored in the application's temp directory. 

o The index is regenerated every 24 hours via a cron job that runs on the server.

Test Plan DocumentThe test plan document is used by the QA team to verify the requirements and design are met and can be tested against every release.  The test plan is used to create automation against (Selenium,  WinRunner, etc)

Automation of the test plan ensures consistent user experience.

Test Plan ContentsREQ123 - Search for product Test Cases

ID: 1Action: Create a new product then type in the product name in the search barExpected: The product that was created appears in the search listActual Result:Pass/Fail:  Comment:   

Documentation: But I just want to codeThis is where setting team expectations comes in.

Documentation is part of the job of any engineer, including software engineers. It helps new product managers, stakeholders, developers get an understanding of what a product is supposed to do and why it's doing it.  The goal should not be to generate the most documentation but "enough" documentation to get the job done.

Source Control Management

• There is no right tool, just make sure you have a tool

• Make sure that everyone follows the best practices set by the team leads.

• Pick the appropriate tool for your team and dev platforms.o Is your team distributed? Perhaps GITo Windows Only? VSS o Mix of platforms? SVN. CVS

SCM Rules•  Your source should be backed up both on

and off site• At least 1 person should be in charge of the

administration of monitoring that product. (At a previous job we found out the SCM admin hadn't backed up the source in weeks due to a failing backup script).

• Branch a release, only release a branch• Incorporate continuous integration hooks

Coding Standards• Pick a coding standard and stick with it

(PEAR, Zend, etc)• Create an environment that promotes

developer peer reviews• If it becomes a contention point for the

team, use an automated plugin to inspect checked in code vs team standards (PHP CodeSniffer)

FrameworksIn multi developer environments it makes sense to invest in frameworks.  At a high level Zend Framework, Cake, Symphony all offer similar feature sets. To maximize code reuse standardize on a single one for the team.

Frameworks cost upfront time but that is paid back many ways down the line.

Life Before A Framework What was the problem? • Numerous tool sites all coded in different

styles and languages. • Operations team had different deployment

strategies for each• No roll up reporting across content tools• Dozens of sites to maintain and secure   

Life Before A Framework

No sharing of code bases, no common standards for security, access control, authorization, UI presentation, reporting.

Why I chose Zend Framework for Enterprise• Large user base• Company behind the product• Training / Consulting available• Products built around the framework• Modular design• Can use components individually• Large feature set• Heavily unit tested• Can get new developers up to speed with

minimal instruction

After Zend Framework

Sites are plugged in as modules. Allowing for complete sharing of all features, giving the user a consistent experience while also getting the benefits of easier deployments and maintenance for the operations team.

The myth that Zend Framework is slow• Developers are expensive, hardware is

cheap(usually)!• Opcode cache increases performance

considerably• Memory caching (memcached)• Disk based caching • Database tuning(proper indexing, multi-tier)• Specialized AJAX bootstrap  Weigh the cost and time of all decisions.  

Unit Testing  Code does not exist until it's tested.

The "Liger" Effect 

Unit Testing: The Gift that Keeps On Giving• Unit tests should be a cornerstone of your

development• They're no longer "optional" • What is Unit vs Integration?• Continuous Integration after every checkin • Include the time for unit tests in your

estimates    

Unit Testing: The Cornerstone• Standards should be enforced teamwide as

to what the testing expectations are. • Code should not be checked in without tests

behind it• At the bare minimum the business logic

should have a high code coverage level. • PHP Unit can be run with XDebug's code

coverage to allow you to see exactly what code was run.

• Aim for % of acceptable covereage• Tie in Unit Test Coverage to

bonuses/reviews• Hold team meetings to help get new

developers on board  

Unit Testing: Unit vs ItegrationGenerally a unit test is considered a unit test when you're testing one coheisive action. A unit test may span multiple classes if you control those classes.  Once a test touches 3rd party libraries or external resources (databases, filesystem, web service) it's generally considered an integration test.

If your test class needs a tear down method you most likely have an integration test.

  

  

Unit Testing: Unit vs ItegrationSo what's the rub against integration tests?  Answer: NONE! Most web applications benefit more from integration tests. Both require a decent amount of setup (resource management vs mock management).  Your suites should include both however for speed purposes you may want to separate the integration tests into a separate suite due to the slower run time. 

Unit Testing: Itegration testing practicesUse Builders to help with schema change If my test requires a new user to test against I could insert that into the db by hand which would lead to brittle tests if  you have 3000 tests and all the sudden you change the user schema. 

Or I could use a UserBuilder to abstract that API to a central place.

Unit Testing: Builders

public function testUserCanUpgradeStatus(){    $userBuilder = new UserBuilder();    $user = $userBuilder->withFirstName('Jim')                                      - >withEmail('jim@test.com')                                      ->build();

    $user->upgradeToStatus('preferred');    $this->assertTrue($user->canReserveSuite() === true,              "User was not able to upgrade to preferred status");

}

Unit Testing: Builder BenefitsUsing Builders allows you to abstract the creation of test data into a series of fluent APIs.

Tests are no longer as brittle with all tests now using Builders instead of direct SQL manipulation.

You can have as many builders as you need!

UserBuilder, CompanyBuilder, ProductBuilder, OrderBuilder, etc...

Unit Testing: Builder Performance BenefitsUsing Builders can also speed up your integration suite as the builder can be combined with object caching or data caching to reduce the duplicate data that exists across tests.

If 1000 tests use the same User information to base tests against, return the instance of the 1st created one (if unchanged).

Unit Testing: Continuous Integration

Unit Testing: Continuous ItegrationThe goal of continuous integration is to spot issues early and reliably. You create a script that runs every x seconds or monitors your repository. When it detects a change it grabs that code, packages up your project, runts the unit test suites, and alerts developers of any immediate issues.  Common CI Frameworks: PHPUnderControl, Xinc, CruiseControl, or write your own. The concepts are simple.

Unit Testing: Continuous ItegrationYour script should be fully automated and reliable. If there is a failure it's the teams priority to address it. 

If the code passes all of the tests, package it up and make it available for non-developers or QA teams (if needed).

Developers should checkin early and often as pieces are complete.

Every checkin should be uniquely validated against to have fine grained reporting on which change "broke" the build.

Release Process: Getting Code to CustomersThe release process defines who, how, and when code is release to a production environment.

Release Process: Quality Expectations

Local Dev Test Production

Release Process: Best PracticesMinimizing release issues can save time and headaches. If there is one thing you should define first, it's your code release process.

It should be controlled and precise. Developers generally should not be pushing to test or production.

Release Process: Dev Servers• Should allow developers full access to

system• Should run on production type hardware

with production OS and libraries • Should support multiple dev instances of a

site• Should be able to be wiped out without

ramifications• QA should use to develop test cases and

automation

Release Process: Test Servers• Should be a mirror of production

environment• Should be controlled by operations team• Issue should be created with branch or

package to push to test and assigned to ops team

• QA runs automation and regression tests against

• Code on test should be the same code targeted to a production release

• Code pushed to test should be based on a branch of the code

Release Process: The bug bash• The bug bash is a pre-release event before a

major milestone release to flush out uncaught issues.

• Generally held after hours with people from across the company

• Prizes awarded• Food provided• Team atmostphere

Release Process: The bug bash• Have one or more people from the bug

scrub team monitoring and filtering the issues tracker live. 

• Award cash prizes for most bugs found, most severe bug, most show stoppers, biggest security hole.

• Budget this cost into your initial project estimates. 

• It's better for people in the company find major bugs before customers.

• Everyone meets in one or more conference rooms with laptops or desktops set up.

• Developers do not test their own code • Don't assume you can release the day after

a bug bash, estimate some time to address show stoppers.

Maintenance• Issue Tracking• Working with Testers • Customer Support• Monitoring

Issue TrackingA common issue tracking system must be used across the teams.  The issue tracking life cycle at a high level:1.Issue reported2.Issue verified/rejected3.Issue assigned to a release and fixed4.Reporting user notified of status

Issue Tracking: Bug ScrubsA bug scrub is a daily or weekly meeting with technical leads, managers and product managers with the aim of addressing the open issue list for a give project or projects. 

The goal is to make sure bugs/feature requests are addressed in a timely manner with the appropriate release tags and supporting information.

Issues can be rolled into iterations as they get processed.

Issue Tracking: Bug ScrubsThe bug scrub meetings should be as brief as possible, technical details should be handled offline with the appropriate parties.  Bug Scrub Flow:1. View open issues/feature requests2. If already assigned and processed

1.follow up with owners as to why the item is still outstanding

3. If open• Reject or accept• Assign a release and owner to with information

discussed

Issue TrackingIf the issues are tagged with the appropriate release information, QA is able to pull a report from the system to use for release testing without having to guess at what's in the release or rely on developer's memories.

If issues are further labeled with modules or functional pieces of code, metrics reports can be run that can identify key pieces that made be candidates for re-factoring or redesign.

e.g., if 60% of the issues are in user management, more testing efforts can be directed to it.

Customer SupportAfter a major release you have to expect an increased flow of issues into the system. Daily bug scrubs may be required to properly filter out duplicates and "training" issues. 

Set up a front line person that can immediately filter and address issues as they come in. Rotate this position around the team weekly. This allows everyone to get a glimpse into how the customers interact with the system and where the areas of concern are so they can apply those to future development

Monitoring: System and Business• The operations team should not only

monitor the system processes but have the developers contribute plugins for business processes as well.

• e.g., set a min threshold for how many users you expect to sign up in a given hour, if under that threshold notifications go out. This may indicate a problem with the sign up form.

• ProActive business rule monitoring allow faster response times to application issues

Monitoring: Disaster Recovery• A Disaster recovery plan should be created,

implemented, tested and made available to all parties. 

• Ideally your disaster recovery site should be located outside unstable areas.o if primary is in earthquake prone area, backup

should be elsewhereo if primary is in an unstable country politically,

backup should be elsewhere• Disaster recovery is just that... for Disaster,

hardware failures should be handled by your primary site by taking the node offline. 

• Proper monitoring should allow automatic fail over.

• Data should be propagated over on a regular basis. (snapmirror)

Monitoring: Disaster RecoveryExecute your disaster recovery at least once to ensure procedures work correctly for moving the pointers to the secondary data center but also for moving that data back to the primary when and if it's made available again.You don't want to guess that your D.R. site works.

SecuritySecurity is often a last thought on dev teams

• Plan to be attacked• Code defensively• In-House Social Engineering• Security Audits

Security: Plan to be attackedIf your site offers any value, chances are it will be probed by malicious users. The value of your data directly correlates to the intensity of the attacks.  You should proactively monitor security breaches both in the operations team and in code. Have a plan of action to take when an attack has been verified. What information to log, authorities to notify, upper management notices, etc...

Security: Code Defensively• Did a user's browser profile just change?

Have them re-login (mitigate session hijacking)

• Filter all input• Escape all output• Encrypt/hash(with salt) all passwords• If backend is networked across un-trusted

network segments, encrypt transmission to avoid man in the middle attacks.

• Don't offer clues to attackers on create/login pages(user account found but password is invalid)

Security: Social Engineering• Plan random social engineering attempts

against administrators/data holders.• How much information about your

clients/sites is available on searches? • How easily is it to use that information to

make educated attacks? • If new user accounts must be authorized

first, is there a concrete plan in place to verify information.

• Use email and phone to gather information and make requests.

Security: Audits• If possible budget for quarterly/annual

security audit• Set up a separate non-production instance• Pick a 3rd party who specializes in this field• Audit the application and the network

independently. 

The Endcontact:

jiminoc@gmail.com litfuel.net/plushtwitter: jiminoc

http://joind.in/talk/view/931