building a better beta · • something that should be confined to the product management or...

BuildingaBetterBetaSaeedKhan

TransformationLabs

http://www.transformationlabs.io

[email protected]

(c)2019TransformationLabs 2

Buildingabetterbeta

Buildingabetterbeta.........................................................................................................2

Introduction........................................................................................................................3

Originsofbetaprograms....................................................................................................4

Whatabetaisnot..............................................................................................................5

Betaprogramoverview......................................................................................................7

PhaseI–Pre-Beta...............................................................................................................9DefineBetaObjectives...................................................................................................9BetaRequirementsDocument(BRD)...........................................................................10BetaProspecting..........................................................................................................11PrioritizeBetaCustomers.............................................................................................12RecruitInternalResources...........................................................................................13ConductInternalEducation..........................................................................................13EducateBetaSites........................................................................................................13ProvideTestCaseScenarios.........................................................................................14BetaSoftwareReadiness..............................................................................................14

PhaseII–Beta..................................................................................................................18LaunchtheBeta............................................................................................................18WorkingwithBetaSites...............................................................................................18RecordandDisseminateCustomerFeedback..............................................................19ProvideTechnicalSupport..........................................................................................20CustomerVisits.............................................................................................................21DevelopReferenceSites...............................................................................................21ExtendingtheBeta.......................................................................................................22

PhaseIII–Post-Beta.........................................................................................................25Finalcustomercalls......................................................................................................25SummarizeandAddressBetaFindings.........................................................................26InternalRetrospecitveSessionsandPublishingResults...............................................26

Conclusion........................................................................................................................27

ContactUs........................................................................................................................28


IntroductionIsitjustme,orisdeliveringasuccessfulbetaprogramoneofthehardestthingstodoasaproductmanager?Thedifficultyarisesbecausesomuchisoutoftheproductmanager’sdirectcontrolandinthehandsofthebetatestersthemselves.Andifyoucan’tgetmotivatedbetacustomerstoprovideyouwithdetailedfeedback,youcansignificantlydiminishordelaytherevenuepotentialforyourproduct.

Managinganeffectivebetaprogrammeansdoingwhatittakestominimizetheaspectsthatareoutoftheproductmanager’sdirectcontroland,likeanyotherprogramthatisexecuted,bringasmuchpredictabilitytoitaspossible.

Thinkaboutit.Abetaprograminvolveshavingpeoplespendtheirvaluabletimeusingyourunreleased,andinmanycases,buggysoftware.Andontopofthat,youusuallyhaveafairlyshortperiodoftimetogainvaluableinsightsonwhatissuesneedtobeaddressedwiththeproductbeforeitisreleased.


OriginsofbetaprogramsTheterms“alpha”and“beta”werecoinedbyIBMinthe1960stolabelproductdevelopmentcheckpoints.Alphameantaunit,moduleorcomponenttestphaseandbetawastheinitialsystemstest.ThesetermscamefromevenearlierA,BandCtestsforhardware.TheA-testwasafeasibilitytestthatwasdonebeforeconductinganydesignordevelopment.TheB-testwasademonstrationorprototypeshowingthattheengineeringmodelfunctionedaccordingtospecifications.AndfinallytheC-test(whichmapstothecurrentusageofthebetatest),wasperformedonearlysamplesoftheproductbeingcreated.

Softwarebuilttodayisquitecomplex,andwhetheritiscalledbetatesting,externalvalidation,earlyreleaseorsomethingelse,itisalmostuniversallyacceptedthatsoftwaredevelopmentisaniterativeprocessandthatuserfeedbackiscriticaltodevelopinggreatsoftwareproducts.Andtomaximizethevalueoftheacquiredfeedback,astructuredprocessmustbeinplacetomanagethebetaprogramfromendtoend.


WhatabetaisnotBeforeIgetintothedetailsofwhatabetaprogramisandhowtooptimallymanageit,it’simportanttounderstandwhatabetaprogramisnot.It’snot:

• asubstituteforgoodqualityassurance• asubstituteforproperdesignandusabilitytesting• atoolforthesalesteamtohelpclosedealsintheirfunnel• somethingthatshouldbeconfinedtotheproductmanagementorproduct

developmentgroupsinacompany

First,abetaprogramisnotasubstituteforgoodqualityassurance.Sendingoutanearlyreleasetobetatestersandcountingonthemtofindbugsthatshouldbefixedisaverybadobjectiveforabetaprogram.Infact,ifyourdevelopmentorQAteamsdisagreewiththeprevioussentence,thentheyshouldrealizethatwhattheyaresayingisthatsomethingwhichisfundamentallytheirresponsibilityshouldbeoutsourcedtoagroupofexternalpeople.ThenextlogicalstepwouldbetooutsourcetheentiredevelopmentorQAfunctionaltogether.

Relatedtothepreviouspoint,abetaprogramisnotasubstituteforproperdesignandusabilitytesting.Therearetwomajorreasonsforthis.Thefirstissimplythatsomethingthatisanintrinsicpartofthesoftwaredevelopmentprocesscannotbelefttobetacustomerstoresolve.

Efficientuserinterfacesneedtobedesignedandimplementedwithanunderstandingofuserneeds.Theymustbebasedoneasilyunderstoodcognitivemodels,supportspecificworkflowpatterns,minimizehiddenorhard-to-accessfunctionality,andinmanycases,beadaptiveorconfigurabletosupportadiversepoolofusers.Thesethingsneedthought,planning,andsufficientleadtimetoimplement.Andthus,ifleftuntilbeta,almostguaranteethatsignificantrewritingwillbeneeded,possiblycausingadelayintheproduct’sreleasedate.

Next,abetaprogramisnotatoolforthesalesteamtohelpclosedealsintheirfunnel.I’mnotsayingthatexposuretobetasoftwarecan’thelpcloseadeal,butdirectprospectparticipationinabetaprogramisabadidea.Whynotgiveaprospectearlyaccesstosoftwaretogetthemexcitedaboutwhatiscomingdownthepipeline?Well,firstofall,youcannotsellbetasoftwareandthereareeasierwaystoprovideprospectsinformationaboutwhatiscominginthenextrelease.Thiscanbeachievedviaon-sitedemonstrations,Webinars,orevenwhitepapersandpresentations.


Additionally,simplesoftwareproductsnotwithstanding,thecostofsupportingabetasiteandtheriskofexposingtheprospecttopotentiallybuggyfunctionality(itisnotfullytestedanddebugged,afterall)mayoutweighanyadvantagesorbenefitsinacceleratingthesoftwaresale.

Finally,abetaprogramisnotsomethingthatshouldbeconfinedtotheproductmanagementorproductdevelopmentgroups.Ifasoftwarecompanyisworkingonamajorproductrelease,andparticularlyifthatproductgeneratesasignificantportionofthecompany’srevenue(i.e.itisnotamarginalproduct),thenpeoplefromacrosstheentirecompanymustworktogethertosetobjectivesforthebetaprogramandensuretheyaremetorexceeded.

Afterall,thebetaprogramisalmostalwaysthefirstexternalexposureoftheupcomingsoftwareproduct.Andifthisproductisexpectedtodrivefuturerevenuesforthecompany,thenthevalueof,andinvestmentin,theprogrammustbelookedatfromanoverallcompanyperspective.Virtuallyeverycustomer-facinggroupinthecompanyshouldbeinvolvedinthebetaprogram.WhetherthegoalisrelatedtotechnicalenablementforgroupssuchasCustomerSupportorProfessionalServices,identifyingpressandanalystreferencesforMarketing,oracceleratingtimetorevenueforSalesbydevelopingearlyadoptersites;takingabroad,company-wideviewofthebetawillmaximizethebenefitsofthebetaprogramfortheentirecompany.


BetaprogramoverviewBetaplanningandexecutioncanbebrokenupintothreephases,witheachstagehavingmultipletasksandobjectives.It’simportanttonotethatthefollowingtableprovidesadetailedsetoftasksforacomprehensivebetaprogram.Noteverybetaforeveryproductwillneedallofthesetasks.It’suptoyoutodecidehowmuchorlittleyouneedtofollowforyourbetaprogram.

PhaseI–Pre-Beta PhaseII–Beta PhaseIII–PostBeta

DefineBetaobjectivesandtheBetaRequirementsDocument

LaunchtheBeta Finalcustomercalls

SolicitBetacustomers WorkwithBetasites SummarizeBetafindings

PrioritizeBetacustomers Recordanddisseminatecustomerfeedback

AddressBetafindings

Identifyinternalresourcesandresourcereadiness

Providetechnicalsupport

Internalretrospective

EducateinternalteamsandBetacustomers

Visitcustomers Publishpostmortemresults

Createtestcasescenarios Developreferencesites

PhaseIisthepre-Beta(orplanning)phase.

PhaseIIistheactualtesting(orexecution)phase.

AndPhaseIIIisthePost-Beta(oranalysis)phase.

Thefollowingsectionsprovidedetailsoneachofthesephasesandthetasksinthosephases.


Phase1–PreBeta


PhaseI–Pre-Beta

DefineBetaObjectives

Asinmostsignificantendeavors,planningandpreparationarekeystosuccess.Withbetaprograms,successwilldependheavilyuponproperexecutionofactivitiesbeforethebetatestingactuallybegins.Thefirstthingthatmustbedoneistoidentifythegoalsofthebeta.Todothis,anumberofquestionsmustbethoughtthroughandanswered.

Whatarethedesiredoutcomesofbeta?Themainobjectiveforbetais,ofcourse,productfeedback.But,whatkindoffeedbackisneeded?Aretherespecificfunctionalareasoftheproductthatrequirefeedback?Isitfocusedontheusability,performance,scalability,userworkflows,securityetc.?

Aretherecertainconfigurations,platforms,usecases,orothersscenariosthatneedtobetested?Howmanybetasitesneedtoprovidefeedbackforeachoftheseconfigurations,platforms,usecasesandscenarios?

Forexample,iftheproductissomethinglikeanAPI,withnewfeaturesrelatedtoscalabilityandtransactionprocessing,withsupportforREST,PythonandJava,youneedtodefinehowmanybetaparticipantsyouwanttestingthesefeatures,whatkindofteststheycanperformwithwhichlanguages.

Youshouldalsoseeifexistingcustomerscandosomehead-to-headperformanceandscalabilitytestscomparingyouroldversionwiththenewone.

Beyondproductfeedback,itistypicalinenterprisesoftwarecompaniestosolicitsomebetaparticipantsasreferencesitesforpressquotesandanalystbriefings.Howmanyofthesereferencesitesareneededandwhatistheprofileofthosesites?Dotheyneedtobedirectcustomersorcantheybepartners?Dotheyneedtobelarge,name-brandenterprisesorcantheybesmaller,lesswell-knowncompanies?Dotheyneedtobeinparticulargeographiesaroundtheworld?Ifso,howmanyareneededineachgeographicregion?

Otherquestionstoanswerwhenplanningforbetainclude:

• Definingthebetatimeframe.Whenwillitstart,andwhenwillitend?

Assumingyouhavesoftwarethatneedstobeinstalledatcustomersites:

• Willcustomersinstallitorwillyousendsomeonetheretoinstallitforthem?• Howwillyoudistributepatchesorinterimbuildstocustomersduringthe

betaprogram?


• Docustomersneedtrainingonthebetasoftware?Ifyes,howwillitbedelivered?

• Whenwillitbedelivered?• Howwillcustomerscommunicatetheirfeedbacktoyouduringbeta?How

willtheyreceivetechnicalsupport?• Willbetacustomersbeallowedtocommunicatewitheachother?Ifyes,then

howwillyoumakethathappen?

Thesearekeyquestionsthatshouldbediscussedinternallyandagreeduponintheearlyplanningstagesofthebetaprogram.

InabetaImanagedforamajorsoftwarerelease,wediscussedalloftheissuesaboveanddecidedthefollowing:

• Thebetawouldrunforaperiodofthreemonths,startingaboutsevenmonthsbeforetheexpectedreleasedateoftheproduct..

• Fortraining,wewouldholdthree90-minutewebinarsforthebetacustomersintheweekbeforetheBetastarted.

• TheseWebinarswouldcoverallofthemajorfunctionalityareas.• Wealsohadlivedemosonthesefunctionalareasinthewebinars.• Wesetupweekly30-minutecallswitheachbetasiteandassignedaproduct

managerandaprogrammanagertoeachcustomertomanagecommunicationsandtracktheirprogress.

• Customerswoulduseourstandardcustomersupportprocesstoaccesstechnicalassistanceduringbetaandwouldusetheweeklycalltodiscusshigher-levelissuessuchasspecificproductfeedback.

BetaRequirementsDocument(BRD)

Asyoucansee,therearemanyquestionsregardingthebetathatneedtobeansweredintheplanningstage.

AllofthisinformationshouldbecollectedanddocumentedinwhatIcallaBetaRequirementsDocument(BRD).TheBRDisasinglepointofreferenceforallaspectsofthebeta.Itcontainsinformationonthekeyinternalstakeholdersforthebeta(i.e.fromProductManagement,ProductMarketing,CustomerSupport,etc.),theirresponsibilities,keymilestones,andprocessdescriptionsaboutthebeta(i.e.howmanycustomersareexpectedinthebeta,theirproposedprofile,thebetaregistrationandapprovalprocess,howsoftwarewillbedistributed,howsupportwillbeprovided,howfeedbackwillbecollectedandrecorded,etc.).

BycreatingandpublishingtheBRD,anyonewithinthecompanywhoneedsanunderstandingofthebetaprogramcanreadtherelevantportions.Italsobecomesa


referencepointagainstwhichyoucanmeasuresuccessafteryouhavecompletedthebeta.

BetaProspecting

TheBRDshoulddefinetheidealprofileoftheparticipantsyouwanttohaveinthebeta.Howmanytotalbetaparticipantsarerequired?Whatmixofnewandexperiencedusersdoyouwant?Whatmixofdirectcustomersandpartnersareneeded,etc.?Butgettingtherightnumberandmixofparticipantstocommittobetaisanotherstory.

Remember,participatinginabetaprogrammeansthatpeoplearegoingtospendtheirtimeonyourunreleasedsoftware.Givenpeoples’busyworkschedules,findingasignificantnumbertoactivelyparticipateduringthetimeframeofyourbetaisachallenge.

Lookatprospectinginthesamewayyoulookatsales:buildafunnelofqualifiedprospectsthatissomemultiple(e.g.twoorthreetimes)ofthetotalnumberofbetaparticipantsyouneed,andthenworktocloseasmanyofthemaspossible.Andofcourse,tobuildthisfunnel,youwillneedtocontactanevenlargermultipleofpeople.

Asanexample,inmybetaprogram,weidentifiedsixmajorfunctionalareaswherewewantedbetafeedback.Wealsodecidedthatourgoalwastohavebetweenthreetofiveactivebetaparticipantsperfunctionalarea.By“active”Imeanpeoplewhowouldregularlyusingthebetasoftwareandprovidingmeaningfulanddetailedfunctionalityfeedback.

So,threetofiveactiveparticipantsperareatranslatedto18to30activebetaparticipantsoverall.Butensuringthatwemetthisnumbermeantrecruitingatleasttwiceasmanyintothebetaprogram.

Therearealwayssitesthatinitiallycommittobetabutthendropoutordonotparticipateasactivelyasplanned,soensurethatyouhavesignificantlymorethanthetargetnumbercommittedtobeta.Assumingroughlya50%dropoffbetweenthosewhocommittobetaandthosewhoactuallyparticipate,wedecidedwewouldneedatleast40to50fullyqualifiedandapprovedbetaparticipantsatthebeginningoftheprogram.

Wedidalotofprospectingtoacquirethesequalifiedsites.Over500customersandpartnerswerecontacted.Weworkedcloselywithoursalesandmarketingteamstoidentifyprospects.Wealsocanvassedcustomerswehadvisitedinearlyrequirementsgatheringtripsforfunctionalityinthisrelease.


Partofourqualificationprocessincludedunderstandingwhythepersonorpeoplewereinterestedinparticipatinginthebeta.Wasthereacriticalbusinessproblemthatneededtobesolvedordidtheyjustwantto“kickthetires”ofthenewrelease?Wealsofoundpartners,specificallyVARs,SIs,partnerISVs,andevenindependentconsultantstobeparticularlygoodbetacandidates.Mosthadaclearbusinessreasonforwantingaccesstothebetaandhadstaffthatcouldbededicatedtothebetaandworkwithustoprovideverydetailedproductfeedback.

Withdirectcustomers,thestorywasalittledifferent.Whileweidentifiedmanywell-intentioneddirectcustomers,spendingtimeonbetameantanadditionaltaskthathadtobedoneontopoftheirfull-timejobs.Thus,itwassomethingthatmanyofourbetacontactswerenotkeentotackle.Inthosecases,weworkedwithouraccountmanagerstoescalatethebenefitsofthebetaprogramhigherupatthecustomersiteandtriedtogetamanagementsponsorforparticipatinginthebeta.Overall,thiswasaverylaboriousprocess,andtookabouttwomonthstocomplete.

Atthatcompany,ProductManagementrecruitedthebetasites,butproductmanagersIhavespokenwithinothercompaniessaythattheyhaverecruitingteamsintheirorganizationswhocallandsolicitbetacustomers,thoughinthosecases,itisstillProductManagement’stasktodefinethebetacustomerprofilesandrecruitmentcriteria.

Keepinmindthatrecruitingisthemostcriticalaspectofthebetaprogram.Getitwrong—thewrongsites,thewrongmixofsites,ortoofewsites—andyouareunlikelytomeetanyofthebetaprogram’sdownstreamgoals.Andwhilegettingitrightdoesnotguaranteesuccess,itdoessignificantlyincreasetheprobabilityinreachingthosegoals.

PrioritizeBetaCustomers

Notallbetacustomersarecreatedequal.Duringtheprospectingprocess,itwillbecomeevidentthatsomecustomersaremorelikelytoprovidegoodfeedbackthanothers.Asmentionedearlier,thosewithclearbusinessneedsaddressedbythebetasoftware,aswellaspartnercompanies,typicallyhavetheincentivetoinvesttimeandresourcesintoyourbetaprogram.Inadditiontothesecompanies,theremaybesomebetaparticipantswhoareconsideredmarqueecustomersandarepotentialbetareferencesites.Inshort,asubsetofthebetaparticipantswillwarrantextraattentionandsupportduringthebetaprogramastheyaremorelikelythanotherstohelpyoureachyourtargets.

Identifythesemarqueecompaniesearly.Makeallinternalpartiesinyourcompanywhoareinvolvedinthebetaprogramawarethatthesefirmswillbemonitoredclosely,andifneeded,givenadditionalhelpandsupporttoensuretheyare


successfulinthebeta.Assignnamedtechnicalresourcestotheseparticipantstobetterensuretheirbetaprogramsuccess.

Theproductmanagementteamshouldworkcloselywiththesecustomers,visitingthemorengaginginadditionalcontacttohelpresolveissuesthatarise.

Youmaynotbeabletoaccuratelyidentifyallmarqueesitesuntilthebetaisactuallyinprogress.Therearemanythingsthatcanhappenbetweenthetimeacustomercommitstoparticipatinginthebetaandwhenthebetaactuallybegins.Insomecases,whatlookedlikeapromisingsiteduringrecruitmentturnsouttobeadudoncetestingbegins.Andtheoppositeisalsotrue.Whatsoundslikeasleepybetaprospectturnsintoaveryactiveandengagingbetasitethatfarexceedstheoriginalobjectives.Thereisnoharminupdatingthemarqueelistoncebetastarts,andinfact,itmaybecomeanecessityasthebetaprogresses.

RecruitInternalResources

Asbetarecruitingistakingplace,youneedtoensureinternalresourcesareidentifiedandwillbepreparedforbeta.Whowillprovidetechnicalsupporttothebetasites?Isitindividualsfromthetechnicalsupportorganization,fromprofessionalservices,salesconsulting,orcoulditevenbefromyourdevelopmentandqualityassuranceteams?Itvariesacrosscompanies,typicallydependingonthenatureofthesoftwareandthesizeofthecompany.Forsmallercompanies,betasupportmaybedeliveredbyEngineering,whereasinlargercompanies,theCustomerSupportteammayhandleit.

ConductInternalEducation

Onceyouhavedefinedtheinternalteammembers,ensureappropriateindividuals—particularlythosewhowillprovidethetechnicalsupportforthebeta—areproperlyeducatedonthebetasoftware.

Dependingonthecomplexityoftheproduct,thiscouldbeassimpleasgivingtheinternalgroupsaccesstothesoftwareandhavingthemeducatethemselvesviathedocumentation,oritcouldrangeallthewayuptodeliveringformaltrainingclassesaheadofthereleaseofbetasoftwaretocustomers.

Theobjectiveensuresthatyourinternalteamsareatleastonestepaheadofcustomers,sothatascustomersstartusingthesoftware,theinternalteamsarereadytorespondtoanyproblems.

EducateBetaSites

Betacustomerstypicallyneedsomeformofeducationortrainingbeforeusingthebetasoftware.Ifthesoftwareisaconsumer-orientedproduct,thenideally,littleor


notrainingshouldberequired.Worstcase,thecustomershouldbeabletoreadorviewashortonlinetutorialtogetthemgoing.Conversely,complexbusinessorenterprisesoftwaremayrequireextensivetraining.

Asmentionedearlier,weheldthree90-minuteWebinarsforourbetacustomers.VirtuallyallbetacustomersattendedallthreeWebinarsandwereceivedpositivefeedbackfromtheattendees.Whilethatmadeusfeelgood,wealsoknewthatWebinarswouldonlypartiallypreparecustomerstousetheproductandwewouldhavetohelpthembyexplainingfunctionalityduringthebetaitself.Wealsoprovideddocumentationwiththebeta,butthattoowasina“beta”state,andwhilemostsectionswerereasonablycomplete,moredetailcouldandwouldbeaddedlaterinthedevelopmentcycle.

ProvideTestCaseScenarios

Withoutsomeguidance,alotofbetafeedbackcanbevagueandgeneric.Statementslike,“Itriedthis,thisandthat,andtheyallseemedtoworkOK,”or“ItseemstoworkasIwouldexpect,butInoticedthebuttonsinthedialogboxarenotproperlyaligned.”,arenotuncommonwhentestersarelefttotheirowndevices.

Tohelpguidethetestingdoneateachsite,providethebetacustomerswithsomestructure.Forexample,definebetatestscenariosforeachareaoffunctionalitythatneedstobetested.Thepurposeofthesescenariosistoprovideaclearframeworktothecustomerinwhichtotestthesoftware.Itisnottoprovideastep-by-stepsetofactionsthattheyshouldperform.

Thescenariosshouldfocusontypicalusecasesyouexpectcustomerstoperformoncethesoftwareisreleased.Thescenarioscanalsofocusonspecificaspectsoffunctionalitythatneedtestingincustomerenvironments.Donotexpectmanybetacustomerstobeabletoperformin-depthstresstestingofyoursoftware.Thiscanberathertimeconsumingtoprepareforandexecute,andmostbetacustomerswillnothavethetimeandresourcestodothisleveloftesting.

ThescenariosshouldbemadeavailableatthebeginningofPhaseIIandcanbeintroducedduringthebetaWebinarsorinthefirstcustomercallsoncebetahasbegun.

BetaSoftwareReadiness

Anotherimportanttaskinpreparingforbetaistotrackthereadinessofthebetasoftware.Therearetwomajorquestionstoanswer:

Howcanyoumeasurewhetherthesoftwareisreadytoshipwhenthebetabegins?Whatdoyoudoifyoudeterminethatitisnotready?


Itcomesdowntowhetheryouwanttobedate-driven,quality-driven,orinmostcases,anacceptablemixofthetwo.

Determiningshippingreadinessmeansmeasuringthequalityofthesoftwareanddecidingifitmeetspreset,agreed-uponcriteria.Ifitmeetsthecriteria,itisreadytoship.Ifnot,moreworkneedstobedoneonit.Softwarequalitycanbeanelusiveconcept,sohowdoesonemeasureit?

Thefirstthingthatcomestomindisthenumberofopenbugsandtheirseverity.AlotofdiscussionwilltakeplaceinDevelopmentorQAmeetingsaboutthenumberofcriticalorshowstopperbugsthatareinthecurrentrelease.Manytimesstatementsaremadethatsoundsomethinglike,“WecannotreleasethebetauntilallcriticalbugsarefixedandtherearelessthanXnumberofseverebugs.”,whereXissomearbitrary,butreasonablesoundingnumber.Theproblemwithastatementlikethisisthat,inreality,itisverydifficulttoenforce.

First,asthetargetbetadateapproaches,andifthenumberofcriticalandseverebugsremainshigh,thedefinitionsofbothcriticalandseverewillmagicallyloosen.Whatwasonceacriticalbugtwomonthspriortobetabecomesaseverebugtwoweeksbeforebeta.Andwhatwasonceaseverebug,maybecomeamoderatebug.

Second,whatifallbut“afew”criticalbugscanbefixedbythebetadate,willyoustillshipthebeta?Whatif,insteadofXseverebugs,thereare10%morethanXseverebugs,willyoushipit?Youcanalwaysissueapatchorupdateofthebetasoftwarelater,can’tyou?

Third,thestatementsaboutnumberofbugsareone-dimensional.Thestatementstreatallcriticalbugsthesameandallseverebugsthesame.Inreality,differentfunctionalareasneedtobetrackedseparatelywithmetricsforeacharea.Insomecases,theremaybelowertolerancelevelsacceptableforbugs,inothercases,itmaybehigher.

Overall,thingslikeusability,performance,securityetc.needtobemeasuredandtrackedseparatelywiththeirowncriteriaandrequirements.

Usabilityissuesareanotherfacetofapplicationsthatmustbetrackedseparately.ManyUIsdonotpassthroughrigorousdesign,reviewandimplementationprocesses.TheUIisthefaceoftheproduct,andthebestcanonlybehamperedbyadifficultUI.

WithrespecttoUIsandbetaprograms,rememberthat:

• Thebetaisusuallythefirstexposurethepublichastothesoftwareandtheywillformopinionsofthesoftwarebasedonthebeta.AsignificantpartofthatopinionwillbebasedontheiremotionalexperiencewiththeUI.


• Manyofthesesamepeoplewillbecandidatesformarketingreferencesites.Iftheiremotionalexperienceisnegative,theywilllikelynotwanttobeassociatedwiththeproduct,evenifyoutellthemthateverythingwillbe“fixed”whenthesoftwareisreleased.

• Usabilityreviewsshouldbeperformedonthepre-betasoftwareandacommitmentmadetoaddressthemostimportantissuesbeforethebeta.Thecompletionofthoseissuesbecomesonemetricusedtomeasurereleasereadiness.

Allofthereadinesscriteriashouldbeconvertedtoanumberofmetricsthataretrackedandreviewedviaamanagementdashboard.Thisdashboardshowshigh-levelstatusforeachcategory(e.g.viared,yellow,andgreenindicators)withtheabilitytodrilldownintodetailswhereneeded.Ifavailable,youcanuseadedicatedreportingordashboardingtool.Butifnot,thenvirtuallyanyapplication,suchasaspreadsheet,presentationtoolorwordprocessorwilldo.

Amanagementdashboardprovidesatop-downviewofthestateofproductquality.Alotofredonthedashboardafewweeksbeforebetaissettolaunchmaygiveyouearlywarningthatyouwillneedtoslipthereleasedateorthatadditionalfocusorresourcesareneededtomeetyourexistingdates.

Ifmostareasshowgreen,butafewarepersistentlyredoryellow,itmayindicateissueswiththeteamsworkingonthoseareasoffunctionality,andspecificactioncanbetakentohelpthoseteammembers.Anotherobviousbenefitofsuchadashboardisthatitgivesuppermanagementclearvisibilityintothestatusofthesoftwareandtheabilitytomakebetterandquickerdecisionstoreallocateresourcestowheretheymaybeneeded.


PhaseII-Beta


PhaseII–Beta

LaunchtheBeta

Afterallofthispreparation,youarenowreadytodeliverthesoftwaretocustomersandbegintheactualbetatesting.Inmostcases,customersshouldbeabletoaccessthesoftwareontheirownwithlittleassistancefromyoursupportteam.Soeithertheyshouldcreateaccountsorhavespecialbetaaccountscreatedforthemreadytoaccess.

WorkingwithBetaSites

Themajoraimofthebetaistocollectfeedbackfromthebetasites.Despiteanyassurancestothecontraryfrombetasites,mostsiteswillneedalotofproddingandassistanceduringthebetaprogramifyouwanttheirsubstantivefeedback.

Basedonmyownbetaexperiences,combinedwithsomeanecdotaldatafromotherenterprisesoftwareproductmanagers,roughlyhalfofallbetasites(50%)willendupusingthesoftwareinsomeway,androughlyhalfofthose(25%)willprovidemeaningfulfeedbackonatleastoneareaoffunctionality.

Thus,ifyoustartwith50registeredbetasites,approximately20to30willactuallyusethesoftwareandprovidesomelevelofusefulfeedback,withonlyabout10to15givingdetailedandverymeaningfulfeedback.Ofcourseeverysituationisdifferent,buttheseratioshaveheldtrueformeacrossseveralbetaprograms.

Regardlessofthenumberofactivebetasites,therearemanywaystocollectfeedback.Dependingonyourgoals,youcanusethefollowingtechniques:

• Weeklycallswithindividualparticipants• Regularconferencecallswithmultipleparticipants• Betaparticipantblogand/ordiscussionforums• Asneededemailandphoneconversations

Weeklycallswithindividualparticipantsworkverywell.Althoughitcanbetimeconsumingtoholdweeklymeetingswitheachbetasite,customerswillcommittoaweekly30-minutecall,andinsomecasesa60-minutecall,todiscusstheirissuesandfeedback.Thebenefitoftheweeklycallisthatthebetacustomercanblockoffafixedamountoftimeeachweekandknowstheywillgetyourundividedattentionforthatperiodoftime.

Iftheyhaveproblemswiththebetasoftware,theweeklycallcreatesanopportunitytodiscusstheissuesandgetthecustomerbackontrack.Iftheyaremakingprogress


withthebeta,thisweeklycallgivesyoutheopportunitytodiscusstheirfeedbackingreatdetailandguidethemfurtheralonginthedirectionsyouwantthemtogo.

Inadditiontoweeklycallswithindividualbetacustomers,youcansetupregular,perhapsbiweekly,conferencecallswithmultiplecustomers.Thereisariskofthesecallsdevolvingintoagripesession—itonlytakesoneunhappybetacustomertogetagroupstarted—butifmanagedwell,listeningtocustomersdiscusstheirissueswithoneanothercanleadtosomevaluableinsights.

Anothermechanismforcommunicatingwithbetacustomersistoprovideablogorweb-baseddiscussionforumtoenablethemtoaskandanswerquestions,postfindingsorsimplydiscussrelevantissueswithoneanother.

Thebenefitofprovidinganelectronicforumisthattheinformationisautomaticallypersistedforyou,andifsupportedbythesoftware,customerscanuploadfiles,images,reportsoranyotherdocumentsthatarerelatedtothebetaandtheirfindings.Theseweb-basedforumscanalsobeusedascustomerself-helptools,enablingbetacustomerstoaskandanswerquestionsamongstthemselves.

Lastand—inmyopinion—least,letthecustomercontactyouviaemailorphoneontheirownscheduleandnotsetupformalweeklyorbiweeklycalls.Whileadedicatedbetacustomermaybequiteproactiveandeloquentifleftontheirown,inmyexperience,leavingituptothebetacustomertocontactyouwhentheycanwillleadtolittle,ifany,meaningfulfeedback.

RecordandDisseminateCustomerFeedback

Collectingcustomerfeedbackisonething,butrecordinganddisseminatingitwithinyourorganizationissomethingcompletelydifferent.Rememberthatdifferentinternalgroupsrequirevariouslevelsofdetailofthebetafeedback.

Obviouslythemostdetailedinformationisintheformoftherawnotesandfindingstakenfromweeklycustomercalls.Thisrawfeedbackshouldbepostedinaninternalforum,suchasonawikiorportal,sothatallpartiescanaccessiteasily.Themainconsumersofthislevelofinformationwillmostlikelybeengineersandproductmanagers.Rawfeedbackcontainsvarioustypesofinformationaboutcustomerissues,positiveandnegativecommentsonvariouspartsofthesoftware,accesstotechnicalsupport,plans,expectations,etc.

Theinformationcollectedfromthesecallsshouldbeslicedanddiced,andpresentedinmultiplewayssothatitcanbeefficientlyutilized.Informationcanbeavailableinany(orall)ofthefollowingways:

• Weeklycallnotesbycustomer(i.e.allcallsforindividualcustomers)• Customercallnotesbyweek(i.e.allcallsforagivenweek)


• Detailedfeedbackbyareaoffunctionality• High-levelsummaryofbetafindings• Overallbetacustomerstatusbyweek

Theremaybeotherwaystopresentthedata,butthoselistedabovewillgivemostinterestedpartieswhattheyneed.Accountmanagerswillbeveryinterestedintheweeklyprogressoftheiraccounts.Productmanagersanddevelopmentmanagerswillfocusonthedetailedfeedbackbyareaoffunctionality.Overallbetacustomerstatusbyweekisinterestingtotrack.

Ascustomersbegintheirbetaparticipation,itisimportanttotrackahigh-levelstatusforthemeachweek.Thisstatusshouldbebasedonclearindicatorsoftheirprogressduringthecourseofbeta.Asetofstatuses,includingregistered,accessedsoftware,testinginprogress,blocked,andcompletedbeta,appliedtoeachbetacustomereachweekcanbeavaluabletoolintrackingissuesandprogressduringthebeta.

Thesetypesofsimplemetricscanbeusedtocreatedashboardsorindicatorssoseniormanagementcangetaclearpictureovertimeoftheprogressandstatusofthebetaprogram.

Forexample,if,earlyinthebetaprogram,asignificantnumberofcustomersareblockedbyconfigurationissues,thiswillbeclearlyvisibleinthedashboard.Correctiveactioncanbetaken,butiftheissuepersists,decisionscanbemadetoprovideadditionalsupport,provideon-sitehelpifwarranted,orevenconsiderextendingthebetatomakeupforlosttime.

ProvideTechnicalSupport

Mostsoftwarecompanieshavewell-definedpoliciesandproceduresfordeliveringcustomersupport.Typicallyeachcustomerhassomeformofmaintenanceagreementwiththevendor.Butduringbetaprograms,whereevensmallissuescanblockcustomerprogress,theremayneedtobeasimplifiedandexpeditedmechanismforcustomerstologissuesthatneedtoberesolved.

Ihaveheardmanycustomersrequestthisduringbetaprograms.Thiscouldmeanthatthenormalprocesstologandtroubleshootcasesistoocumbersome,andwhilethatmaybetrue,itcanalsoindicatethatcustomersarenotwillingtoinvestasmuchtimetoresolvebetaissuesastheyareregularsoftwareissues.

Anexpeditedprocessshouldbesetupforbeta.Itcanbeassimpleasadedicatedemailaddressforcustomerstousetoindicateaproblem.Ifyoupreferalittlemoresophistication,itisverysimpletosetupastructuredwebformthatcustomerscanusetosubmitissues.


Anotherideaistosetupashareddiscussionboardforbetacustomerssotheycanlogissues.Resolutionstocommonproblemscanalsobepostedonthediscussionboard.Sometimes,abetacustomerwhologsoneproblemmaybeabletoofferasolutiontoanother.Thisway,notonlycansomeoftheloadbetakenoffofthecustomersupportteam,butbetacustomerscanachievealevelofself-serviceinresolvingproblems.

CustomerVisits

Aseveryproductmanagerknows,thereisnothinglikeavisittoacustomersitetogetadeeperunderstandingofcustomerissues.Additionally,customersoftenlookforwardtovisitsfromproductmanagersandwillputasideseveralhoursoftheirtime,ifneeded,todiscusstheirconcerns.Sowhatbettertimetovisitcustomersanddiscussissuesandconcernsthanduringabetaprogram?

Partofthebetaplanningprocessshouldincludebudgetandgoalsforcustomervisits.Theprimarygoalofthevisitsistogetmoredetailedinformationaboutthebetaexperienceofthesite,butdonotlimitittothat.Thevisitisagreatopportunitytodiscussfutureplansforyourproductintheirorganization.Youcanalsouncovernewprojectsorinitiativeswhereyourproductcanbeusedandevenmeetotherpeopleintheorganizationwhocaninfluencetheuseofyourproductinthecompany.

Beverycarefultokeepthetoneofthevisitasaresearchvisitandnotturnitintoasalesorprospectingcall.Asaproductmanager,youhavealotofcredibilityandinfluencewiththecustomer,sodon’tjeopardizethatbyfocusingonsalesobjectivesinsteadofthecustomerneeds.

DevelopReferenceSites

Inmyexperience,thereisnodirectrelationshipbetweensomeonehavingapositivebetaexperienceandtheirabilitytobecomeareferencesite.Certainlyapositiveexperienceimprovesthechancesofsomeonebecomingareference,buttherearenumerousfactorsthatcanpreventthat.

Manycompanieshavestrictpoliciesabouttheiremployeesspeakingpubliclyonbehalfofavendor.Forthosecompaniesthatallowtheiremployeestospeakpublicly,thereareusuallystrictguidelinesastowhattheycansayandtowhomtheycansayit.Thosewhodospeakusuallydosobecausetheyhaveaverygoodpersonalrelationshipwithindividualsatyourcompany,anddonotnecessarilyneedtobeactivelyinvolvedinafull-blownbetaprogramtoassesstheproductanddeterminetheiropinion.Perhapsalivedemooracustomerdayattheirsitecanbe


sufficienttogetthemacquaintedandcomfortableenoughwithyourproducttospeakaboutit.

Thepointisthatyoushouldnotdependsolelyonthebetaprogramtodeveloppressandanalystreferencesforyourproductlaunch.Certainly,somebetasitescanturnintoreferencesifyourbetaprogramprogresseswell,andthosewhohavegoodbetaexperiencesalsohavetherightprofile,authorityandpermissiontospeakonbehalfoftheircompanyaboutyourproduct.

But,thatleavesfartoomuchtochance.Whatifveryfewbetasiteshaveagoodexperience?Whatifthosesitesthatdohaveagoodexperiencewiththebetadonotfittheprofileoftypeofreferencesiteyouwant?Whatifthosewhodofittheprofileandhaveagoodexperiencedonothavethepermissiontospeakpublicly?

Tomitigatetheserisks,youneedtohaveaseparatebutparallelefforttoidentifypotentialreferencesitesandworkwiththemtogetthemtoapointthattheywillbecomereferences.ThiseffortistypicallyperformedbyMarketingorProductMarketingandnotbyProductManagement.

Asmentionedearlier,thereferenceprospectsmayalreadyhaveapersonalrelationshipwithindividualsinyourcompanyandmaynotneedfullbetaexposuretogaintheproductknowledgetheyneedinordertospeakaboutit.Ifsomeofthesepotentialreferencesitesdorequirebetaparticipationbeforebecomingreferences,thenletthemparticipate.Butoncetheyenterthebeta,allinteractionwiththemshouldbehandledwiththefullknowledgethattheyareapotentialreferencesite,andnotasitethatwillnecessarilyprovidedetailedfeedbackonfeaturesandfunctionality.

ExtendingtheBeta

Dependingonhowyourbetaprogramprogresses,youmaywantorneedtoextendthetimeperiodoverwhichyouholdthebeta.Sometimescustomersstartlateinthebetacycleandwantadditionaltimetoprovidefeedbackorthebetaprogramidentifiesanumberofissuesintheproduct,andyourealizethatthoseissuesareimportantenoughthatmoreproductfeedbackisneededontheproduct.Regardlessofthereason,itisimportanttoknowthatthisisapossibilityandtoprepareforitinadvance.

Ifyouembedatime-boundlicensewithyourbetasoftware,makesurethatyoucanupdatethatlicenseifyouextendthebeta.Ifyoucannotextenditafterthefact,buildadditionaltimeintotheoriginallicensesothatifyouextendthebeta,thosecustomerswhodocontinuetousethesoftwarefortheextendedperiodcandosowithoutinterruption.


Also,makesurethattheinternalresourcessuchasTechnicalSupportandProfessionalServiceswhowillbeassistingcustomersinthebetaareavailablebeyondtheinitialbetaperiod,ifneeded.Theirmanagersmaynotbeawareofthepotentialextendedtimeframeandmayhaveotherplansforthemoncetheregularbetaperiodends.


PhaseIII–PostBeta


PhaseIII–Post-Beta

Finalcustomercalls

DuringPhaseIIofthebeta,thereisregularcustomercontactfocusingonprogresswiththebeta,withthesuppliedusecasesandanyissuesthatthecustomerfaceswiththeproduct.Thisistypicallytacticalinformationfocusedonthecurrentweekorstageofbetacustomerparticipation.

Tocloseoutthebetaprogramwithcustomers,youshouldholdafinalwrapupcalltodiscusshigher-levelissuesandobtainfeedbackontheoverallbetaprogram.Thisinformationisextremelyvaluableinassessingoverallbetasuccessandtoidentifythedo’sanddon’tsforfuturebetaprograms.

ThefollowingisashortlistofquestionsIhaveaskedcustomersatthecloseofbetaprograms.Notethatthesearediscussionpoints,andnotsurveyquestions,anditisuptoyoutodrilldownwiththebetacustomertodrawoutadditionaldetails,whereneeded.

• Didthetrainingprovidesufficientinformationtohelpyouunderstandthecapabilitiesofthebetasoftware?

• Weretheweeklyconferencecallshelpfultoyou?o Whatwashelpfulaboutthem?o Whatwasnothelpful,andwhatcouldweimprove?

• Whatcouldbedonetomakethetimeonthecallsmoreproductive?• Didyoufindtheprocessofrequestingsupportintuitiveand

straightforward?• Didthetechnicalsupportprocessmeetyourneeds?• Howcouldsupportbeimprovedforfuturebetaprograms?• Didyouhaveadequatehardwaretotestoutthebeta?• Wereyouabletoeasilygetupandrunningwiththebetasoftware?

o Whatcouldbedonetomaketheprocesseasier?• Foreachareaoffunctionalitytested,whatwerethestandoutpositivesand• Negatives?

o Whatchanges/enhancementsarecriticaltotheinitialreleaseoftheo product?

• Doyouplantoupgradetothenewrelease?Ifso,when?Ifnot,whynot?

Giventhedetailofsomeofthesequestions,youshouldscheduleatleastonehourforthesecallsandideallysendthequestionsinadvancetothebetacustomers.Gettingthequestionsinadvancewillletthemcollectdetailedandaccurateanswers


tothesequestionsand,ifneeded,polltheircoworkerswhoalsoparticipatedinthebetaprogram.

SummarizeandAddressBetaFindings

Oncethebetaisoverandthefinalcustomercallsarewrappedup,allthefindingsneedtobeconsolidatedandsummarizedintoproductandprocesscategories.

Theproductfeedbackfromthebetamustbeconvertedintospecificrequirementsthatneedtobeaddressed.Thecriticalrequirementscanbeaddedtoyouroverallreleasecriteriafortheproduct.Thatis,thesoftwarecannotbereleasedtomarketwithoutaddressingthesecriticalneeds.Theremainingrequirementscanbescheduledforfuturefeatureormaintenancereleases,andtheirprioritycanbeassessedinthecontextofotherrequirementsforthosereleases.

Areportshouldbecreatedthatsummarizesthefindingsandanyothernotabledetailsgleanedfromthebeta.ThisreportisintendedforSr.Managementorothergroupswhoareinterestedinthebetabutdon’tneedthedetailedproductrequirementsderivedfromthebetafeedback.

InternalRetrospecitveSessionsandPublishingResults

Theprocessfindingsalsoneedtobeunderstoodandimplemented.Butunliketheproductfindingsthatwillcomealmostexclusivelyfrombetacustomers,theprocessfeedbackwillhavebothanexternalaswellasinternalcomponent.

Toobtaintheinternalbetaprocessfeedback,betaretrospectiveorreviewsessionsneedtobeheld.ThesesessionsshouldbeledbyProductManagement(orProgramManagement)andbeheldwithindividualteamssuchasTechnicalSupport,Marketing,QualityAssurance,andDevelopment.

Thereasontoholdthemwithseparateteamsandnotinonelargesessionissothediscussioncanremainfocusedandcandrillintodetailontheissuesforeachgroup.

Oncetheindividualsessionsareheld,thefindingsfromallthegroupscanbecombinedandpresentedbackinajointmeetingtotheinternalteams.Thisfinalmeetinghelpsbringallgroupstogetherandeveryonecancollectivelydiscusswhatshouldandshouldnotbedoneinfuturebetaprograms.


ConclusionExecutingasuccessfulbetaprogramrequiresapproachingitinaveryanalyticmanner.Youmustfocusonminimizingtheinherentuncertaintyincollectingfeedbackonapre-releaseproductfromverybusycustomersandpartners.Noonewillarguethatbetaprogramsareakeyelementofthesoftwaredevelopmentprocess,butplanningforandexecutingasuccessfulbetaprogramisadifficulttask.Bytakingarigorousandmethodicalapproach,andworkingtomakeeachindividualbetacustomersuccessful,youincreasethelikelihoodofmeetingyourgoals,deliveringnecessarymarketfeedbacktoyourorganization,andsignificantlyenhancingthechancesofsuccessforyourproduct.


ContactUsDoyouhavequestionsaboutbetaprogramsorothertopicsrelatedtoProductManagement,ProductMarketing,Innovation?

Feelfreetocontactus.We’realwaysinterestedinhearingyourquestionsandseeingifwecanhelpyouout.

Anddon’tworry,wewon’tchargeyouanythingjustforaskingquestions.☺

Wehelpcompaniesbuildbetterproductsandproductorganizations.Letushelpyou.

http://www.transformationlabs.io

[email protected]

building a better beta · • something that should be confined to the product management or...

Documents