tpc express benchmarktm iot (tpcx-iot) standard ... · includes andy bond, bhaskar gouda, karthik...

TPC Express Benchmark IoT (TPCx-IoT) Specification 1

TPC EXPRESS BENCHMARKTM IoT (TPCx-IoT)

Standard Specification Version 1.0.1

September,2017

TransactionProcessingPerformanceCouncil(TPC)

www.tpc.org

mailto:[email protected]

©2017TransactionProcessingPerformanceCouncil

AllRightsReserved


Legal Notice The TPC reserves all right, title, and interest to this document and associated source code as provided under U.S. and international laws, including without limitation all patent and trademark rights therein. Permission to copy without fee all or part of this document is granted provided that the TPC copyright notice, the title of the publication, and its date appear, and notice is given that copying is by permission of the Transaction Processing Performance Council. To copy otherwise requires specific permission. No Warranty TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, THE INFORMATION CONTAINED HEREIN IS PROVIDED “AS IS” AND WITH ALL FAULTS, AND THE AUTHORS AND DEVELOPERS OF THE WORK HEREBY DISCLAIM ALL OTHER WARRANTIES AND CONDITIONS, EITHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, ANY (IF ANY) IMPLIED WARRANTIES, DUTIES OR CONDITIONS OF MERCHANTABILITY, OF FITNESS FOR A PARTICULAR PURPOSE, OF ACCURACY OR COMPLETENESS OF RESPONSES, OF RESULTS, OF WORKMANLIKE EFFORT, OF LACK OF VIRUSES, AND OF LACK OF NEGLIGENCE. ALSO, THERE IS NO WARRANTY OR CONDITION OF TITLE, QUIET ENJOYMENT, QUIET POSSESSION, CORRESPONDENCE TO DESCRIPTION OR NON-INFRINGEMENT WITH REGARD TO THE WORK. IN NO EVENT WILL ANY AUTHOR OR DEVELOPER OF THE WORK BE LIABLE TO ANY OTHER PARTY FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO THE COST OF PROCURING SUBSTITUTE GOODS OR SERVICES, LOST PROFITS, LOSS OF USE, LOSS OF DATA, OR ANY INCIDENTAL, CONSEQUENTIAL, DIRECT, INDIRECT, OR SPECIAL DAMAGES WHETHER UNDER CONTRACT, TORT, WARRANTY, OR OTHERWISE, ARISING IN ANY WAY OUT OF THIS OR ANY OTHER AGREEMENT RELATING TO THE WORK, WHETHER OR NOT SUCH AUTHOR OR DEVELOPER HAD ADVANCE NOTICE OF THE POSSIBILITY OF SUCH DAMAGES. Trademarks TPC Benchmark and TPC Express are trademarks of the Transaction Processing Performance Council.


Acknowledgments Developing a TPC benchmark for a new environment like the Internet of Things (IoT) required a huge effort to work and contributions of the TPCx-IoT subcommittee member conceptualize research, specify, review, prototype, and verify the benchmark. The TPC acknowledges the companies in developing the TPCx-IoT Specification. The list of contributors to this version includes Andy Bond, Bhaskar Gouda, Karthik Kulkarni, Chaitanya Kundety, Chinmayi Narasimhadevara, Da Qi Ren, David Grimes, Meikel Poess, Nicholas Wakou, Jamie Reding, John Poelman, Ken Rule, Hamesh Patel, Mike Brey, Matthew Emmerton, Paul Cao, Reza Taheri, and Tariq Magdon-Ismail. Document Revision History Table 1: Document Revision History Date Version Description 06/07/2017 1.0.0 Draft proposed for GC

approval with all changes since formal review

09/21/2017 1.0.1 Editorial fixes. Workload is modified to include analytics query over randomly selected interval


TPC Membership TPC membership as of September 2017.


TableofContents

Clause1Introduction.........................................................................................................................71.1Preamble................................................................................................................................................71.2TPCx-IoTKitandLicensing......................................................................................................................71.3GeneralImplementationGuidelines......................................................................................................71.4GeneralMeasurementGuidelines.........................................................................................................9

Clause2:WorkloadandExecution...................................................................................................102.1TPCx-IoTKit..........................................................................................................................................102.1.1KitContents.......................................................................................................................................102.1.2TPCx-IoTKitUsage.............................................................................................................................102.1.3KitModification.................................................................................................................................10

2.1.3.1MinorShellScriptModifications................................................................................................102.1.3.2MajorShellScriptModifications................................................................................................112.1.3.2JavaCodeModifications............................................................................................................112.1.4FutureKitReleases........................................................................................................................11

2.2BenchmarkWorkload.........................................................................................................................112.3BenchmarkExecution...........................................................................................................................122.4ConfigurationandTuning.....................................................................................................................13

Clause3:SystemUnderTestandBenchmarkDriver......................................................................153.1SystemUnderTest...............................................................................................................................15

Clause4:ScaleFactorandMetrics...................................................................................................174.1ScaleFactor..........................................................................................................................................174.2Metric...................................................................................................................................................174.3PerformanceMetric.............................................................................................................................174.4PricePerformanceMetric.................................................................................................................174.5AvailabilityDate...................................................................................................................................174.6MetricComparison...............................................................................................................................174.7RequiredReportingComponents......................................................................................................17

Clause5:Pricing................................................................................................................................195.1PricedSystem.......................................................................................................................................195.2AllowableSubstitutions........................................................................................................................19

Clause6:FullDisclosureReportandExecutiveSummary...............................................................216.1ReportingRequirements......................................................................................................................216.2FormatGuidelines................................................................................................................................216.3FullDisclosureReport...........................................................................................................................216.4GeneralItems.......................................................................................................................................216.5WorkloadRelatedItems.......................................................................................................................246.6AuditRelatedItems..............................................................................................................................246.7ExecutiveSummary..............................................................................................................................246.8ImplementationOverview...................................................................................................................256.9PricingSpreadsheet..............................................................................................................................27


6.10NumericalQuantitiesSummary.........................................................................................................276.11TPCx-IoTRunReport..........................................................................................................................27

TherunreportfromTPCx-IoTmustbeincludedintheExecutiveSummary..................................276.12 AvailabilityoftheFullDisclosureReport......................................................................................276.13RevisionstotheFullDisclosureReport..............................................................................................28

Clause7:Audit..................................................................................................................................297.1GeneralRules.......................................................................................................................................29

ThePre-PublicationBoardconsistsofthreemembersoftheTPCx-IoTcommittee...............................297.2AuditCheckList....................................................................................................................................30

7.2.1Clause2:WorkloadandExecutionRelatedItems.........................................................................307.2.2Clause3:SystemUnderTestandDriverRelatedItems.................................................................307.2.3Clause4:ScaleFactorsandMetricsRelatedItems.......................................................................307.2.4Clause5:PricingRelatedItems.....................................................................................................307.2.5Clause7:FullDisclosureRelatedItems.........................................................................................30

Clause8:SampleExecutiveSummary..............................................................................................31


Clause1Introduction1.1Preamble InternetofEverything(IoT)representsaglobalmarkettransitiondrivenbyasurgeinconnectionsamongpeople,processesandthings.IoTisbeingadoptedacrossalmosteveryindustrytriggeringamassiveinfluxofdatathathastobeanalyzedforinsights.TypicalIoTtopologyconsistsofthreetiers:edgedevices,gatewaysystemsandbackenddatacenter.Whilethereexistworkloadsforbackenddatacenter,therearenorealisticandprovenmeasurestocomparedifferentsoftwareandhardwaresolutionsforgatewaysystems.Toaddressthis,TPChasdevelopedTPCExpressBenchmarkTMIoT(TPCx-IoT).

TPCx-IoTprovidesanobjectivemeasureofhardware,operatingsystem,datastorageanddatamanagementsystemstoprovidethe industrywithverifiableperformance,price-performanceandavailabilitymetricsforsystemswhicharemeanttoingestandpersistmassiveamountsofdatafromlargenumberofdevices,andprovidereal-timeinsights,typicalinIoTgatewaysystemsrunningcommerciallyavailablesoftwareandhardware.

TheTPCx-IoTbenchmarkmodelsacontinuoussystemavailable24hoursaday,7daysaweek.The TPCx-IoT canbeused to assess a broad rangeof system topologies and implementationmethodologiesinatechnicallyrigorous,directlycomparable,vendor-neutralmanner.

1.2TPCx-IoTKitandLicensingTPCx-IoTisaTPCExpressbenchmarkandafullkit(TPCx-IoTKit)isprovidedbytheTPC.Vendorsare required to use this Kit for benchmark publications. The Kit includes a set of scripts togeneratedatasimulatingIoTsensors,datainject,analytics,calculatethemetricsandvalidation.

ThedatageneratedisingestedandpersistedintotheSystemUnderTest(SUT)andcontinuouslyqueriedtosimulatesimpleanalyticsusecases.TheSystemUnderTest(SUT)representsanIoTgateway system consisting of commercially available servers and storage systems running acommerciallyavailableNoSQLdatamanagementsystem.

TheKitisavailableatTPCDownloadspage.Usersmustsign-upandagreetotheTPCx-IoTUserLicensingAgreement(ULA)todownloadtheKit.

Toaddsupportforanewdatabase,followtheinstructionsinthe‘HowtoAddaNewDatabase’documentincludedintheKit.

1.3GeneralImplementationGuidelinesThepurposeofTPCbenchmarksaretoproviderelevant,objective,andverifiableperformance


datatoindustryusers.Toachievethatpurpose,theTPCBenchmarkSpecificationsrequirethatbenchmarktestsbeimplementedwithsystems,products,technologiesandpricingthat:

• Arecommerciallyavailable;• Aregenerallyavailabletoallusers;• ArerelevanttothemarketsegmentthattheindividualTPCbenchmarkmodels;• Wouldplausiblybeimplementedbyasignificantnumberofusersinthemarket

segmentthebenchmarkmodels.

Theuseofnewsystems,products,technologies(softwareorhardware)solongastheymeetthe requirements above. Specifically prohibited are benchmark systems, products,technologiesorpricing(hereafterreferredtoas"implementations")whoseprimarypurposeisperformanceoptimizationofTPCbenchmarkresultswithoutanycorrespondingapplicabilityto real-world applications and environments. In other words, all "benchmark special"implementationsthatimprovebenchmarkresultsbutnotreal-worldperformanceorpricing,areprohibited.The following characteristics shall be used as a guide to judge whether a particularimplementationisa“benchmarkspecial”implementation.It isnotrequiredthateachpointbelowbemet,butthatthecumulativeweightoftheevidencebeconsideredto identifyanunacceptableimplementation.Absolutecertaintyorcertaintybeyondareasonabledoubtisnotrequiredtomakeajudgmentonthiscomplexissue.Thequestionthatmustbeansweredis: "Based on the available evidence, does the clear preponderance (the greater share orweight) of evidence indicate that this implementation’s primary purpose is performanceoptimizationofTPCbenchmarkresultswithoutanycorrespondingapplicabilitytoreal-worldapplicationsandenvironments?"Thefollowingcharacteristicsshallbeusedtomakethisjudgment:

• Istheimplementationgenerallyavailable,externallydocumentedandsupported?• Doestheimplementationhavesignificantrestrictionsonitsuseorapplicabilitythat

limitsitsusebeyondTPCx-IoTbenchmark?• Istheimplementationorpartoftheimplementationpoorlyintegratedintothelarger

product?• DoestheimplementationtakespecialadvantageofthelimitednatureoftheTPCx-IoT

benchmarkinamannerthatwouldnotbegenerallyapplicabletotheenvironment


thebenchmarkrepresents?• Istheuseoftheimplementationdiscouragedbythevendor?(Thisincludesfailingto

promotetheimplementationinamannersimilartootherproductsandtechnologies.)

• Doestheimplementationrequireuncommonsophisticationonthepartoftheend-user,programmerorsystemadministrator?

• Istheimplementation(includingbeta)beingpurchasedorusedforapplicationsinthemarketareathebenchmarkrepresents?Howmanysitesimplementedit?Howmanyend-usersbenefitfromit?Iftheimplementationisnotcurrentlybeingpurchasedorused,isthereanyevidencetoindicatethatitwillbepurchasedorusedbyasignificantnumberofend-usersites?

• TherulesforpricingareincludedintheTPCPricingSpecificationlocatedattheTPCDocumentationwebpage.

1.4GeneralMeasurementGuidelinesTPC benchmark results are expected to be accurate representations of systemperformance.Therefore,therearecertainguidelinesthatareexpectedtobefollowedwhenmeasuringthoseresults. The approach ormethodology to be used in themeasurements are either explicitlydescribedintheSpecificationorlefttothediscretionofthetestsponsor.Whennotdescribedinthe Specification, the methodologies and approaches used must meet the followingrequirements:

• Theapproachisanacceptedengineeringpracticeorstandard.• Theapproachdoesnotenhancetheresult.• Equipmentusedinmeasuringtheresultsiscalibratedaccordingtoestablishedquality

standards.Fidelityandcandorismaintainedinreportinganyanomaliesintheresults,evenifnotspecifiedintheTPCbenchmarkrequirements.


Clause2:WorkloadandExecution Thisclausedefinesworkloadandexecution. 2.1TPCx-IoTKit ThefollowingsectionsprovidesthecontentsofthebenchmarkTPCx-IoTKitandusageguidelines. 2.1.1KitContentsTheTPCx-IoTKitcontainsthefollowing:

• TPCx-IoTSpecification(thisdocument)• TPCx-IoTUserGuide• Adocumentwithinstructionsonhowtoaddanewdatabase• DriverProgram• Scriptstosetupthebenchmarkenvironment,capturesysteminventory,runthe

benchmark,andvalidatetherun• Javacodetoexecutethebenchmarkload

2.1.2TPCx-IoTKitUsageTosubmitacompliantTPCx-IoTbenchmarkresult,thetestsponsorisrequiredtousetheTPCx-IoTkKtasprovidedexceptformodificationsexplicitlylistedinClause2.1.3TheKitmustbeusedasoutlinedintheTPCx-IoTUserGuide.TheoutputoftheKitiscalledtherunreportwhichincludesthefollowing:

• VersionnumberofKit• ChecksumfortheTPCx-IoTprograms• Validationforcompliance(numberofrecordsingested,datareplicationfactor)• Verificationofdata

IfthereisaconflictbetweentheTPCx-IoTSpecificationandtheTPCprovidedcode,theTPCprovidedcodeprevails.

2.1.3KitModification2.1.3.1MinorShellScriptModifications MinormodificationstotheprovidedshellscriptsintheTPCx-IoTKittofacilitateoperatingsystemdifferencesorthestoragethatisbeingusedareallowedwithoutTPCapproval.


Thefollowingchangesareconsideredminormodifications:• Shellscriptchangesnecessaryforthescriptstoexecuteonaparticularoperatingsystem

aslongasthechangesdonotaltertheexecutionlogicofthescript2.1.3.2MajorShellScriptModifications MajormodificationsmustbeapprovedbytheTPCpriortobeingusedinabenchmarksubmission.ItwillbethejudgmentoftheTPCmembersreviewingthesubmissionortheTPCx-IoTcertifiedauditor(ifbeingused)astowhetherscriptingchangesareconsideredminorormajor.IfthetestsponsorhasanydoubtstheyareencouragedtohavethechangesapprovedbytheTPCpriortobeingusedinasubmission.2.1.3.2JavaCodeModificationsNomodificationsareallowedtothejavacodeprovidedintheTPCx-IoTKit. 2.1.4FutureKitReleasesTheTPCwillreleasefutureTPCx-IoTKitatitsdiscretiontofixbugsoraddfeatures.WhenanewKitversionisreleasedtheTPCwillreleaseatimetableregardingthelastdateabenchmarksubmissioncanbemadeusingthepreviousKitversion.Afterthisdate,onlysubmissionsusingthenewKitversionwillbeconsidered,andsubmissionsusingthepreviousKitversionwillimmediatelybefoundnon-compliant.IfthetestsponsorwouldlikenewscriptsorexistingscriptchangestobeincludedinafuturereleaseoftheKit,thenthetestsponsorcandonatethescriptsorscriptcodechangestotheTPCandworkwiththeTPCtogetthemincludedinthenextrelease.IfatestsponsorwouldliketoseechangesmadetothejavacodeoftheKit,thenthechangesshouldbeprovidedtotheTPCforpotentialinclusioninthenextreleaseoftheKit. 2.2BenchmarkWorkloadTheTPCBenchmarkTMIoT(TPCx-IoT)benchmarkworkloadisdesignedbasedonYahooCloudServingBenchmark(YCSB)1.ItisnotcomparabletoYCSBduetosignificantchanges.TheTPCx-

1 Yahoo! Cloud Serving Benchmark (YCSB) References (i) Wikipedia: https://en.wikipedia.org/wiki/YCSB (ii) Benchmarking Cloud Serving Systems with YCSB: https://www.cs.duke.edu/courses/fall13/cps296.4/838-CloudPapers/ycsb.pdf (iii) YCSB+T: Benchmarking web-scale transactional databases: https://www.computer.org/csdl/proceedings/icdew/2014/3481/00/06818330.pdf


IoTworkloadsconsistsofdataingestionandconcurrentqueriessimulatingworkloadsontypicalIoTGatewaysystems.Thedatasetrepresentsdatafromsensorsfromelectricpowerstation(s).Thedataingestionandqueryworkloadsaredetailedinthefollowingsection.

Eachrecordgeneratedconsistsofdriversystemid,sensorname,timestamp,sensorreading

andpaddingtoa1Kbytesize.Thedriversystemidrepresentsapowerstation.Thedataset

representsdatafrom200differenttypesofsensors.TheSUTmustrunadatamanagement

platformthatiscommerciallyavailableanddatamustbepersistedinanon-volatiledurable

mediawithaminimumoftwo-wayreplication.Theworkloadrepresentsdatainjectintothe

SUTwithanalyticsqueriesinthebackground.Theanalyticqueriesretrievethereadingsofa

randomlyselectedsensorfortwo30secondtimeintervals,TI1andTI2.ThefirsttimeintervalTI1

isdefinedbetweenthetimestampthequerywasstartedTsandthetimestamp30secondsprior

toTS,i.e.TI1=[TS-30,TS].Thesecondtimeintervalisarandomlyselected30secondstime

intervalTI2withinthe1800secondstimeintervalpriortothestartofthefirstquery,TS-30.If

TS<=1830,thenthetimeintervalis1730secondspriortothestartofthefirstquery,TS-30.

2.3BenchmarkExecution DataingestionandqueryareperformedagainsttheSUTbythedriverprogramincludedintheTPCx-IoTKit. Thebenchmarktestconsistsoftworuns,Run1andRun2.EachrunconsistsofaWarmupRunandMeasuredRun.Noactivities otherthandatabasecleanuptriggeredbythecontrolscriptsareallowedbetweenWarmupRunandMeasuredRun.NoactivitiesareallowedbetweenRun1andRun2.ThetotalelapsedtimeforthePerformancerun,inseconds(T),isusedforthePerformanceMetriccalculation.ThePerformanceRunisdefinedastheMeasuredRunwiththelowerPerformanceMetric.TheReportedPerformanceMetricisthePerformanceMetricforthePerformanceRun.Noconfigurationortuningchangesareallowedbetweentheruns.ThebenchmarkexecutionphasesareshownintheFigure1.


Figure1:BenchmarkExecutionPhases Comment:NopartoftheSUTanddriver(s)mayberebootedorrestartedduringorbetweentheruns.Ifthereisanon-recoverableerrorreportedbyanyoftheapplications,operatingsystem,orhardwareinanyofthephasesorbetweenRun1andRun2,therunisconsideredinvalid.Ifarecoverableerrorisdetectedinanyofthephases,andisautomaticallydealtwithorcorrectedbytheapplications,operatingsystem,orhardwarethentherunisconsideredvalid,providedtherunmeetsallotherrequirements.However,manualinterventionbythetestsponsorisnotallowed.Iftherecoverableerrorrequiresmanualinterventiontodealwithorcorrect,thentherunisconsideredinvalid. 2.4ConfigurationandTuning TheSUTcannotbereconfigured,changed,orre-tunedbythetestsponsorduringorbetweenanyofthephasesorbetweenRun1andRun2.AnymanualtuningstotheSUTmustbe


performedbeforethebeginningofPhase1ofRun1,andmustbefullydisclosed.Automatedchangesandtuningperformedbetweenanyofthephasesareallowed.Anychangestodefaulttuningsorparametersoftheapplications,operatingsystems,orhardwareoftheSUTmustbedisclosed.


Clause3:SystemUnderTestandBenchmarkDriverThisclausedefinestheSystemUnderTest(SUT)andthebenchmarkdriver.

3.1SystemUnderTest TheSUTiscomposedofthosesoftwareandhardwarecomponentsthatareemployedintheperformancetestandwhoseperformanceandcostaredescribedbythebenchmarkmetrics.SeeFigure2.Specifically,theSUTconsistsof:

• Devices,forexamplecomputedevicesand/ordatastoragedevices,includinghardwareandsoftwarecomponents,

• AnyhardwareandsoftwaredevicesofallnetworksrequiredtoconnectandsupporttheSUTsystems,

• Eachcomputedeviceincludesabenchmarkspecificsoftwarelayer,thebenchmarkimplementation,andothercommerciallyavailablesoftwareproducts,

Thebenchmarkdriver(s)mayresideononeofthecomputedevicesoronaseparatesystem.Incasethedriverresidesonaseparatecomputedevice,thisdeviceisnotconsideredaspartoftheSUT.

Comment:Exceptforthebenchmarkimplementationandthebenchmarkdriver,allSUTcomponentsmustbecommerciallyavailablesoftwareorhardwareproducts.


Figure2:SystemUnderTest(SUT)

Comment:Thesourcecodeofanynon-commerciallyavailablecomponentsusedtoimplementtheSUT(suchasscriptstoconfigurethesystem,settunables,etc.)mustbeincludedintheFullDisclosureReport.SeeClause6.Comment:Thedriver(s)presenttheworkloadtotheSUT.


Clause4:ScaleFactorandMetricsThisclausedefinesScaleFactorandMetrics. 4.1ScaleFactorThecurrentversionoftheTPCx-IoTKitfollowsacontinuousScaleFactor.ScaleFactorisdefinedasthenumberofrecordstobeingested.BenchmarksponsorcanpickanyScaleFactor.4.2MetricTPCx-IoTdefinesthefollowingprimarymetrics:

1. IoTps,thePerformanceMetric2. $/IoTps,thePrice-Performancemetric3. Systemavailabilitydate

4.3PerformanceMetricTheperformancemetricrepresentstheeffectivethroughputcapabilityoftheSUT

IoTps=SF/(T)WhereSFistheScaleFactor.SeeClause4.1.Tisthetimeelapsedinseconds

4.4PricePerformanceMetricTheprice-performancemetricisdefinedas:

$/IoTps=P/IoTpsPisthetotalcostofownershipoftheSUT.

4.5AvailabilityDate TheAvailabilityDateisdefinedintheTPCPricingSpecificationlocatedattheTPCDocumentationwebpage. 4.6MetricComparison ATPCx-IoTResultisonlycomparablewithotherTPCx-IoTResults.

4.7RequiredReportingComponentsTobecompliantwiththeTPCx-IoTSpecificationandTPCPolices,theURLtothebenchmarkresultandAvailabilityDateofthecompleteconfigurationmustbeincludedforallpublic


references.TPCPoliciescanbefoundattheTPCDocumentationwebpage.


Clause5:PricingRulesforpricingthePricedSystemandassociatedsoftwareandmaintenanceareincludedintheTPCPricingSpecificationlocatedattheTPCDocumentationwebpage. 5.1PricedSystem Thesystemtobepricedshall include thesoftwareandhardwarecomponentspresent in theSystemUnderTest(SUT)andmaintenance.Calculationofthepricedsystemconsistsof:

• PriceoftheSUTasinClause3;• Priceofadditionalproducts(softwareorhardware)presentinthesystem;• Priceofadditionalproducts (softwareorhardware) required forcustomaryoperation,

administrationandmaintenanceoftheSUTforaperiodof3years.• Priceofallproductsrequiredtocreate,execute,administerandmaintaintheexecutables

ornecessarytocreateandpopulatethetestenvironment.Specificallyexcludedfromthepricedsystemcalculationare:

o End-usercommunicationdevicesandrelatedcables,connectors,andswitcheso Equipment and tools used exclusively in the production of the Full Disclosure

Report.5.2AllowableSubstitutionsSubstitutionisdefinedasadeliberateacttoreplacecomponentsofthePricedConfigurationbythetestsponsorasaresultoffailingtheavailabilityrequirementsoftheTPCPricingSpecificationorwhenthepartnumberforacomponentchanges.Comment:Correctionsor"fixes"tocomponentsofthePricedConfigurationareoftenrequiredduringthelifeofproducts.ThesechangesarenotconsideredSubstitutionssolongasthepartnumberofthepricedcomponentdoesnotchange.SuppliersofhardwareandsoftwaremayupdatethecomponentsofthePricedConfiguration,buttheseupdatesmustnotnegativelyimpactthereportedperformancemetricornumericalquantitiesmorethantwopercent.Thefollowingarenotconsideredsubstitutions:

• softwarepatchestoresolveasecurityvulnerability• siliconrevisiontocorrecterrors• newsupplieroffunctionallyequivalentcomponents(forexamplememorychips,disk

drivesetc.)


SomehardwarecomponentsofthePricedConfigurationmaybesubstitutedafterthetestsponsorhasdemonstratedtotheauditor'ssatisfactionthatthesubstitutingcomponentsdonotnegativelyimpactthereportedperformancemetricornumericalquantities.AllsubstitutionsmustbereportedintheFullDisclosureReportandnotedintheauditor'sattestationletter.Thefollowinghardwarecomponentsmaybesubstituted:

• Durablemedium(forexamplediskdrives)andcablesComment:DurableMediumisdefinedasadatastoragemediumthatisinherentlynon-volatilesuchasamagneticdiskortape.Comment:IfanyhardwarecomponentissubstitutedthentheresultmustbeauditedbyaTPCcertifiedAuditororPre-CertificationBoard.SeeClause7.


Clause6:FullDisclosureReportandExecutiveSummary6.1ReportingRequirementsAFullDisclosureReportandExecutiveSummaryinpdfarerequiredaspartofthebenchmarksubmission.Theintentofthisdisclosureistosimplifycomparisonbetweenresultsandforacustomertobeabletoreplicatetheresultsofthisbenchmarkgivenappropriatedocumentationandproducts. 6.2FormatGuidelinesWhileestablishedpracticeorpracticallimitationsmaycauseaparticularbenchmarkdisclosuretodifferfromtheexamplesprovidedinvarioussmallways,everyeffortshouldbemadetoconformtotheformatguidelines.Theintentistomakeitaseasyaspossibleforareviewertoread,compareandevaluatematerialindifferentbenchmarkdisclosures.Allsectionsofthereport,includingappendices,mustbeprintedusingfontsizesofaminimumof8points.TheExecutiveSummarymustbeincludednearthebeginningoftheFullDisclosureReport. 6.3FullDisclosureReportTheFullDisclosureReportshouldbesufficienttoallowaninterestedreadertoevaluateand,ifnecessary,recreateanimplementationofresult.IfanysectionsintheFullDisclosureReportrefertoanothersectionofthereport,thenamesofthereferencedscripts/programsmustbeclearlylabeledineachsection.Unlessexplicitlystatedotherwise“disclosed”referstodisclosedintheFullDisclosureReport.Comment:Sincethebuildingtestenvironmentmayconsistofasetofscriptsandcorrespondinginputfiles,itisimportanttodiscloseandclearlyidentify,byname,scriptsandinputfilesintheFullDisclosureReport.Theorderandtitlesofsectionsinthetestsponsor'sFullDisclosureReportmustcorrespondwiththeorderandtitlesofsectionsfromtheSpecification(i.e.,thisdocument). 6.4GeneralItemsAstatementidentifyingthebenchmarksponsor(s)andotherparticipatingcompaniesmustbeprovided. Settingsmustbeprovidedforallcustomer-tunableparametersandoptionsthathavebeenchangedfromthedefaultsfoundinactualproducts,includingbutnotlimitedto:

• Configurationparametersandoptionsforserver,storage,networkandotherhardwarecomponentsincorporatedintothepricingstructure


• Configurationparametersandoptionsforoperatingsystemandfilesystemcomponentsincorporatedintothepricingstructure

• Configurationparametersandoptionsforanyothersoftwarecomponentsincorporatedintothepricingstructure

• Compileroptimizationoptions

Comment1:Intheeventthatsomeparametersandoptionsaresetmultipletimes,itmustbeeasilydiscerniblebyaninterestedreaderwhentheparameteroroptionwasmodifiedandwhatnewvalueitreceivedeachtime.Comment 2: This requirement can be satisfied by providing a full list of all parameters andoptions,aslongasallthosethathavebeenmodifiedfromtheirdefaultvalueshavebeenclearlyidentifiedandtheseparametersandoptionsareonlysetonce. Explicitresponsetoindividualdisclosurerequirementsspecifiedinthebodyofearliersectionsofthisdocumentmustbeprovided. Diagramsofbothmeasuredandpricedconfigurationsmustbeprovided,accompaniedbyadescriptionofthedifferences.Thisincludes,butisnotlimitedto:

• Totalnumberofnodesused• Totalnumberandtypeofprocessorsused/totalnumberofcoresused/totalnumberof

threadsused(includingsizesofL2andL3caches)• Sizeofallocatedmemory,andanyspecificmapping/partitioningofmemoryuniqueto

thetest• Numberandtypeofdiskunits(andcontrollers,ifapplicable)• Numberofchannelsorbusconnectionstodiskunits,includingtheirprotocoltype• NumberofLAN(forexample,Ethernet)connectionsandspeedforswitchesandother

hardwarecomponentsphysicallyusedinthetestorareincorporatedintothepricingstructure

• Typeandtherun-timeexecutionlocationofsoftwarecomponentsThefollowingsamplediagramillustratesameasuredbenchmarkconfigurationusingEthernet,anexternaldriver,andfourprocessorseachwithtwocoresandfourthreadspernodeinthe


SUT.NotethatthisdiagramdoesnotdepictorimplyanyoptimalconfigurationfortheTPCx-IoTbenchmarkmeasurement.DependingontheimplementationoftheSUT,thecomponentsforthestoragesystembeingused,theheadnode,theworkernodesetc.orthefunctionalequivalentsmustbespecifiedinthediagram.

Figure3:SampleConfigurationDiagram

• 4xMyServerModelB,4/32/64MyCPUModelZ(2.7GHz,20MBcache,130W),128GB,MyRAIDControllerwith1GBBBWC

• 4xMyStorageArrayModelAwith8X1TB10KSASHDD• 2xMySwitchModelX10GbE

Comment:Detaileddiagramsforsystemconfigurationsandarchitecturescanvarywidely,anditisimpossibletoprovideexactguidelinessuitableforallimplementations.Theintenthereistodescribethesystemcomponentsandconnectionsinsufficientdetailtoallowindependentreconstructionofthemeasurementenvironment.Thisexamplediagramshowshomogeneousnodes.Thisdoesnotprecludetestssponsorsfromusingheterogeneousnodesaslongasthesystemdiagramreflectsthecorrectsystemconfiguration.Thedistributionofdatasetacrossallmediamustbeexplicitlydescribedusingaformatsimilartothatshowninthefollowingexampleforthetestedsystem.


Table1:SampleLayoutDescription

6.5WorkloadRelatedItemsScriptortextusedtosetallhardwareandsoftwaretunableparametersmustbereported.TherunreportgeneratedbytheTPCx-IoTKitforPerformanceRunandRepeatabilityRunmustbereported. 6.6AuditRelatedItemsIfthebenchmarkisauditedbyanIndependentAuditor,theauditor'sagencyname,address,phonenumber,andattestationletterwithabriefauditsummaryreportindicatingcompliancemustbeincludedintheFullDisclosureReport.Astatementshouldbeincludedspecifyingwhomtocontactinordertoobtainfurtherinformationregardingtheauditprocess.6.7ExecutiveSummaryTheExecutiveSummaryismeanttobeahigh-leveloverviewoftheimplementation.Itshouldprovidethesalientcharacteristicsofabenchmarkexecution(metrics,configuration,pricing,etc.)withouttheexhaustivedetailfoundintheFullDisclosureReport.

Theexecutivesummaryhasthreecomponents:• ImplementationOverview• PricingSpreadsheet

ServerController DiskDrive DescriptionofContent

1 40A 0 Operatingsystem,root,swap,NoSQLMaster

1-12 MasterServerFilesystemMetadata

2 40A 0 Operatingsystem,root,swap

1-12 NoSQLworkerdatanodes






• NumericalQuantitiesEachcomponentoftheexecutivesummaryshouldappearonapagebyitself.Eachpageshoulduseastandardheaderandformat,including

• 1/2inchmargins,topandbottom• 3/4-inchleftmargin,1/2-inchrightmargin• 2pt.framearoundthebodyofthepage.Allinteriorlinesshouldbe1pt.

6.8ImplementationOverviewTheimplementationoverviewpagecontainsfivesetsofdata,eachlaidoutacrossthepageasasequenceofboxesusing1pt.rule,withatitleabovetherequiredquantity.Bothtitlesandquantitiesshouldusea9-12pt.Timesfontunlessotherwisenoted.Thefirstsectioncontainsinformationaboutthesponsorandsystemidentification.Table2:SponsorandSystemIdentification

Title FontSponsorNameorLogo 16-20pt.Bold(forName)

SystemIdentification16-20pt.Bold

VersionNumbersforTPCx-IoT,TPC-Pricing 16-20pt.Bold

ReportDate 16-20pt.Bold

Comment1:Itispermissibletouseorincludecompanylogoswhenidentifyingthesponsor.Comment2:Thereportdatemustbedisclosedwithaprecisionofoneday.Thepreciseformatislefttothetestsponsor.ThesecondsectioncontainstheTotalSystemCost;and,TPCx-IoTPerformanceMetricandPrice/Performancefortheperformancerun.Table3:TestResultsTitle Quantity Precision FontTotalSystemCost 3yr.Costof

Ownership(1 16-20pt.Bold

TPCx-IoTPerformanceMetric

IoTps 0.01 16-20pt.Bold

Price/Performance $/IoTps 0.01 16-20pt.Bold


DependingonthecurrencyusedforpublicationthissignhastobeexchangedwiththeISOcurrencysymbol.Thethirdsectioncontainsdetailedthesystemconfiguration.Table4:SystemConfigurationInformationTitle Quantity FontStorageSystemSoftware ProductNameandProduct

Version9-12pt.Times

OperatingSystem ProductName,SoftwareVersionforOS,FileSystemTypeandVersion

9-12pt.Times

OtherSoftware ProductNameandSoftwareVersionofothersoftwarecomponents(exampleJava)

9-12pt.Times

SystemAvailabilityDate TheAvailabilityDateofthesystem,definedinClause0oftheTPCPricingSpecification.

9-12pt.Times

Comment:TheSoftwareVersionmustuniquelyidentifytheorderablesoftwareproductreferencedinthePricedConfiguration(forexample,RALF/20004.2.1)Thefourthsectioncontainsthecomponents,including:

• Totalnumberofnodesused/totalnumberofprocessorsusedwiththeirtypesandspeedsinGHz

• Totalnumberofcoresused/totalnumberofthreadsused,SeeClause7.• Mainandcachememorysizes• NetworkandI/Oconnectivity• Diskquantityandgeometry• TotalRackUnits(RU)occupiedbytheSUT

Comment:RackUnits(RU)occupiedbytheSUTincludeservers,storage,connectivitydevicesandanyadditionalrackspacerequiredtobekeptemptybythedevices.Freespaceavailableontherack,spaceforverticalmountPDUetc.arenottobeincluded.


Ex:8Servers (2RUeach)and2Switches (1RUeach)mounted ina42RUrack.TheRackUnitsoccupiedbytheSUTis8x2+2x1=18RU 6.9PricingSpreadsheetThemajorcategoriesinthePriceSpreadsheet,asappropriate,are:

• Network(s)• Server(s)/Node(s)• Storage• Software

Discounts(mayoptionallybeincludedwithabovemajorcategorysubtotalcalculations). 6.10NumericalQuantitiesSummary TheNumericalQuantitiesSummarypagecontainstwosetsofdata,presentedintabularform,detailingtheexecutiontimingsforthereportedexecutionoftheperformancetest.Eachsetofdatashouldbeheadedbyitsgiventitleandclearlyseparatedfromtheothertables. Thefirstsectioncontainsmeasurementresultsfromthebenchmarkexecution.Table5:ResultsforMeasuredRunItemTitle PrecisionRunStartTime yyyy-mm-ddhh:mm:ssRunEndTime yyyy-mm-ddhh:mm:ssIoTps 0.01Secondsectioncontainsthemeasurementresultfortherepeatabilityrun.SeeTable5:forcontentsandprecision. 6.11TPCx-IoTRunReportTherunreportfromTPCx-IoTmustbeincludedintheExecutiveSummary.

6.12 AvailabilityoftheFullDisclosureReportTheFullDisclosureReportmustbereadilyavailabletothepublic.Thereportmustbemadeavailablewhenresultsaremadepublic.Inordertousethephrase“TPCExpressBenchmark


IoT”,theFullDisclosureReportmustbesubmittedelectronicallytotheTPCusingtheproceduredescribedintheTPCPolicieslocatedattheTPCDocumentationwebpage.TheFullDisclosureReportmustbeavailableinEnglishbutmaybetranslatedtoadditionallanguages. 6.13RevisionstotheFullDisclosureReportRevisionstotheFullDisclosureReportshallbehandledasfollows:

• Substitutionswillbeopentochallengefora60-dayperiod.NootherportionoftheFFullDisclosureReportandsupportingfilesarchivearechallengeable.

• Duringthenormalproductlifecycle,problemswillbeuncoveredthatrequirechanges,sometimesreferredtoasECOs,FCOs,patches,updates,etc.WhenthecumulativeresultofappliedchangescausestheIoTpsratingofthesystemtodecreasebymorethantwopercentfromtheinitiallyreportedIoTps,thenthetestsponsorisrequiredtore-validatethebenchmarkresults.Thecompleterevisionhistoryismaintainedfollowingthetimingintervalsectionshowingtherevisiondateanddescription.

• FullDisclosureReportandsupportingfilesarchiverevisionsmayberequiredforotherreasonsaccordingtoTPCPolicieslocatedattheTPCDocumentationwebpage.


Clause7:AuditRulesforauditingPricinginformationareincludedintheTPCPricingSpecificationlocatedattheTPCDocumentationwebpage. 7.1GeneralRulesThebenchmarkresultmustbecertifiedbyaTPCCertifiedAuditorortheTPCx-IoTPre-PublicationBoard.SeeSection10ofTPCPolicieslocatedattheTPCDocumentationwebpage.Thetermindependentisdefinedas“theoutcomeofthebenchmarkcarriesnofinancialbenefittotheauditingagencyotherthanfeesearneddirectlyrelatedtotheaudit.”Theauditingagencycannothavesuppliedanyperformanceconsultingundercontractforthebenchmark.TheIndependentAuditormustmeetthefollowing:

• TheauditorholdsanactiveTPCcertificationforaTPCenterprisebenchmarkoranexpressbenchmark.

• Theauditingagencycannotbefinanciallyrelatedtothesponsor.Forexample,theauditingagencyisfinanciallyrelatedifitisadependentdivisionofthesponsor,themajorityofitsstockisownedbythesponsor,etc.

• Theauditingagencycannotbefinanciallyrelatedtoanyoneofthesuppliersofthemeasured/pricedconfiguration.

• Theauditor'sattestationletteristobemadereadilyavailabletothepublicaspartoftheFullDisclosureReport.Adetailedreportfromtheauditorisnotrequired.

ThePre-PublicationBoardconsistsofthreemembersoftheTPCx-IoTcommittee.Eachmemberservesaperiodofsixmonths.ThemembershipwillberotatedthroughtheTPCx-IoTmembership.Thesubmissionisconfidentialtothepeerreviewcommitteeuntiltheresultispublished.Thepeerreviewcommitteemustcompletethereviewin10businessdays.Ifnoissuesareraisedin10days,theresultisconsideredvalid.

TPCx-IoTresultscanbeusedasthebasisfornewTPCx-IoTresultsifandonlyif:

• TheAuditororPre-PublicationBoardensurethatthehardwareandsoftwareproductsarethesameasthoseusedinthepriorresult;

• TheAuditororPre-PublicationBoardreviewtheFofthenewresultsandensuresthattheymatchwhatiscontainedintheoriginalsponsor'sFullDisclosureReport;

• TheAuditororPre-PublicationBoardcanattesttothevalidityofthepricingusedinthenewFullDisclosureReport.


Comment1:Theintentofthisclauseistoallowaresellerofequipmentfromagivensuppliertopublishunderthere-seller'snameaTPCx-IoTresultalreadypublishedbythesupplier. 7.2AuditCheckList7.2.1Clause2:WorkloadandExecutionRelatedItemsVerifythattheTPCx-IoTkitisusedVerifythatallphasesarecompletewithnoerrorinRun1andRun2Verifythatallscriptsandsourcecodetoimplementthebenchmarkisincluded.7.2.2Clause3:SystemUnderTestandDriverRelatedItemsVerifythatallcomponentsoftheSUTarecommerciallyavailableasperTPCPricingSpecificationVerifythatallcomponentsoftheSUTisincludedinthepricing7.2.3Clause4:ScaleFactorsandMetricsRelatedItemsVerifythatthesystemisscaledaspertheSpecificationVerifythatthemetricsarereportedaspertheprecisionrequirements7.2.4Clause5:PricingRelatedItemsVerifythatthebenchmarkisincompliancewiththeTPCPricingSpecification7.2.5Clause7:FullDisclosureRelatedItemsVerifythatFullDisclosureReportandexecutivesummaryreportareaccurateandcomplywiththereportingrequirements.Thisincludes:

• Systemavailability• Thediagramsofbothmeasuredandpricedconfiguration.• Systempricing• Thenumericalquantitysummary


Clause8:SampleExecutiveSummary My Company Logo My Server Model B TPCx-IoT Rev. 1.1.0

TPC-Pricing Rev. 2.0.1 Report Date: May1, 2017

Total System Cost Performance Metric Price / Performance $99,996.13 USD 390.99

IoTps $255.76 USD $ / IoTps

Database Management System

Operating System Other Software

Availability Date

My NoSQL Software 1.0

My OS V2.0 None May 1, 2017

System Configuration

Servers 4 x My Server Model B

Processors/Cores/Threads/Model 4/32/64 My CPU Model Z (2.7 GHz, 20MB cache, 130W)

Memory

128GB

Storage 2 x 600GB 10K SFF SAS (internal) 1 x My Storage Array Model A with 8 X 1TB 7.2K SAS LFF HDD

Network Rack Unit

2x My Switch Model X 10GbE 10RU

4 x My Storage Array

Model A 4 x My Server Model B

2 x My Switch Model X

10 GbE

SAS 3.0


My Company Logo My Server Model B TPCx-IoT Rev. 1.0.0 TPC-Pricing Rev. 2.0.1 Report Date: May 1, 2017

Description Part Number

Source

Unit Price

Qty Extended Price

3 Year Maint. Price

My Server Model B, 4 My CPU Model Z, 128GB, 2 x 600GB 10K SFF SAS

MY-S-001

1 12,100.77

4 $48,403 $100

My Storage Array Model A

MY-SE-002

1 1,988.00

4 $7,952 $200

My HDD Model xyz 1TB SATA 7.2K LFF

MY-HDD-011

1 800.47 40 $32,019

My OS MY-OS 1 485.24 4 $1,941 My NoSQL Software MY-

NoSQL 1 2,700.0

0 4 $10,800

My Switch Model X My-Switch

1 1,922.12

2 $3,844

Subtotal

$104,959 $300

Large Purchase Discount

5.0% 1 -$5,248 -$15

Pricing: 1=My Company Three-Year Cost of Ownership:

$99,996.1

Audited by My Auditor or Pre-Publication Board All discounts are based on US list prices and for similar quantities and configurations. The discounts are based on the overall specific components pricing from respective vendors in this single quotation. Discounts for similarly sized configurations will be similar to those quoted here, but may vary based on the components in the configuration.

IoTps: 1,100.1

$ / IoTps: $90.9

Prices used in TPC benchmarks reflect the actual prices a customer would pay for a one-time purchase of the stated components. Individually negotiated discounts are not permitted. Special prices based on assumptions about past or future purchases are not permitted. All discounts reflect standard pricing policies for the listed components. For complete details, see the pricing sections of the TPC benchmark Specifications. If you find that the stated prices are not available according to these terms, please inform at [email protected]. Thank you.

tpc express benchmarktm iot (tpcx-iot) standard ... · includes andy bond, bhaskar gouda, karthik...

Documents