SaddleHillA SCSI I/O GeneratorByOwen Parry

Project MotivationTo create an application that exercises new controller firmware and hardware.Provide the ability to rapidly add features.Provide more tolerance for hardware/firmware failures.To Develop a mechanism that allows multiple hosts to randomly access and efficiently share the SATA affiliations.Current methods seek to avoid the Serial ATA limitation; using single initiator in a SAS domain; limiting communication with the disk drive to only 1 initiator at a time.Need simple and decentralized strategy for an embedded environment.Achieve long-term max-min fairness.Avoid violating I/O time limits.

BackgroundSCSI targeted at the enterprise storage market. Used primarily to attach hard disk drives.High performanceRPM: 10K, 15KSeek Time: 3.2 7.4 msGreater reliability.MTBF: 1.2 M HrCapacity, 18 300GbExpensive: $160 - $1400Multiple host support.ATA targeted at the desktop market.Medium performanceRPM: 5400, 7200Seek Time: 8.9 9.5 msMediocre reliabilityMTBF: 500 K hrCapacity, 40GB 1TbCheap: $75 - $300Single host.

BackgroundSerial Attached SCSI is the new Transport protocol replacing parallel SCSI.SAS Advantages.Faster Data Rates.SAS-1:300 MB/sSAS-2:600MB/sLarger Drive counts.Typical Domain size 12816K addresses using fan-out expandersIncreased data integrity.Configuration flexibility.Supports Serial ATA Drives.

BackgroundProblem with SATA in SAS topologyArchitecture only allows a single host.SAS uses mutual exclusion called an Affiliation. The first initiator to open a connection may own the affilation indefinitely.Vendors want to simultaneously issue commands to SATA disks from multiple initiators.

Related WorkUnable to locate other works in the storage area.Closely Related ResearchWireless LANsBandwidth sharing schemes:Maxmin Fair Scheduling in Wireless Networks, Leandros Tassiulas and Saswati Sarkar.Channel time sharing schemes:Proportional Fairness in Wireless LANs and Ad Hoc Networks, Li Bin Jian, and Soung Chang LiewTime-based fairness improves Performance in Multi-rate WLANs, Godfrey Tan and John Guttag.

SaddleHill Design / Implementation

text

text

SCSI IO System

SCSI Target

Service Delivery Subsystem

SCSI Initiator

Linux Operating System

SAS Initiator Port

SaddleHill Driver

SaddleHill

SaddleHill Design / ImplementationBuilt using Trolltechs Qt 4.2.3Compiled for x86_64 bit systems.Comprised of four logical blocks.

SaddleHill Design / ImplementationMainWindowLists PCI SAS Initiators devices.Lists SAS Target devices.Displays Live Test Statistics.Displays Application messages.Accepts user input.

SaddleHill Design / ImplementationManagement UnitManages SaddleHills physical I/O Data buffers, and Initiator operational buffers.Address conversion: Virtual to Physical; Physical to Virtual.Maintains a list of SAS Initiators and Targets.Maintain the application message log.Maintain the model objects (system device, message, statistics) which are used by the GUI to gather and display information to user.Distributes device configurations.Starts/Stops I/O tests. Calculates I/O and Throughput rates.

SaddleHill Design / ImplementationIO EngineInitializes SAS targets.Maintains disk SAS Addresses, and Target ID.Generates, Issues, and Completes SCSI Commands. e.g. Read10, Write10, Write And Verify10, Inquiry, Read Capacity etc.Comprises three threads to perform each of the above tasks.Maintains statistics:Number of I/Os issuedNumber of I/Os completedError countAmount of Data Transferred.I/O Response times.

SaddleHill Design / ImplementationHardware Abstraction LayerSaddleHillDriverRegisters with linux kernel as a character device.Registers with PCI core.Allocates blocks of physical memory. Currently 16 MB.Reserves the physical memory to prevent swapping.Provides the facilities to map PCI SAS I/O control registers to user space.Provides the facilities to map the physical memory to user space.Provides PCI Device configuration information to user space application.HAL (User Level)Implements the MPI specificationInitializes the SAS AdapterConverts Requests from IO Engine to the MPI specific format.Sends requests to and receive replies to/from the Initiator via the PCI control registers.Processes MPI Replies and completes request to IO Engine.Manages STP Affiliations.Maintains test statisticsNumber of I/Os issues.Number of I/Os completed.Error Count.Amount of Data Transferred.I/O Response Times.Affiliation ownership times.Affiliation synchronization count.

Affiliation SynchronizationUses idea put forward in Proportional Firness in Wireless LANs and Ad Hoc Networks.Fix the maximum transmission time.Contend fairly among the initiators for the mutex.ImplementationAffiliation AcquisitionAcquisition started be reception of new I/OCalculate back-off.Use uniform distribution random number generator to choose back-off time within contention window size.Generate SCSI Inquiry commandSleep for length of back-offIssue InquiryFailed synchronization attempt doubles contention window size.Start timer on successful acquisitionAffiliation ReleaseResource released if no I/Os are waiting to be sent.Resource released after ownership timer expires.There are no preemptions.I/Os are placed into a waiting state during the release and acquisition process.I/Os outstanding at the time of release are allowed to completed.The truncated binary exponential back-off strategy is used to calculate the back-off times.

Finding the Back-Off StrategyConsidered strategies for back-off included:No Back-offFixed WindowBEBTBEBLogarithmicTest Strategy:Read10, Write10 commandsSingle Block TransfersSame LBADrive Caching EnabledNCQ enabledDrive Q-Depth = 83Gb SATA diskMultiple Initiators

Finding the Back-Off StrategySTP Ownership Times

Chart14

17736101770986

11873801220190

15747731760360

16632581719267

17453181746005

17459071745360

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

Sheet1

Two Adapters - Run Time = 60 seconds

BackOff AlgorithmAvg. RespLargest RespNum Sync. RequestsRsc Time

No BackOff57753248929378

Fixed Window14340576720589

BEB311496918232691

TBEB149557118326690

LOG58821103229042

TLOG59792102029133

No BackOff59813253129262

Fixed Window14428457920058

BEB2491349716820045

TBEB129581118328216

LOG58817103428542

TLOG631097104028225

Four Adapters - Run Time 60 seconds

No BackOff1883150376016036

Fixed Window4516572859080

BEB5631189415911797

TBEB387537923211549

LOG2913654173912344

TLOG1892786160614259

No BackOff2802794409813062

Fixed Window34533979911092

BEB3781127017213585

TBEB383557324812013

LOG1622521156216152

TLOG1853690155315313

No BackOff2772646402912702

Fixed Window6316025907108

BEB490106581849874

TBEB353762924014646

LOG2644551165013745

TLOG2474035164713443

No BackOff1643110369016562

Fixed Window33148108911064

BEB2921230718417475

TBEB288475921715039

LOG2062625155215343

TLOG2253810164114275

Two Adapters - Run Time 60 minutes

No BackOff699401495851773610

Fixed Window163696246601187380

BEB1543826776951574773

TBEB1287102123431663258

LOG571657858891745318

TLOG571212856431745907

No BackOff699101503961770986

Fixed Window155615145961220190

BEB1203963576841760360

TBEB1196537121401719267

LOG571229855851746005

TLOG571140856311745360

Four Adapters - Run Time 60 minutes

No BackOff2389134308174867054

Fixed Window38193625810618523

BEB3543363410843822688

TBEB3961086214271790866

LOG2176247128800850684

TLOG2496267100385857229

No BackOff2348822306691876004

Fixed Window41581295874586537

BEB3913951810626775765

TBEB371904414418819852

LOG2045998127432879152

TLOG235736199478883262

No BackOff23411161307522869000

Fixed Window415104715850588228

BEB3912705510542752990

TBEB372884614381827747

LOG2035665127281879839

TLOG2577460100843848736

No BackOff2268612304844891607

Fixed Window41693185802584862

BEB3344110810788854700

TBEB398890014336780650

LOG2106959128301865056

TLOG233568499443886117

Sheet1

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

Sheet2

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Count

Synchronization Requests

Sheet3

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 2

Disk 4

Algorithm

Count

Synchronization Requests

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Time

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Num. Attempts

Synchronization Requests

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response TIme

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Number of Attempts

Synchronization Requests

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Times

Chart12

2926229378

2005820589

2004532691

2821626690

2854229042

2822529133

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

Sheet1

Two Adapters - Run Time = 60 seconds

BackOff AlgorithmAvg. RespLargest RespNum Sync. RequestsRsc Time

No BackOff57753248929378

Fixed Window14340576720589

BEB311496918232691

TBEB149557118326690

LOG58821103229042

TLOG59792102029133

No BackOff59813253129262

Fixed Window14428457920058

BEB2491349716820045

TBEB129581118328216

LOG58817103428542

TLOG631097104028225

Four Adapters - Run Time 60 seconds

No BackOff1883150376016036

Fixed Window4516572859080

BEB5631189415911797

TBEB387537923211549

LOG2913654173912344

TLOG1892786160614259

No BackOff2802794409813062

Fixed Window34533979911092

BEB3781127017213585

TBEB383557324812013

LOG1622521156216152

TLOG1853690155315313

No BackOff2772646402912702

Fixed Window6316025907108

BEB490106581849874

TBEB353762924014646

LOG2644551165013745

TLOG2474035164713443

No BackOff1643110369016562

Fixed Window33148108911064

BEB2921230718417475

TBEB288475921715039

LOG2062625155215343

TLOG2253810164114275

Two Adapters - Run Time 60 minutes

No BackOff699401495851773610

Fixed Window163696246601187380

BEB1543826776951574773

TBEB1287102123431663258

LOG571657858891745318

TLOG571212856431745907

No BackOff699101503961770986

Fixed Window155615145961220190

BEB1203963576841760360

TBEB1196537121401719267

LOG571229855851746005

TLOG571140856311745360

Four Adapters - Run Time 60 minutes

No BackOff2389134308174867054

Fixed Window38193625810618523

BEB3543363410843822688

TBEB3961086214271790866

LOG2176247128800850684

TLOG2496267100385857229

No BackOff2348822306691876004

Fixed Window41581295874586537

BEB3913951810626775765

TBEB371904414418819852

LOG2045998127432879152

TLOG235736199478883262

No BackOff23411161307522869000

Fixed Window415104715850588228

BEB3912705510542752990

TBEB372884614381827747

LOG2035665127281879839

TLOG2577460100843848736

No BackOff2268612304844891607

Fixed Window41693185802584862

BEB3344110810788854700

TBEB398890014336780650

LOG2106959128301865056

TLOG233568499443886117

Sheet1

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

Sheet2

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Count

Synchronization Requests

Sheet3

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 2

Disk 4

Algorithm

Count

Synchronization Requests

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Time

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Num. Attempts

Synchronization Requests

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response TIme

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Number of Attempts

Synchronization Requests

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Times

Finding the Back-Off StrategySynchronization Requests

Chart8

25312489

7967

168182

183183

10341032

10401020

Disk 1

Disk 2

Algorithm

Count

Synchronization Requests

Sheet1

Two Adapters - Run Time = 60 seconds

BackOff AlgorithmAvg. RespLargest RespNum Sync. RequestsRsc Time

No BackOff57753248929378

Fixed Window14340576720589

BEB311496918232691

TBEB149557118326690

LOG58821103229042

TLOG59792102029133

No BackOff59813253129262

Fixed Window14428457920058

BEB2491349716820045

TBEB129581118328216

LOG58817103428542

TLOG631097104028225

Four Adapters - Run Time 60 seconds

No BackOff1883150376016036

Fixed Window4516572859080

BEB5631189415911797

TBEB387537923211549

LOG2913654173912344

TLOG1892786160614259

No BackOff2802794409813062

Fixed Window34533979911092

BEB3781127017213585

TBEB383557324812013

LOG1622521156216152

TLOG1853690155315313

No BackOff2772646402912702

Fixed Window6316025907108

BEB490106581849874

TBEB353762924014646

LOG2644551165013745

TLOG2474035164713443

No BackOff1643110369016562

Fixed Window33148108911064

BEB2921230718417475

TBEB288475921715039

LOG2062625155215343

TLOG2253810164114275

Two Adapters - Run Time 60 minutes

No BackOff699401495851773610

Fixed Window163696246601187380

BEB1543826776951574773

TBEB1287102123431663258

LOG571657858891745318

TLOG571212856431745907

No BackOff699101503961770986

Fixed Window155615145961220190

BEB1203963576841760360

TBEB1196537121401719267

LOG571229855851746005

TLOG571140856311745360

Four Adapters - Run Time 60 minutes

No BackOff2389134308174867054

Fixed Window38193625810618523

BEB3543363410843822688

TBEB3961086214271790866

LOG2176247128800850684

TLOG2496267100385857229

No BackOff2348822306691876004

Fixed Window41581295874586537

BEB3913951810626775765

TBEB371904414418819852

LOG2045998127432879152

TLOG235736199478883262

No BackOff23411161307522869000

Fixed Window415104715850588228

BEB3912705510542752990

TBEB372884614381827747

LOG2035665127281879839

TLOG2577460100843848736

No BackOff2268612304844891607

Fixed Window41693185802584862

BEB3344110810788854700

TBEB398890014336780650

LOG2106959128301865056

TLOG233568499443886117

Sheet1

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

Sheet2

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Count

Synchronization Requests

Sheet3

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 2

Disk 4

Algorithm

Count

Synchronization Requests

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Time

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Num. Attempts

Synchronization Requests

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response TIme

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Number of Attempts

Synchronization Requests

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Times

Chart10

149585150396

46604596

76957684

1234312140

8588985585

8564385631

Disk 1

Disk 2

Algorithm

Num. Attempts

Synchronization Requests

Sheet1

Two Adapters - Run Time = 60 seconds

BackOff AlgorithmAvg. RespLargest RespNum Sync. RequestsRsc Time

No BackOff57753248929378

Fixed Window14340576720589

BEB311496918232691

TBEB149557118326690

LOG58821103229042

TLOG59792102029133

No BackOff59813253129262

Fixed Window14428457920058

BEB2491349716820045

TBEB129581118328216

LOG58817103428542

TLOG631097104028225

Four Adapters - Run Time 60 seconds

No BackOff1883150376016036

Fixed Window4516572859080

BEB5631189415911797

TBEB387537923211549

LOG2913654173912344

TLOG1892786160614259

No BackOff2802794409813062

Fixed Window34533979911092

BEB3781127017213585

TBEB383557324812013

LOG1622521156216152

TLOG1853690155315313

No BackOff2772646402912702

Fixed Window6316025907108

BEB490106581849874

TBEB353762924014646

LOG2644551165013745

TLOG2474035164713443

No BackOff1643110369016562

Fixed Window33148108911064

BEB2921230718417475

TBEB288475921715039

LOG2062625155215343

TLOG2253810164114275

Two Adapters - Run Time 60 minutes

No BackOff699401495851773610

Fixed Window163696246601187380

BEB1543826776951574773

TBEB1287102123431663258

LOG571657858891745318

TLOG571212856431745907

No BackOff699101503961770986

Fixed Window155615145961220190

BEB1203963576841760360

TBEB1196537121401719267

LOG571229855851746005

TLOG571140856311745360

Four Adapters - Run Time 60 minutes

No BackOff2389134308174867054

Fixed Window38193625810618523

BEB3543363410843822688

TBEB3961086214271790866

LOG2176247128800850684

TLOG2496267100385857229

No BackOff2348822306691876004

Fixed Window41581295874586537

BEB3913951810626775765

TBEB371904414418819852

LOG2045998127432879152

TLOG235736199478883262

No BackOff23411161307522869000

Fixed Window415104715850588228

BEB3912705510542752990

TBEB372884614381827747

LOG2035665127281879839

TLOG2577460100843848736

No BackOff2268612304844891607

Fixed Window41693185802584862

BEB3344110810788854700

TBEB398890014336780650

LOG2106959128301865056

TLOG233568499443886117

Sheet1

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

Sheet2

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Count

Synchronization Requests

Sheet3

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 2

Disk 4

Algorithm

Count

Synchronization Requests

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Time

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Num. Attempts

Synchronization Requests

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response TIme

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Number of Attempts

Synchronization Requests

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Times

Finding the Back-Off StrategyAverage I/O Response

Chart3

5957

144143

24931

129149

5858

6359

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

Sheet1

Two Adapters - Run Time = 60 seconds

BackOff AlgorithmAvg. RespLargest RespNum Sync. RequestsRsc Time

No BackOff57753248929378

Fixed Window14340576720589

BEB311496918232691

TBEB149557118326690

LOG58821103229042

TLOG59792102029133

No BackOff59813253129262

Fixed Window14428457920058

BEB2491349716820045

TBEB129581118328216

LOG58817103428542

TLOG631097104028225

Four Adapters - Run Time 60 seconds

No BackOff1883150376016036

Fixed Window4516572859080

BEB5631189415911797

TBEB387537923211549

LOG2913654173912344

TLOG1892786160614259

No BackOff2802794409813062

Fixed Window34533979911092

BEB3781127017213585

TBEB383557324812013

LOG1622521156216152

TLOG1853690155315313

No BackOff2772646402912702

Fixed Window6316025907108

BEB490106581849874

TBEB353762924014646

LOG2644551165013745

TLOG2474035164713443

No BackOff1643110369016562

Fixed Window33148108911064

BEB2921230718417475

TBEB288475921715039

LOG2062625155215343

TLOG2253810164114275

Two Adapters - Run Time 60 minutes

No BackOff699401495851773610

Fixed Window163696246601187380

BEB1543826776951574773

TBEB1287102123431663258

LOG571657858891745318

TLOG571212856431745907

No BackOff699101503961770986

Fixed Window155615145961220190

BEB1203963576841760360

TBEB1196537121401719267

LOG571229855851746005

TLOG571140856311745360

Four Adapters - Run Time 60 minutes

No BackOff2389134308174867054

Fixed Window38193625810618523

BEB3543363410843822688

TBEB3961086214271790866

LOG2176247128800850684

TLOG2496267100385857229

No BackOff2348822306691876004

Fixed Window41581295874586537

BEB3913951810626775765

TBEB371904414418819852

LOG2045998127432879152

TLOG235736199478883262

No BackOff23411161307522869000

Fixed Window415104715850588228

BEB3912705510542752990

TBEB372884614381827747

LOG2035665127281879839

TLOG2577460100843848736

No BackOff2268612304844891607

Fixed Window41693185802584862

BEB3344110810788854700

TBEB398890014336780650

LOG2106959128301865056

TLOG233568499443886117

Sheet1

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

Sheet2

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Count

Synchronization Requests

Sheet3

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 2

Disk 4

Algorithm

Count

Synchronization Requests

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Time

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Num. Attempts

Synchronization Requests

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response TIme

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Number of Attempts

Synchronization Requests

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Times

MBD0019B0B4.xls

Chart1

188280277164

451345631331

563378490292

387383353288

291162264206

189185247225

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

Sheet1

Two Adapters - Run Time = 60 seconds

BackOff AlgorithmAvg. RespLargest RespNum Sync. RequestsRsc Time

No BackOff57753248929378

Fixed Window14340576720589

BEB311496918232691

TBEB149557118326690

LOG58821103229042

TLOG59792102029133

No BackOff59813253129262

Fixed Window14428457920058

BEB2491349716820045

TBEB129581118328216

LOG58817103428542

TLOG631097104028225

Four Adapters - Run Time 60 seconds

No BackOff1883150376016036

Fixed Window4516572859080

BEB5631189415911797

TBEB387537923211549

LOG2913654173912344

TLOG1892786160614259

No BackOff2802794409813062

Fixed Window34533979911092

BEB3781127017213585

TBEB383557324812013

LOG1622521156216152

TLOG1853690155315313

No BackOff2772646402912702

Fixed Window6316025907108

BEB490106581849874

TBEB353762924014646

LOG2644551165013745

TLOG2474035164713443

No BackOff1643110369016562

Fixed Window33148108911064

BEB2921230718417475

TBEB288475921715039

LOG2062625155215343

TLOG2253810164114275

Two Adapters - Run Time 60 minutes

No BackOff699401495851773610

Fixed Window163696246601187380

BEB1543826776951574773

TBEB1287102123431663258

LOG571657858891745318

TLOG571212856431745907

No BackOff699101503961770986

Fixed Window155615145961220190

BEB1203963576841760360

TBEB1196537121401719267

LOG571229855851746005

TLOG571140856311745360

Four Adapters - Run Time 60 minutes

No BackOff2389134308174867054

Fixed Window38193625810618523

BEB3543363410843822688

TBEB3961086214271790866

LOG2176247128800850684

TLOG2496267100385857229

No BackOff2348822306691876004

Fixed Window41581295874586537

BEB3913951810626775765

TBEB371904414418819852

LOG2045998127432879152

TLOG235736199478883262

No BackOff23411161307522869000

Fixed Window415104715850588228

BEB3912705510542752990

TBEB372884614381827747

LOG2035665127281879839

TLOG2577460100843848736

No BackOff2268612304844891607

Fixed Window41693185802584862

BEB3344110810788854700

TBEB398890014336780650

LOG2106959128301865056

TLOG233568499443886117

Sheet1

No BackOffNo BackOff

Fixed WindowFixed Window

BEBBEB

TBEBTBEB

LOGLOG

TLOGTLOG

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

Sheet2

No backoffNo backoff

Fixed WindowFixed Window

BEBBEB

TBEBTBEB

LOGLOG

TLOGTLOG

Disk 1

Disk 2

Algorithm

Count

Synchronization Requests

Sheet3

No BackOffNo BackOff

Fixed WindowFixed Window

BEBBEB

TBEBTBEB

LOGLOG

TLOGTLOG

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

No BackOffNo BackOffNo BackOffNo BackOff

Fixed WindowFixed WindowFixed WindowFixed Window

BEBBEBBEBBEB

TBEBTBEBTBEBTBEB

LOGLOGLOGLOG

TLOGTLOGTLOGTLOG

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

No backOffNo backOffNo backOffNo backOff

Fixed WindowFixed WindowFixed WindowFixed Window

BEBBEBBEBBEB

TBEBTBEBTBEBTBEB

LOGLOGLOGLOG

TLOGTLOGTLOGTLOG

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response Time

No BackOffNo BackOffNo BackOffNo BackOff

Fixed WindowFixed WindowFixed WindowFixed Window

BEBBEBBEBBEB

TBEBTBEBTBEBTBEB

LOGLOGLOGLOG

TLOGTLOGTLOGTLOG

Disk 1

Disk 2

Disk 2

Disk 4

Algorithm

Count

Synchronization Requests

No BackOffNo BackOffNo BackOffNo BackOff

Fixed WindowFixed WindowFixed WindowFixed Window

BEBBEBBEBBEB

TBEBTBEBTBEBTBEB

LOGLOGLOGLOG

TLOGTLOGTLOGTLOG

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Time

No BackoffNo Backoff

Fixed WindowFixed Window

BEBBEB

TBEBTBEB

LOGLOG

TLOGTLOG

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

No BackOffNo BackOff

Fixed WindowFixed Window

BEBBEB

TBEBTBEB

LOGLOG

TLOGTLOG

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

No BackoffNo Backoff

Fixed WindowFixed Window

BEBBEB

TBEBTBEB

LOGLOG

TLOGTLOG

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

No BackoffNo Backoff

Fixed WindowFixed Window

BEBBEB

TBEBTBEB

LOGLOG

TLOGTLOG

Disk 1

Disk 2

Algorithm

Num. Attempts

Synchronization Requests

No BackoffNo Backoff

Fixed WindowFixed Window

BEBBEB

TBEBTBEB

LOGLOG

TLOGTLOG

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

No BackoffNo BackoffNo BackoffNo Backoff

Fixed WindowFixed WindowFixed WindowFixed Window

BEBBEBBEBBEB

TBEBTBEBTBEBTBEB

LOGLOGLOGLOG

TLOGTLOGTLOGTLOG

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

No BackoffNo BackoffNo BackoffNo Backoff

Fixed WindowFixed WindowFixed WindowFixed Window

BEBBEBBEBBEB

TBEBTBEBTBEBTBEB

LOGLOGLOGLOG

TLOGTLOGTLOGTLOG

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response TIme

No BackoffNo BackoffNo BackoffNo Backoff

Fixed WindowFixed WindowFixed WindowFixed Window

BEBBEBBEBBEB

TBEBTBEBTBEBTBEB

LOGLOGLOGLOG

TLOGTLOGTLOGTLOG

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Number of Attempts

Synchronization Requests

No BackoffNo BackoffNo BackoffNo Backoff

Fixed WindowFixed WindowFixed WindowFixed Window

BEBBEBBEBBEB

TBEBTBEBTBEBTBEB

LOGLOGLOGLOG

TLOGTLOGTLOGTLOG

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Times

Chart2

6969

163155

154120

128119

5757

5757

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

Sheet1

Two Adapters - Run Time = 60 seconds

BackOff AlgorithmAvg. RespLargest RespNum Sync. RequestsRsc Time

No BackOff57753248929378

Fixed Window14340576720589

BEB311496918232691

TBEB149557118326690

LOG58821103229042

TLOG59792102029133

No BackOff59813253129262

Fixed Window14428457920058

BEB2491349716820045

TBEB129581118328216

LOG58817103428542

TLOG631097104028225

Four Adapters - Run Time 60 seconds

No BackOff1883150376016036

Fixed Window4516572859080

BEB5631189415911797

TBEB387537923211549

LOG2913654173912344

TLOG1892786160614259

No BackOff2802794409813062

Fixed Window34533979911092

BEB3781127017213585

TBEB383557324812013

LOG1622521156216152

TLOG1853690155315313

No BackOff2772646402912702

Fixed Window6316025907108

BEB490106581849874

TBEB353762924014646

LOG2644551165013745

TLOG2474035164713443

No BackOff1643110369016562

Fixed Window33148108911064

BEB2921230718417475

TBEB288475921715039

LOG2062625155215343

TLOG2253810164114275

Two Adapters - Run Time 60 minutes

No BackOff699401495851773610

Fixed Window163696246601187380

BEB1543826776951574773

TBEB1287102123431663258

LOG571657858891745318

TLOG571212856431745907

No BackOff699101503961770986

Fixed Window155615145961220190

BEB1203963576841760360

TBEB1196537121401719267

LOG571229855851746005

TLOG571140856311745360

Four Adapters - Run Time 60 minutes

No BackOff2389134308174867054

Fixed Window38193625810618523

BEB3543363410843822688

TBEB3961086214271790866

LOG2176247128800850684

TLOG2496267100385857229

No BackOff2348822306691876004

Fixed Window41581295874586537

BEB3913951810626775765

TBEB371904414418819852

LOG2045998127432879152

TLOG235736199478883262

No BackOff23411161307522869000

Fixed Window415104715850588228

BEB3912705510542752990

TBEB372884614381827747

LOG2035665127281879839

TLOG2577460100843848736

No BackOff2268612304844891607

Fixed Window41693185802584862

BEB3344110810788854700

TBEB398890014336780650

LOG2106959128301865056

TLOG233568499443886117

Sheet1

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

Sheet2

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Count

Synchronization Requests

Sheet3

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 2

Disk 4

Algorithm

Count

Synchronization Requests

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Time

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Num. Attempts

Synchronization Requests

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response TIme

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Number of Attempts

Synchronization Requests

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Times

MBD0019B0B4.xls

Chart1

188280277164

451345631331

563378490292

387383353288

291162264206

189185247225

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

Sheet1

Two Adapters - Run Time = 60 seconds

BackOff AlgorithmAvg. RespLargest RespNum Sync. RequestsRsc Time

No BackOff57753248929378

Fixed Window14340576720589

BEB311496918232691

TBEB149557118326690

LOG58821103229042

TLOG59792102029133

No BackOff59813253129262

Fixed Window14428457920058

BEB2491349716820045

TBEB129581118328216

LOG58817103428542

TLOG631097104028225

Four Adapters - Run Time 60 seconds

No BackOff1883150376016036

Fixed Window4516572859080

BEB5631189415911797

TBEB387537923211549

LOG2913654173912344

TLOG1892786160614259

No BackOff2802794409813062

Fixed Window34533979911092

BEB3781127017213585

TBEB383557324812013

LOG1622521156216152

TLOG1853690155315313

No BackOff2772646402912702

Fixed Window6316025907108

BEB490106581849874

TBEB353762924014646

LOG2644551165013745

TLOG2474035164713443

No BackOff1643110369016562

Fixed Window33148108911064

BEB2921230718417475

TBEB288475921715039

LOG2062625155215343

TLOG2253810164114275

Two Adapters - Run Time 60 minutes

No BackOff699401495851773610

Fixed Window163696246601187380

BEB1543826776951574773

TBEB1287102123431663258

LOG571657858891745318

TLOG571212856431745907

No BackOff699101503961770986

Fixed Window155615145961220190

BEB1203963576841760360

TBEB1196537121401719267

LOG571229855851746005

TLOG571140856311745360

Four Adapters - Run Time 60 minutes

No BackOff2389134308174867054

Fixed Window38193625810618523

BEB3543363410843822688

TBEB3961086214271790866

LOG2176247128800850684

TLOG2496267100385857229

No BackOff2348822306691876004

Fixed Window41581295874586537

BEB3913951810626775765

TBEB371904414418819852

LOG2045998127432879152

TLOG235736199478883262

No BackOff23411161307522869000

Fixed Window415104715850588228

BEB3912705510542752990

TBEB372884614381827747

LOG2035665127281879839

TLOG2577460100843848736

No BackOff2268612304844891607

Fixed Window41693185802584862

BEB3344110810788854700

TBEB398890014336780650

LOG2106959128301865056

TLOG233568499443886117

Sheet1

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

Sheet2

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Count

Synchronization Requests

Sheet3

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 2

Disk 4

Algorithm

Count

Synchronization Requests

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Time

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Average IO Response Times

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

Largest IO Response Time

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Num. Attempts

Synchronization Requests

00

00

00

00

00

00

Disk 1

Disk 2

Algorithm

Time (ms)

STP Resource Ownership Time

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Average IO Response Times

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

Largest IO Response TIme

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Number of Attempts

Synchronization Requests

0000

0000

0000

0000

0000

0000

Disk 1

Disk 2

Disk 3

Disk 4

Algorithm

Time (ms)

STP Resource Ownership Times

Finding the Back-Off StrategyNo Back-OffToo many synchronization attemptsDepending on topology configuration will favor some initiatorsFixed WindowThere is no way to chose the appropriate window size.BEBViolates the I/O time limits in long test runs.LogarithmicAchieves near perfect max-min fairness in resource ownership in both short and long terms.Large number of synchronization requests. Unacceptable in large topologies.The Truncated Binary Exponential Strategy was chosen for the implementation of the synchronization algorithmClosely achieves long-term max-min fairnessLow number of synchronization attempts.

PerformanceTransaction processing profile was used. Small Block Transfer (1-16 Blocks)Concerned with I/O Rates rather than throughput.Single InitiatorSame IO size ~2250 IOPS.Random IO sizes ~902 IOPS.Dual InitiatorsSame IO sizes ~ 2075 IOPS. 8% Performance decrease.Random IO sizes ~ 786 IOPS. 12% Performance decreaseQuad InitiatorsSame IO sizes ~ 1975 IOPS. 14% Performance decreaseRandom IO sizes ~745 IOPS. 17% Performance decrease

Performance

Future DirectionsDue to the challenges of SATA in enterprise storage environments. Vendors are employing varying strategies to deal with the SATA problem. These include:Completely removing SATA from topologies.Building special hardware that increase the affiliation resources. The STP Resource sharing algorithm will be moved to the SAS Initiator port.Requires a change in the mechanism that acquires an releases affiliations.Utilize the SAS CLOSE(CLEAR AFFILIATION) primitive when tearing down connections.Simply convert and issue host IO.

SaddleHillShort TermSupport SAS-2 InitiatorSupport additional SBC and SPC commandsSupport SSC and MMC SCSI command setsFW Upgrade SupportInitiator Configuration ModificationLong TermBuild into a automated firmware unit test system.

ConclusionAll project goals achievedUser-Level SCSI I/O generatorSynchronization algorithm that meets the simplicity, fairness and decentralization objectives.

QUESTIONS?