enabling local and remote high performance backups shyamsundar r software engineer, sms novell, inc....
TRANSCRIPT
Enabling Local and Remote High Performance Backups
Shyamsundar RSoftware Engineer, SMSNovell, Inc.
Vijai Babu MadhavanSoftware Engineer, SMSNovell, Inc
© March 9, 2004 Novell Inc.2
one Net: Information without boundaries…where the right people are connected with the right information at the right time to make the right decisions.
The one Net vision
Novell exteNd™
Novell Nsure™
Novell Nterprise™
Novell NgageSM
:
:
:
:
© March 9, 2004 Novell Inc.3
The one Net vision
Novell Nterprise is an innovative family of products which gives you the power to enable and manage the constant interaction of people with your business systems — regardless of who they are or where they are.
Novell Nterprise™
Novell exteNd™
Novell Nsure™
Novell Nterprise™
Novell NgageSM
:
:
:
:
© March 9, 2004 Novell Inc.4
Agenda
Technology overview
SMS technology advancements
Troubleshooting backup on NetWare
Making the most out of your storage
Related technology
© March 9, 2004 Novell Inc.5
Backups
Backup is a second class citizen…
© March 9, 2004 Novell Inc.6
Backup data flow
TSAFS
*
Backup Engine
File Systems
Local backup data flow
© March 9, 2004 Novell Inc.7
Backup data flow
TSAFS
Backup Engine
File Systems*
Local backup data flow
© March 9, 2004 Novell Inc.8
Backup data flow
TSAFS
Backup Engine
File Systems
*
Local backup data flow
© March 9, 2004 Novell Inc.9
Backup data flow
TSAFS
Backup Engine
File Systems
*
Local backup data flow
© March 9, 2004 Novell Inc.10
Backup data flow
TSAFS
Backup Engine
File Systems
*
Local backup data flow
© March 9, 2004 Novell Inc.11
Backup data flow
SMDRBackup Engine SMDR
Network
*
TSAFS
Backup Engine
File Systems
Remote backup data flow
TSAFS
File Systems
© March 9, 2004 Novell Inc.12
Backup data flow
SMDRBackup Engine SMDR
Network
*Remote backup data flow
TSAFS
File Systems
TSAFS
Backup Engine
File Systems
© March 9, 2004 Novell Inc.13
Backup data flow
SMDRBackup Engine SMDR
Network
*Remote backup data flow
TSAFS
File Systems
TSAFS
Backup Engine
File Systems
© March 9, 2004 Novell Inc.14
Backup data flow
SMDRBackup Engine SMDR
Network
*Remote backup data flow
TSAFS
File Systems
TSAFS
Backup Engine
File Systems
© March 9, 2004 Novell Inc.15
Backup data flow
SMDRBackup Engine SMDR
Network
*Remote backup data flow
TSAFS
File Systems
Backup Engine
TSAFS
File Systems
16
Local backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
Storage (SCSI) Bus
Disk
I/O (PCI) bus
HBA
Memory
*
17
Local backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
Storage (SCSI) Bus
I/O (PCI) bus
HBA
Memory
*
Disk
18
Local backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
Storage (SCSI) Bus
I/O (PCI) bus
HBA
Memory
*
Disk
19
Local backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
Storage (SCSI) Bus
I/O (PCI) bus
HBA
Memory
*
Disk
20
Local backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
Storage (SCSI) Bus
I/O (PCI) bus
HBA
Memory*
Disk
21
Local backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
Storage (SCSI) Bus
I/O (PCI) bus
HBA
Memory
*
Disk
22
Local backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
Storage (SCSI) Bus
I/O (PCI) bus
HBA
Memory
*
Disk
23
Local backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
Storage (SCSI) Bus
I/O (PCI) bus
HBA
Memory
*
Disk
24
Local backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
Storage (SCSI) Bus
I/O (PCI) bus
HBA
Memory
*Disk
25
Remote backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
I/O (PCI) bus
HBA
Memory
Network
Network Controller
Storage (SCSI) Bus
Disk*
26
Remote backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
I/O (PCI) bus
HBA
Memory
*
Network
Network Controller
Storage (SCSI) Bus
Disk
27
Remote backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
I/O (PCI) bus
HBA
Memory
*
Network
Network Controller
Storage (SCSI) Bus
Disk
28
Remote backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
I/O (PCI) bus
HBA
Memory*
Network
Network Controller
Storage (SCSI) Bus
Disk
29
Remote backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
I/O (PCI) bus
HBA
Memory
*
Network
Network Controller
Storage (SCSI) Bus
Disk
30
Remote backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
I/O (PCI) bus
HBA
Memory
*
Network
Network Controller
Storage (SCSI) Bus
Disk
31
Remote backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
I/O (PCI) bus
HBA
Memory
*Network
Network Controller
Storage (SCSI) Bus
Disk
32
Remote backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
I/O (PCI) bus
HBA
Memory
*Network
Network Controller
Storage (SCSI) Bus
33
Remote backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
I/O (PCI) bus
HBA
Memory
Network
Network Controller
Storage (SCSI) Bus
*
34
Remote backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
I/O (PCI) bus
HBA
Memory
*
Network
Network Controller
Storage (SCSI) Bus
35
Remote backup data flow: Systemic view
CPU-memory bus
Cache
CPU
Bridge
I/O (PCI) bus
HBA
Memory
*Network
Network Controller
Storage (SCSI) Bus
© March 9, 2004 Novell Inc.36
Available network bandwidth
Ethernet wire capacity in theory• 100 Mbps = 94.44 Mb/s ~= 676 MB/min
– Accommodating space loss due to protocol headers
– Accommodating time loss due to inter-frame gap• 1000 Mbps ~= 6755 MB/min
Practical dedicated Ethernet wire capacity• 100 Mbps – ~656 MB/min – 97%• 1000 Mbps – ~6417 MB/min – 95%
“Ideal bulk transfer applications should be bandwidth bound”
© March 9, 2004 Novell Inc.37
Available network bandwidth
Ethernet wire capacity in theory• 100 Mbps = 94.44 Mb/s ~= 676 MB/min
– Accommodating space loss due to protocol headers
– Accommodating time loss due to interframegap• 1000 Mbps ~= 6755 MB/min
Practical dedicated Ethernet wire capacity• 100 Mbps – ~656 MB/min – 97%• 1000 Mbps – ~6417 MB/min – 95%
“Ideal bulk transfer applications should be bandwidth bound”
© March 9, 2004 Novell Inc.38
Agenda
Technology overview
SMS technology advancements
Troubleshooting backup on NetWare
Making the most out of your storage
Related technology
© March 9, 2004 Novell Inc.39
How fast can remote backups go?
We can do 1TB in 3.5 Hours on the wire at 4.8GB/Min, with the streaming SMDR prototype
© March 9, 2004 Novell Inc.40
Streaming SMDR prototype results
“Remote matching local performance...”
Setup#1 Setup#2 Setup#3
470 1080 2050
510 1100 2100 Streaming PrototypeLocal
05/24/1901 01/04/1903 09/30/190504/14/1901 12/15/1902 08/11/1905
0
500
1000
1500
2000
2500
Thr
ough
put
(MB
/Min
)
8.5 1.9 2.4
Streaming PrototypeLocal% Difference
Setup #1: 2GB SYS volume with an average file size of 27KB
Setup #2: 2GB SYS volume with an average file size of 40KB
Setup #3: 6GB volume with 6000 files each being 1MB in size
© March 9, 2004 Novell Inc.41
Streaming SMDR prototype results
“Remote matching local performance...”
Setup#1 Setup#2 Setup#3
470 1080 2050
510 1100 2100 Streaming PrototypeLocal
05/24/1901 01/04/1903 09/30/190504/14/1901 12/15/1902 08/11/1905
0
500
1000
1500
2000
2500
Thr
ough
put
(MB
/Min
)
8.5 1.9 2.4
Streaming PrototypeLocal% Difference
Setup #1: 2GB SYS volume with an average file size of 27KB
Setup #2: 2GB SYS volume with an average file size of 40KB
Setup #3: 6GB volume with 6000 files each being 1MB in size
© March 9, 2004 Novell Inc.42
Streaming SMDR prototype results
“Remote matching local performance...”
Setup#1 Setup#2 Setup#3
470 1080 2050
510 1100 2100 Streaming PrototypeLocal
05/24/1901 01/04/1903 09/30/190504/14/1901 12/15/1902 08/11/1905
0
500
1000
1500
2000
2500
Thr
ough
put
(MB
/Min
)
8.5 1.9 2.4
Streaming PrototypeLocal% Difference
Setup #1: 2GB SYS volume with an average file size of 27KB
Setup #2: 2GB SYS volume with an average file size of 40KB
Setup #3: 6GB volume with 6000 files each being 1MB in size
© March 9, 2004 Novell Inc.43
Streaming SMDR prototype results
Improvements over existing SMDR
0
500
1000
1500
2000
2500M
B /
min Existing SMDR
Prototype 1
Prototype 2
Existing SMDR 212 305 560
Prototype 1 216 440 1550
Prototype 2 470 1080 2050
Setup 1 Setup 2 Setup 3
© March 9, 2004 Novell Inc.44
Streaming SMDR prototype results
“Improved remote restore performance”
0
500
1000
1500
2000
2500
3000M
B /
min Existing SMDR
Prototype 1
Prototype 2
Existing SMDR 600
Prototype 1 1800
Prototype 2 2600
© March 9, 2004 Novell Inc.45
How Streaming SMDR works
Alleviates latency effects on throughput by anticipation
Keeps the network pipe full
Better network bandwidth utilization
Read ahead, lazy write
Decouples the interface from the implementation
© March 9, 2004 Novell Inc.46
Who gets the benefit
Applications that use SMDR for remote backups
Data movement applications• Novell migration wizard• Novell server consolidation utility• Novell DFS volume move/volume split• Commercial copy applications
© March 9, 2004 Novell Inc.47
SMDR: Features
Existing Features• Provides location transparency• Abstracts communication details, supports both
TCP/IP and IPX/SPX• Advertisement/Name resolution• Synchronous RPC model
Recent Features• Multiple IP address support• DNS support for name resolution• Policy based discovery/name resolution mechanism
order• iManager and server console based configuration
© March 9, 2004 Novell Inc.48
SMDR: Features
Existing Features• Provides location transparency• Abstracts communication details, supports both
TCP/IP and IPX/SPX• Advertisement/Name resolution• Synchronous RPC model
Recent Features• Multiple IP address support• DNS support for name resolution• Policy based discovery/name resolution mechanism
order• iManager and server console based configuration
© March 9, 2004 Novell Inc.49
TSAFS: Features
Scalability• Serial nature of backup does not limit scalability• Re-architected TSA to scale with disk configurations
Manageability• Improved manageability, helps identify performance sweet
spots• iManager and server console based configuration
Availability• Cluster enabled, supports continuation of backup after a
fail-over/fail-back• Not just with NCS, even with server restarts
Ships with NetWare® 6.5, NetWare® 6.0 SP4
© March 9, 2004 Novell Inc.50
TSAFS: Features
Scalability• Serial nature of backup does not limit scalability• Re-architected TSA to scale with disk configurations
Manageability• Improved manageability, helps identify performance sweet
spots• iManager and server console based configuration
Availability• Cluster enabled, supports continuation of backup after a
fail-over/fail-back• Not just with NCS, even with server restarts
Ships with NetWare® 6.5, NetWare® 6.0 SP4
© March 9, 2004 Novell Inc.51
TSAFS: Features
Scalability• Serial nature of backup does not limit scalability• Re-architected TSA to scale with disk configurations
Manageability• Improved manageability, helps identify performance sweet
spots• iManager and server console based configuration
Availability• Cluster enabled, supports continuation of backup after a
fail-over/fail-back• Not just with NCS, even with server restarts
Ships with NetWare® 6.5, NetWare® 6.0 SP4
© March 9, 2004 Novell Inc.52
Sample TSAFS test results
0
2000
4000
6000
8000Th
roug
hput
(MB
/min
)
TSA600 1380 1400
TSAFS 3300 6120
Tweaked 5880 6120
Setup #1 Setup #2
Setup #1: Ultra 160 controller with 4 disks using RAID 5, with PCI-X 64bit 100 MHzSetup #2: Ultra 160 controller with 6 disks using RAID 0, with PCI-X 64bit 100 MHz
TSAFS improvements and tweaking effects
© March 9, 2004 Novell Inc.53
Sample TSAFS test results
TSAFS improvements and tweaking effects
Setup #1: Ultra 160 controller with 4 disks using RAID 5, with PCI-X 64bit 100 MHzSetup #2: Ultra 160 controller with 6 disks using RAID 0, with PCI-X 64bit 100 MHz
0
2000
4000
6000
8000Th
roug
hput
(MB
/min
)
TSA600 1380 1400
TSAFS 3300 6120
Tweaked 5880 6120
Setup 1 Setup 2
© March 9, 2004 Novell Inc.54
Agenda
Technology overview
SMS technology advancements
Troubleshooting backup on NetWare
Making the most out of your storage
Related technology
© March 9, 2004 Novell Inc.55
General Guidelines
Rules of Thumb• Eliminate one component at a time• Try and eliminate broader sub systems first
– For e.g., target the disk or network first
Use tools• DPMETERM• TSATEST• NetWare Remote Manager (NRM)• iManager and Server Console
© March 9, 2004 Novell Inc.56
General Guidelines
Rules of Thumb• Eliminate one component at a time• Try and eliminate broader sub systems first
– For e.g., target the disk or network first
Use tools• DPMETERM• TSATEST• NetWare Remote Manager (NRM)• iManager and Server Console
© March 9, 2004 Novell Inc.57
Troubleshooting demo
Establish a baseline• TSATEST• Backup application
Can it be improved?• DPMETERM
Can we go beyond this?• Using NRM• Using TSATEST• Tweaking TSAFS, ReadThreadsPerJob
and ReadBufferSize
© March 9, 2004 Novell Inc.58
Troubleshooting demo
Establish a baseline• TSATEST• Backup application
Can it be improved?• DPMETERM
Can we go beyond this?• Using NRM• Using TSATEST• Tweaking TSAFS, ReadThreadsPerJob
and ReadBufferSize
© March 9, 2004 Novell Inc.59
Troubleshooting demo
Establish a baseline• TSATEST• Backup application
Can it be improved?• DPMETERM
Can we go beyond this?• Using NRM• Using TSATEST• Tweaking TSAFS, ReadThreadsPerJob
and ReadBufferSize
© March 9, 2004 Novell Inc.60
Troubleshooting SMDR issues
Use tools for performance issues• PERFTEST• NETMON• Other packet capture/analysis utilities
Discovery/Name Resolution issues• SLP• SMDR NEW• SMDR CONN
“Ensure simultaneous remote backups do not flood your network”
© March 9, 2004 Novell Inc.61
Troubleshooting SMDR issues
Use tools for performance issues• PERFTEST• NETMON• Other packet capture/analysis utilities
Discovery/Name Resolution issues• SLP• SMDR NEW• SMDR CONN
“Ensure simultaneous remote backups do not flood your network”
© March 9, 2004 Novell Inc.62
Troubleshooting SMDR issues
Use tools for performance issues• PERFTEST• NETMON• Other packet capture/analysis utilities
Discovery/Name Resolution issues• SLP• SMDR NEW• SMDR CONN
“Ensure simultaneous remote backups do not flood your network”
© March 9, 2004 Novell Inc.63
Troubleshooting TSAFS issues
TSAFS debug log• TSAFS /SmsDebug=<Value> /SmsDebug2=<Value>
Server memory usage• TSAFS /CacheMemoryThreshold=<Percentage>
Other factors• Competing software (Compression/Anti-Virus)• CPU usage/Busy threads, NRM I/O statistics
© March 9, 2004 Novell Inc.64
Troubleshooting TSAFS issues
TSAFS debug log• TSAFS /SmsDebug=<Value> /SmsDebug2=<Value>
Server memory usage• TSAFS /CacheMemoryThreshold=<Percentage>
Other factors• Competing software (Compression/Anti-Virus)• CPU usage/Busy threads, NRM I/O statistics
© March 9, 2004 Novell Inc.65
Troubleshooting TSAFS issues
TSAFS debug log• TSAFS /SmsDebug=<Value> /SmsDebug2=<Value>
Server memory usage• TSAFS /CacheMemoryThreshold=<Percentage>
Other factors• Competing software (Compression/Anti-Virus)• CPU usage/Busy threads, NRM I/O statistics
© March 9, 2004 Novell Inc.66
Agenda
Technology overview
SMS technology advancements
Troubleshooting backup on NetWare
Making the most out of your storage
Related technology
67
Power of aggregation
CPU-memory bus
D
PCI bus
D D D D
D D D D D
D D D D D
D D D D D
D D D D D
D D D D D
D D D D D
D D D D D
D D D D D
D D D D D
D D D D D
D D D D D
SCSI/RAID Controller
SCSI Controller
PCI bus...
...
PCI busPCI busPCI bus ......
Fibre Channel HBA
© March 9, 2004 Novell Inc.68
Suggestions on exploiting bandwidth
Create parallel I/O paths for high performance• Load balance peer-to-peer PCI buses
– Distribute your storage and network controllers
• Use multiple channels per HBA and stripe across them
Use technology to suit performance needs• 1Gbps networks are becoming common, 4Gbps cards are out
in the market, 10Gbps is close to reality
Mix and match with care• Do not use Ultra 320 disks with an Ultra 160 SCSI controller• Performance is limited by the lowest bandwidth component in
your data pipe
Design your storage configuration for high availability as
well
© March 9, 2004 Novell Inc.69
Suggestions on exploiting bandwidth
Create parallel I/O paths for high performance• Load balance peer-to-peer PCI buses
– Distribute your storage and network controllers
• Use multiple channels per HBA and stripe across them
Use technology to suit performance needs• 1Gbps networks are becoming common, 4Gbps cards are out
in the market, 10Gbps is close to reality
Mix and match with care• Do not use Ultra 320 disks with an Ultra 160 SCSI controller• Performance is limited by the lowest bandwidth component in
your data pipe
Design your storage configuration for high availability as
well
© March 9, 2004 Novell Inc.70
Suggestions on exploiting bandwidth
Create parallel I/O paths for high performance• Load balance peer-to-peer PCI buses
– Distribute your storage and network controllers
• Use multiple channels per HBA and stripe across them
Use technology to suit performance needs• 1Gbps networks are becoming common, 4Gbps cards are out
in the market, 10Gbps is close to reality
Mix and match with care• Do not use Ultra 320 disks with an Ultra 160 SCSI controller• Performance is limited by the lowest bandwidth component in
your data pipe
Design your storage configuration for high availability as
well
© March 9, 2004 Novell Inc.71
Suggestions on exploiting bandwidth
Create parallel I/O paths for high performance• Load balance peer-to-peer PCI buses
– Distribute your storage and network controllers
• Use multiple channels per HBA and stripe across them
Use technology to suit performance needs• 1Gbps networks are becoming common, 4Gbps cards are out
in the market, 10Gbps is close to reality
Mix and match with care• Do not use Ultra 320 disks with an Ultra 160 SCSI controller• Performance is limited by the lowest bandwidth component in
your data pipe
Design your storage configuration for high availability as
well
© March 9, 2004 Novell Inc.72
Agenda
Technology overview
SMS technology advancements
Troubleshooting backup on NetWare
Making the most out of your storage
Related technology
© March 9, 2004 Novell Inc.73
Related Technology
File Copy On Write• Permits open file backup
Pool Snapshots• Point in time data snapshots
Freeze/Thaw• Permits data consistency across compliant
applications
Versioning• User managed versioning of files, easing restores
© March 9, 2004 Novell Inc.74
Related Technology
File Copy On Write• Permits open file backup
Pool Snapshots• Point in time data snapshots
Freeze/Thaw• Permits data consistency across compliant
applications
Versioning• User managed versioning of files, easing restores
© March 9, 2004 Novell Inc.75
Related Technology
File Copy On Write• Permits open file backup
Pool Snapshots• Point in time data snapshots
Freeze/Thaw• Permits data consistency across compliant
applications
Versioning• User managed versioning of files, easing restores
© March 9, 2004 Novell Inc.76
Related Technology
File Copy On Write• Permits open file backup
Pool Snapshots• Point in time data snapshots
Freeze/Thaw• Permits data consistency across compliant
applications
Versioning• User managed versioning of files, easing restores
© March 9, 2004 Novell Inc.77
SMS on Linux
Location Transparency
Enterprise scale, logical backup
Multiple file system support
Rich meta-data support
© March 9, 2004 Novell Inc.78
SMS: In Summary
One-stop Network wide Storage Management Solution
• High Performance, Scalable
• Extensible solution
• Multi-Latency/Bandwidth
environments
• True backup virtualization
© March 9, 2004 Novell Inc.79
Backups
Backup is a second class citizen…
Or is it?
General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. Novell, Inc., makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.
No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.