the storage evolution: from blocks, files and objects to object … · 2020-05-10 · education the...
TRANSCRIPT
![Page 1: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/1.jpg)
EDUCATION
The Storage Evolution:From Blocks, Files and Objects to Object Storage Systems
Christian Bandulet, Sun Microsystems
![Page 2: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/2.jpg)
EDUCATION
2The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
SNIA Legal Notice
• The material contained in this tutorial is copyrighted by the SNIA.
• Member companies and individuals may use this material in presentations and literature under the following conditions:– Any slide or slides used must be reproduced without
modification– The SNIA must be acknowledged as source of any
material used in the body of any document containing material from these presentations.
• This presentation is a project of the SNIA Education Committee.
![Page 3: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/3.jpg)
EDUCATION
3The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Abstract
The Storage Evolution: From Blocks, Files and Objects to Object Storage SystemsThis session will appeal to CIOs, CTOs, Consultants, System Architects and Technologists, and those that are seeking a fundamental understanding of the emerging object-based storage technologies. The audience will gain insight into the basic differences of block-, file- and object-based data access methods. The session will delve into the benefits of object storage and it's value and also outline how this technology might impact future directions of storage system architectures.
![Page 4: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/4.jpg)
EDUCATION
4The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Topics
• Block-Based Data Access• File-Based Data Access• Object-Based Data Access
– Object-Based Storage Devices (OSD)– Object Storage Systems
• Object Storage Server (OSS)• Content Addressable Storage (CAS)• Content Aware Storage (CAS)
• Intelligent Storage Nodes (ISN)
![Page 5: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/5.jpg)
EDUCATION
5The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Topics
• Block-Based Data Access• File-Based Data Access• Object-Based Data Access
– Object-Based Storage Devices (OSD)– Object Storage Systems
• Object Storage Server (OSS)• Content Addressable Storage (CAS)• Content Aware Storage (CAS)
• Intelligent Storage Nodes (ISN)
![Page 6: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/6.jpg)
EDUCATION
6The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
The Data Access Taxonomy
DataAccess
Block ? ?
SCSI, SAS,FCP, iSCSI,SRP, ATA
SATA
![Page 7: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/7.jpg)
EDUCATION
7The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
The Block Paradigm
Physical Blocks:e.g. 512 bytes
SCSI, SAS, FCP, SRP, iSCSI, ATA, SATA
Client Interface
0 1 2 3 4
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
![Page 8: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/8.jpg)
EDUCATION
8The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Topics
• Block-Based Data Access• File-Based Data Access• Object-Based Data Access
– Object-Based Storage Devices (OSD)– Object Storage Systems
• Object Storage Server (OSS)• Content Addressable Storage (CAS)• Content Aware Storage (CAS)
• Intelligent Storage Nodes (ISN)
![Page 9: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/9.jpg)
EDUCATION
9The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
The Data Access Taxonomy
DataAccess
Block ? File
SCSI, SAS,FCP, iSCSI,SRP, ATA
SATA
Local FS,Distributed FS:
(NFS, CIFS, WAFS)Shared FS:
(Cluster FS, SAN FS)Global, Distributed
& Parallel FS
Check outSNIA Tutorial:AdvancedData Sharing
![Page 10: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/10.jpg)
EDUCATION
10The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Local File SystemsOne more level of indirection
Application Server
Data File system structure (i.e. inodes)
Physical Blocks on Disk
• file/directory management(~10%of workload)
• block/sector management (~90%of workload)
Logical Blocks on Disk0
16 17 18 19 20
0 1 2 3 45 6 7 8 9
10 11 12 13 1415 16 17 18 19
1 2 3 4 5 6 7 8 9 10
11
12
13
Mapping
Physical Disk
![Page 11: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/11.jpg)
EDUCATION
11The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Distributed File Systemse.g. NAS with NFS,CIFS Protocol
IPIP
SANSAN
NAS Appliance
Data
VDISK
Leaving block management (i.e. 90% to dedicated NAS appliance results in lean clients!
Might becomea bottleneck – hard
to scale.
Physical DiskPhysical Disk
11 12 13 1
Disk Array
11 12 13 1
Physical DiskPhysical Disk
11 12 13 1
11 12 13 1
Physical DiskPhysical Disk
11 12 13 1
Disk Array
11 12 13 1
Physical DiskPhysical Disk
11 12 13 1
11 12 13 1
Physical DiskPhysical Disk
11 12 13 1
Disk Array
11 12 13 1
Physical DiskPhysical Disk
11 12 13 1
11 12 13 1
Physical DiskPhysical Disk
11 12 13 1
Disk Array
11 12 13 1
Physical DiskPhysical Disk
11 12 13 1
11 12 13 1
Application Server Application Server Application Server Application Server Application Server Application Server
![Page 12: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/12.jpg)
EDUCATION
12The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Why should accessing and scaling file storage be any different in enterprises?
NAS N
AS
NAS Aggregation/Virtualization Global Namespace
![Page 13: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/13.jpg)
EDUCATION
13The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
NAS AggregationGlobal Namespace
NAS Appliance
IP
Provides Global
Namespace
Global Namespace
SAN
Data
NAS Appliance
SAN
Data
NAS Appliance
SAN
Data
NAS Appliance
SAN
Data
NAS Appliance
SAN
Data
NAS Appliance
Application Server Application Server Application Server Application Server Application Server Application Server
NAS Router
Data
![Page 14: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/14.jpg)
EDUCATION
14The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Global Namespace
NAS Clusteraka Tightly Coupled NAS
Application Server Application Server Application Server Application Server Application Server Application Server
SAN
NAS Appliance
Data
NAS Appliance
Data
NAS Appliance
Data
NAS Appliance
Data
System AreaNetwork
Single Data ImageParallel File System
VirtualIP Address
System AreaNetwork
![Page 15: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/15.jpg)
EDUCATION
15The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Scalable NAS aka Loosely Coupled NASGlobal Namespace with NFSv4.1 and pNFS
Application Server
IP
StorageDevice
In-Band NAS:
IP
Out-of-Band NAS:
Scalable NASLoosely Coupled NAS Cluster
Application Server Application ServerApplication Server Application ServerApplication Server Application ServerApplication Server Application ServerApplication Server Application ServerApplication Server
SAN SAN
StorageDevice
Data
NAS Appliance
Data
NAS Appliancewith NFSv4.1
pNFS extensions
Storage Protocol:SCSI, NFS, OSD
![Page 16: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/16.jpg)
EDUCATION
16The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Topics
• Block-Based Data Access• File-Based Data Access• Object-Based Data Access
– Object-Based Storage Devices (OSD)– Object Storage Systems
• Object Storage Server (OSS)• Content Addressable Storage (CAS)• Content Aware Storage (CAS)
• Intelligent Storage Nodes (ISN)
![Page 17: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/17.jpg)
EDUCATION
17The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
The Data Access Taxonomy
DataAccess
Block Object File
SCSI, SAS,FCP, iSCSI,SRP, ATA
SATA
Local FS,Distributed FS:
(NFS, CIFS, WAFS)Shared FS:
(Cluster FS, SAN FS)Global, Distributed
& Parallel FS
OSDCASOSS
![Page 18: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/18.jpg)
EDUCATION
18The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
The Old Block Paradigm
Write Block #Write Block #Write Block #Write Block #Write Block #Write Block #Write Block # Status
Return 0
Store
Read Block #Read Block #Read Block #Read Block #Read Block #Read Block #Read Block # Data
Retrieve
0 1 2 3 4
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
0 1 2 3 4
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
![Page 19: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/19.jpg)
EDUCATION
19The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
The New Object Paradigm
Data
Object
OID
Store
OID
Object
Data
Retrieve
![Page 20: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/20.jpg)
EDUCATION
20The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
The New Object Paradigm(cont'd)
Data
Object
OID
Store
• WRITE 26,763 Bytes• QoS = High• Description = “X-Ray”• Retention = 50 years• Access Key = *&^%#• Data Payload........
● Object Storage Responsibilities:● Space Management● Access Control (Identity Mgmt)● QoS Management● Cache, Backup● Policy Migration, Retention
![Page 21: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/21.jpg)
EDUCATION
21The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Self-Contained Objects
self-contained snail
Metadata: creation date/time; ownership, size...
Attributes – inferred:access patterns, content, indexes...
Attributes – user supplied:retention, QoS...
Metadata
Attributes
Object
Data
OIDObject
Object
Object Object
Object
ObjectObject
Object
![Page 22: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/22.jpg)
EDUCATION
22The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Block Access - Inodes• The inode contains a few block numbers to ensure efficient access
to small files. Access to larger files is provided via indirect blocks that contain block numbers
0 1 2 3 4
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
Host
Data BlocksInode
direct 0 data blockdirect 1 data blockdirect 2 data blockdirect 3 data blockdirect 4 data blockdirect 5 data blockdirect 6 data blockdirect 7 data blockdirect 8 data blockdirect 9 data block
singleindirect
doubleindirect
tripleindirect
data block
data block
data block
![Page 23: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/23.jpg)
EDUCATION
23The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Host
Data BlocksInode
Block Access – Inodes (cont'd)
direct 0 data blockdirect 1 data blockdirect 2 data blockdirect 3 data blockdirect 4 data blockdirect 5 data blockdirect 6 data blockdirect 7 data blockdirect 8 data blockdirect 9 data block
singleindirect
doubleindirect
tripleindirect
data block
data block
data block
• The inode also contains file attributes...
File OwnerFile Type
PermissionsLast Access
Size# of links
...File Attributes:
0 1 2 3 4
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
![Page 24: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/24.jpg)
EDUCATION
24The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Inodes vs. Objects
Block Device
1 2
5 5
3 4
6 8
iSCSIFile System
Inodes:Block PointersFile Attributes
Object Device
iSCSIFile System
Objects:Block Pointers
Object Attributes
1 2
3
4
56
iSCSIObject IDs
Host Host
![Page 25: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/25.jpg)
EDUCATION
25The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Object Autonomy
Block Device
iSCSIFile System
Inodes:Block PointersFile Attributes
Object Device
iSCSIFile System
Objects:Block Pointers
Object Attributes
1 2
3
4
56
iSCSIObject IDs
Host HostStorage becomes autonomous:
– capacity planning– load balancing– backup– QoS, SLAs– understand data/object grouping– aggressive pre-fetching– thin provisioning– search– compression/de-
duplication/encryption– strong security– compliance/retention/secure
delete– availability/replication– audit...
1 2
5 5
3 4
6 8
![Page 26: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/26.jpg)
EDUCATION
26The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Data SharingHomogeneous/Heterogeneous
Object Device
iSCSIFile System
Objects:Block Pointers
Object Attributes
1 2
3
4
56
iSCSIObject IDs
Host
Block Device
iSCSIFile System
Inodes:Block PointersFile Attributes
HostiSCSIFile System
Inodes:Block PointersFile Attributes
HostiSCSIFile SystemiSCSIObject IDs
Host
heavy dutyCluster/SANfile system1 2
5 5
3 4
6 8
![Page 27: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/27.jpg)
EDUCATION
27The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Data Migration - ILMHomogeneous/Heterogeneous
Object Device
iSCSIFile System
Objects:Block Pointers
Object Attributes
1 2
3
4
56
iSCSIObject IDs
Host OID
Object
Object Device
iSCSIFile System
Objects:Block Pointers
Object Attributes
1 2
3
4
56
iSCSIObject IDs
Host OID
Object
Object Device
iSCSIFile System
Objects:Block Pointers
Object Attributes
1 2
3
4
56
iSCSIObject IDs
Host OID
Object
...
...
![Page 28: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/28.jpg)
EDUCATION
28The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Additional Layer of Security
Block Device
iSCSI
Object Device
1 2
3
4
56
Host
1 2
5 5
3 4
6 8
Host
• strong security via external service– authentication– authorization – NIS, LDAP....
• fine granularity– per object
![Page 29: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/29.jpg)
EDUCATION
29The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
0 1 2 3 4
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
Object
Object
Object
Object
Object
Object
ObjectObject
Living in a Flat Namespace
TraditionalHierarchical
Flat
File names / inodes Objects / OIDs
![Page 30: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/30.jpg)
EDUCATION
30The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Object Decomposition
Object Layout
OSD OSD OSD OSD
ObjectComponent
ObjectComponent
ObjectComponent
ObjectComponent
MDSOID:LayoutOID:LayoutOID:LayoutOID:LayoutOID:LayoutOID:LayoutOID:Layout
OID
Layout
![Page 31: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/31.jpg)
EDUCATION
31The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Multiple Referenced Objects
iSCSIOID Y
iSCSICustomer ID
iSCSIPreferences
iSCSINext Store
iSCSIRevenue
iSCSICredit History
ObjectData
iSCSILink to OID X
iSCSIOID Z
iSCSIPatient ID
Metadata Record
iSCSIExamination
iSCSIDate
iSCSIDiagnose
iSCSITherapy
iSCSILink to OID X
Metadata Record Reference 2
Reference 3
iSCSIOID W
iSCSIEmployeee ID
Metadata Record
iSCSIJob Level
iSCSISalary
iSCSIStock Options
iSCSIDepartment
iSCSILink to OID X
Reference 1
![Page 32: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/32.jpg)
EDUCATION
32The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Virtual ViewVirtual FS
Virtual ViewVirtual FS
iSCSIMetadata Record
Object ...Object Object ObjectObject
Virtual View / Virtual File Systems
iSCSIMetadata Record
iSCSIMetadata Record
Virtual ViewVirtual FS
![Page 33: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/33.jpg)
EDUCATION
33The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
0 1 2 3 4
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
Virtual View / Virtual File Systems(cont'd)
Traditional Flat Virtual
Virtual View A Virtual View B
Objects / OIDs
File names / inodes
Object
Object
Object
Object
Object
Object
ObjectObject
Object
Object
Object
Object
Object
Object
ObjectObject
![Page 34: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/34.jpg)
EDUCATION
34The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Topics
• Block-Based Data Access• File-Based Data Access• Object-Based Data Access
– Object-Based Storage Devices (OSD)– Object Storage Systems
• Object Storage Server (OSS)• Content Addressable Storage (CAS)• Content Aware Storage (CAS)
• Intelligent Storage Nodes (ISN)Check out
SNIA Tutorial:Object-basedStorage Devices
![Page 35: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/35.jpg)
EDUCATION
35The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
NASDNetwork Attached Secure Device
NAS Appliance
Data
Application Server Application Server Application Server Application Server Application ServerApplication Server
NASD
• bring the whole functionality of a NAS device down to a SCSI devices
![Page 36: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/36.jpg)
EDUCATION
36The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
American National Standards Institute
(ANSI)
International Committee
for InformationTechnology (INCITS**)
T10SCSI, SAS,
SSA, ALUOSD**, SRP*
T11FC, HIPPI,
IPI,FICON
Internet EngineeringTask Force
(www.ietf.org)
IP-Based StorageWork Group
FCIP, iFCP, mFCP,
iSCSI,iSNSSNMP and
MIBs
Note*SRP: SCSI RDMA ProtocolNote**OSD: Object-based Storage DevicesNote*** INCITS – formerly National Committee for Information Technology Standards (NCITS)
T1Sonet
T3ATA,
ATAPI
IEEE
T13S-ATA
Ethernet
Internet Societywww.isoc.org
The World of Standards
Check outSNIA Tutorial:Storage Networking Standards
![Page 37: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/37.jpg)
EDUCATION
37The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
ANSI T10 OSD SCSI Targets
OSD SCSI Controller OSD SCSI Controller
Disk Array
iSCSIFile SystemiSCSIObject IDs
Host
SAS,FCP, iSCSI...
iSCSIFile SystemiSCSIObject IDs
Host
Object
Object
Object
Object
Object
Object
ObjectObject
![Page 38: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/38.jpg)
EDUCATION
38The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Topics
• Block-Based Data Access• File-Based Data Access• Object-Based Data Access
– Object-Based Storage Devices (OSD)– Object Storage Systems
• Object Storage Server (OSS)• Content Addressable Storage (CAS)• Content Aware Storage (CAS)
• Intelligent Storage Nodes (ISN)
![Page 39: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/39.jpg)
EDUCATION
39The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Object Storage Server - OSS
OSS could be a migration path to provide object technologies to legacy block devices
OS (BSD, Solaris, Linux, Windows..)
Object InterfaceFile System
Block Back-End Device Driver
iSCSIFile SystemiSCSIObject IDs
Host
![Page 40: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/40.jpg)
EDUCATION
40The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Topics
• Block-Based Data Access• File-Based Data Access• Object-Based Data Access
– Object-Based Storage Devices (OSD)– Object Storage Systems
• Object Storage Server (OSS)• Content Addressable Storage (CAS)• Content Aware Storage (CAS)
• Intelligent Storage Nodes (ISN)
![Page 41: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/41.jpg)
EDUCATION
41The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Content Addressable Storage(CAS)
OS (BSD, Solaris, Linux...)
Proprietary Object API , XAM, WebDAV, NFS, FTP, CIFS...
File systemDatabase
• OIDs are hash values derived from the objects' content• Used as digital archive systems for long-term fixed content data• ECM applications used as data injection machines
![Page 42: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/42.jpg)
EDUCATION
42The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
RAINRedundant Array of Inexpensive/Independent Nodes
OS (BSD, Solaris, Linux...)
CAS
File systemDatabase
OS (BSD, Solaris, Linux...)
CAS
File systemDatabase
OS (BSD, Solaris, Linux...)
CAS
File systemDatabase
OS (BSD, Solaris, Linux...)
CAS
File systemDatabase
OS (BSD, Solaris, Linux...)
CAS
File systemDatabase
OS (BSD, Solaris, Linux...)
CAS
File systemDatabase
OS (BSD, Solaris, Linux...)
CAS
File systemDatabase
OS (BSD, Solaris, Linux...)
CAS
File systemDatabase
OS (BSD, Solaris, Linux...)
CAS
File systemDatabase
Single Data Image
![Page 43: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/43.jpg)
EDUCATION
43The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Data Placement
Node Node Node Node
Node Node Node Node
Node Node Node Node
Node Node Node Node
Object 1
= Data
= Parity
Object 2
Object 3
![Page 44: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/44.jpg)
EDUCATION
44The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Archiving vs Protection
• Data Protection is about Data Recovery
– e.g. RAID, snapshot, replication, backup...
• Data Archiving is about Data Discovery – Archiving requires data protection
– e.g. index, search, aggregate
Archiving: allow near instantaneous retrieval of images, and do it at tape-like prices
![Page 45: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/45.jpg)
EDUCATION
45The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
The New Challenges
• Fast store and retrieval• Availability • Reliability• Easiness to store, organize, retrieve and dispose• Complex data operations
– aggregate, join, view, sort, convert, encrypt...• Enhanced search operations• Flexibility to present data• Customized storage behavior• Reduced administration costs
New!
New!
New!
New!
New!
New!
New!
![Page 46: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/46.jpg)
EDUCATION
46The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Topics
• Block-Based Data Access• File-Based Data Access• Object-Based Data Access
– Object-Based Storage Devices (OSD)– Object Storage Systems
• Object Storage Server (OSS)• Content Addressable Storage (CAS)• Content Aware Storage (CAS)
• Intelligent Storage Nodes (ISN)
![Page 47: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/47.jpg)
EDUCATION
47The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
CAS: “Content Addressable Storage”
Enterprise Content ManagementInjection Engine
1100010110101111000
Object ID
![Page 48: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/48.jpg)
EDUCATION
48The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
CAS: “Content Aware Storage”
Enterprise Content ManagementInjection Engine
1100010110101111000
Object ID
Extensible Metadata
StorLets
MD/Content Search
![Page 49: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/49.jpg)
EDUCATION
49The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Content Aware StorageFlexibility
• user-developed trigger apps • synchronous:
– modify the behavior of store/retrieve/query/delete
– e.g. transcode, downsample, filter, watermark, extract metadata from file, headers, encrypt, audit log...
• Asynchronous:– process data at rest– e.g. capacity optimization,
scrubbing, migration, sanity check...
Data OID
Store
Input Procedure 1
Input Procedure 2
Input Procedure x
...
Asyn
chro
nous
Pro
cedu
re C
Asyn
chro
nous
Pro
cedu
re D
Asyn
chro
nous
Pro
cedu
re A
Asyn
chro
nous
Pro
cedu
re B
OID Data
Retrieve
Output Procedure x
Output Procedure 2
Output Procedure 1
...
Asyn
chro
nous
Pro
cedu
re C
Asyn
chro
nous
Pro
cedu
re D
Asyn
chro
nous
Pro
cedu
re A
Asyn
chro
nous
Pro
cedu
re B
Object Object
![Page 50: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/50.jpg)
EDUCATION
50The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Topics
• Block-Based Data Access• File-Based Data Access• Object-Based Data Access
– Object-Based Storage Devices (OSD)– Object Storage Systems
• Object Storage Server (OSS)• Content Addressable Storage (CAS)• Content Aware Storage (CAS)
• Intelligent Storage Nodes (ISN)
![Page 51: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/51.jpg)
EDUCATION
51The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
StorageManagement
Knowledge/Information
Management
ContentManagement
DataManagement
ILM, QoS, SLA,content/full text
search,content awareprocessing...
The Evolution of Data Processing
data services:volume mgmt
mirror, replicationsnapshot, backup,
DR...configuration,
topology,RAID...
semantics,heuristic,learning,
problem-solving...
![Page 52: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/52.jpg)
EDUCATION
52The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
The Active Digital ArchiveArchival Process Flow View
DataCreation
Classification/Triage
Access Control/Data Encoding /Signature gen
Pool / TieredMigration
Compression / Encryption DeDup
DataExpiration ......
Note: •Some steps can be done more than once•Not all steps are needed•Some steps can not be done out of order
Secure Deletion
![Page 53: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/53.jpg)
EDUCATION
53The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Storage Applications
DataCompression
Capacity Optimization
DataMigration
DataEncryption
Replication
GlobalFile
Systems
NAS
Archiving & Backup
TieredStorage
ContentAddressable
Storage
VirtualTape
Services
ContinuousData
Protection SanityCheck
![Page 54: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/54.jpg)
EDUCATION
54The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Migration of Storage Applications
Server
Network
Storage
DataCompression
Capacity Optimization
DataMigration
DataEncryption
Replication
GlobalFile
SystemsNAS
Archiving &
Backup
TieredStorage
ContentAddressable
Storage
VirtualTape
Services
ContinuousData
ProtectionSanityCheck
• Process the data where it lives...
![Page 55: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/55.jpg)
EDUCATION
55The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
The Evolution of Storage
Application
File System
Storage
DAS
Application
File System
Storage
SAN
Application
NAS
Application
File System
Object API
OSD
Storage
Application
File System
Object API
ISD*
Intelligence
Storage
File System
Storage
Note*: Intelligent Storage Node
![Page 56: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/56.jpg)
EDUCATION
56The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
The Vertical Consolidation
• Storage and server• Migration of data processing applications• No I/O is best I/O
Storage
Server
Storage
ServerServer
Server
Storver
StorageStorage
![Page 57: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/57.jpg)
EDUCATION
57The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
The Intelligent Storage Node
Presentation Layer (Storage Personality)Block I/F File I/F Object I/F
SCSI, FCP, iSCSI, SAS, SRP, SATA, VTL ...
FTP, NFS, CIFS, WebDAV, HTTP...
API, ANSI T10 OSD, CAS, XAM...
Storage Application Layer ( Data Services)Data Management Compliance Services Scheduled ServicesBackup, Classification,
Data Mining, Application Acceleration, Aggregation,
Virtualization,Sort, Views, Search,
Indexing, Data Movers, Distributed Namespace...
WORM, Audit, Retention, Secure Deletion...
De-Duplication,Hash Generation,
Verification,Snapshot, Replication...
Operating SystemFiles systems, HSM, Device Drivers, Clustering, Security....
Persistent StorageDisk, Tap, Library...
External ServicesLDAP
NIS
Active Directory
NTP
MDSGlobal Namespace
others
![Page 58: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/58.jpg)
EDUCATION
58The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Further Reference
• http://www.snia.org/tech_activities/workgroups• http://www.snia.org/apps/org/workgroup/osd/• http://www.snia.org/apps/org/workgroup/fcastwg/• http://www.snia-dmf.org/• http://www.t10.org/ftp/t10/drafts/osd• http://www.t10.org/ftp/t10/drafts/osd2• http://ietf.org/html.charters/webdav-charter.html• http://ietf.org/html.charters/nfsv4-charter.html• http://www.snia.org/education/tutorials/
![Page 59: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/59.jpg)
EDUCATION
59The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Q&A / Feedback• Please send any questions or comments on this presentation to
SNIA: [email protected]
Many thanks to the following individuals for their contributions to this tutorial.
SNIA Education Committee
Christian Bandulet, Sun Microsystems
![Page 60: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/60.jpg)
EDUCATION
60The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Appendix
![Page 61: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/61.jpg)
EDUCATION
61The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
File/record layerFile/record layer
Block layerBlock layer
Storage devices (disks, …)Storage devices (disks, …)
Database(dbms)
File system(FS)
Stor
age
dom
ain
Ser
vice
s
The SNIA Shared Storage Model
Ser
vice
s
Network
Host
DeviceBlock aggregation
Application
![Page 62: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/62.jpg)
EDUCATION
62The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
• Average Disk Metrics:– Capacity (36/72/146/300/400/500
GB...)– Rotation Speed(7200/10,000
/15,000 RPM)– Seek time (3 to 9 msec)– Sustained internal data transfer
rate (60 to 130 MB/s)– IOPS (200 – 300)– Cache (1MB-16MB)
Platter
Tracks (cylinders)
Coil
Interface: SCSI, SAS, FCP, SRP
iSCSI, ATA/SATA Actuator Arm w/ Read/Write Head(s)
Sector
Disk Drive Anatomy
![Page 63: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/63.jpg)
EDUCATION
63The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
~1956 first spinning hard drive (IBM RAMAC)
1956: 5 MB – 2000 bits/in2
2006: 500 GB ~ 200 Gb/in2
100.000.000 x areal density100.000 x capacity
Improvement in 50 years !
Technology Improvements
![Page 64: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/64.jpg)
EDUCATION
64The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Magnetic Disk RecordingLongitudinal Recording
Source: www.horison.com
Blocks, Sectors, Tracks
• Technology is ~50 years old!• First introduced with IBM
RAMAC 5MB in 1956• areal density increases 100% /
year since early 1990s• Disk areal density progress
slowed down in 2003 as recording challenges appeared
![Page 65: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/65.jpg)
EDUCATION
65The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Magnetic Disk RecordingPerpendicular/Vertical Recording
• Expected to delay Superparamagnetic Effect, not eliminate it...
Source: www.horison.com
![Page 66: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/66.jpg)
EDUCATION
66The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Logical Blocks & Physical BlocksLet's have a closer look....
Logical Blocks 0
Physical Blocks
Mapping
Client Interface
16 17 18 19 20
0 1 2 3 4
5 6 7 8 9
10 11 12 13 14
15 16 17 18 19
1 2 3 4 5 6 7 8 9 10
11
12
13
SCSI, SAS, FCP, SRP, iSCSI, ATA, SATA
![Page 67: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/67.jpg)
EDUCATION
67The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
NAS Appliancewith NFSv4.1pNFS extensions
Data
Global Namespace
NAS Appliancewith NFSv4.1
pNFS extensions
Data
Storage Device
Storage Device
Storage Device
Storage Device
Storage Device
NFSv4.1 + pNFS
NFSv4.1
File: NFSv4.1Block: iSCSI, FCP
OSD
one-to-one, stripe, concatenation
Storage Protocol:SCSI, NFS, OSD
MDS createsGlobal
Namespace
Control Protocol
Scalable NAS (cont’d)aka Loosely Coupled NASGlobal Namespace with NFSv4.1 and pNFS
NFSv4.1 client with pNFS
NFSv4.1 client with pNFS
NFSv4.1 client w/o pNFS
![Page 68: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/68.jpg)
EDUCATION
68The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
SCSI Standards Architecture
Source: www.t10.org
![Page 69: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/69.jpg)
EDUCATION
69The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Files Sharing with OSDs
VFS
osdfs
so
SCSI driver
iSCSI
TPC/IP
OSD Initiator
Application
VFS
osdfs
so
SCSI driver
iSCSI
TPC/IP
OSD Initiator
Application
VFS
osdfs
so
SCSI driver
iSCSI
TPC/IP
OSD Initiator
Application
TPC/IP
PolicyManager
SecurityManager
MDS OSD
iSCSI
TPC/IP
OSD Target
OSD
OSD
iSCSI
TPC/IP
OSD Target
OSD
OSD
iSCSI
TPC/IP
OSD Target
OSD
OSD
iSCSI
TPC/IP
OSD Target
OSD
KernelUser
VFS
osdfs
so
SCSI driver
iSCSI
TPC/IP
OSD Initiator
Application
Ethernet
KernelUser
TPC/IP
PolicyManager
SecurityManager
MDSOSD
iSCSI
TPC/IP
OSD Target
OSD
RAID
6
NIS, Kerberos,LDAP... Authentication (optional)
13
5
1. request OID layout from MDS2. request credential from security mgr3. authentication4. request capabilities from policy mgr5. return credential6. Send SCSI CDB
42
![Page 70: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/70.jpg)
EDUCATION
70The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Global, Distributed & Parallel FSWith Object Storage Server (OSS)
IP
Single Data Image
Application Server Application ServerApplication Server Application ServerApplication Server
StorageDevice
StorageDevice
StorageDevice
MDS
Object StorageServer
SAN
Object StorageServer
Object StorageServer
Ethernet, IB,Myrinet, Quadrics...
![Page 71: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/71.jpg)
EDUCATION
71The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Content Addressable/Aware Storageaka CAS
• OIDs are hash values derived from the objects’ content• Objective:
– Store large amounts of data reliably for long periods of time with fast access time to retrieve data
• Target applications:– Homeland security, Satellite imagery, Digital asset management, Medical
imaging, Digital photo services, Seismic data archival, Regulatory compliance, Media preservation
• Not used for:– Online Transaction Processing (OLTP), Enterprise Resource Planning (ERP),
Live database, Small scale file sharing
![Page 72: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/72.jpg)
EDUCATION
72The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
The Digital Archive Problem
• How do you store and organize 100 million things?
?
• Issues of:– Scale Performance
• capacity/workload balancing• automatic capacity expansion
– Organize data• manage metadata
– Search– Reliability/Availability
• data rebuild and/or failover– Cost (OPEX/CAPEX/TCO)– Technology refresh
![Page 73: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/73.jpg)
EDUCATION
73The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Content Aware StorageAttribute Awareness
• Object attributes are stored directly with data object by the application• Attributes are carried automatically between layers and across devices• When objects pass through a certain system layer or device, that layer can act on the values in
the attributes that it understands• All other attributes are passed along unmodified and not acted upon• e.g. Objects marked as high-reliability can be treated differently than objects marked as
temporary• Attributes should be dynamically changeable
Attributes Layer 0Layer 0 Attributes Layer 1 Attributes Layer 2 Attributes Layer 3
Attributes Layer 0Layer 1 Attributes Layer 1 Attributes Layer 2 Attributes Layer 3
Attributes Layer 0Layer 2 Attributes Layer 1 Attributes Layer 2 Attributes Layer 3
Attributes Layer 0Layer 3 Attributes Layer 1 Attributes Layer 2 Attributes Layer 3
![Page 74: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/74.jpg)
EDUCATION
74The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
Content Aware StorageObject Discovery
• Searchable metadata• Name-value based
– OID– Metadata– user derived attributes
• Content– full text search
Metadata
Attributes
Object
Data
OID
![Page 75: The Storage Evolution: From Blocks, Files and Objects to Object … · 2020-05-10 · EDUCATION The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems Christian](https://reader031.vdocuments.us/reader031/viewer/2022013021/5ebbee5d36a8515b776a2be1/html5/thumbnails/75.jpg)
EDUCATION
75The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems© 2007 Storage Networking Industry Association. All Rights Reserved.
• Database acceleration via offloading – health check, multi-level security, db reorganization, image copies, HSM, data mining...
• Business Continuity, Backup, Recovery, D2D2T, CDP...• Data Reduction
– Classification, essential vs non essential, single instance, compression...• Security
– Authentication, authorization, encryption• Data Transformation• Multiple Data Views
– workflow• Real-time Data Analytics
– indexing, search, sort, aggregation• Business Management
– Data Life Cycle, migration, compliance
Growing Storage Computation