red hat storage server: roadmap & integration with openstack · use case: content store for...
Post on 20-May-2020
9 Views
Preview:
TRANSCRIPT
Red Hat Storage Server: Roadmap & integration with OpenStack Sayan Saha Sr. Manager, Storage Product Management, Red Hat Vijay Bellur Architect, Gluster, Red Hat
How many hours of videos are uploaded to YouTube every
hour?
Use Case: File Store for Big Data Analytics
Profile
Leading automobile manufacturer
Store & Analyze sensor data for next generation automobiles
Expected data growth – 200 TB per week
5 PB total data store
Pain Points
Ability to scale rapidly with demand
Cost-effective scaling
Solution & Outcome
Selected Red Hat Storage Server to scale cost-effectively with demand
Leverage SMB & native client client for rapid data ingest
Use Case: Content Store for video production
Profile
Leading “video production as a service” provider
Ingest video from various sources, transcode in 9 formats, distribute via web servers
Pain Points Un-supported storage platform. Using community Gluster Storage platform key to their principal business process Could not afford to be on an un-supported platform
Solution & Outcome
Adopted Red Hat Storage Server to provide a cloud based video platform that enables media processing, distribution & IP streaming
Simplify delivery of content for TV and content providers.
Use Case: Disaster Recovery Using Replicated Storage
Profile
Public Transportation Provider
Filestore for storing data received from various sensors in the subway line that trains monitor
Replicated copy needed in a nearby data center for continuous availability of the monitoring service
Pain Points
Stuck with an End of Life proprietary clustered file system
Unreliable with weak replication capabilities
Wanted to re-use existing commodity hardware
Solution & Outcome
Adopted Red Hat Storage Server to provide a reliable monitoring solution for the subway system
Access from both Windows & Linux boxes
Secondary usage as a document store
Red Hat Storage Roadmap Strategy
Create the best open software-defined storage for File Serving
Create the best storage provider for
RHELOSP
Continue to do foundational work for Big Data Storage
Use Cases & Workloads Use Case Workload Description Example Workload
Instance(s)
Content Cloud – Storage for unstructured data
Storing & Accessing files with I/O patterns - Write infrequently, Read infrequently/many/never
OwnCloud, pydio, backup target for Commvault, Document & File Store
Storage Provider for OpenStack
Scale-out block & object Storage back-end for Cinder, Nova & Glance
Storage for Big Data Analytics
Log analytics, Big Data batch analytics & Big Sciences Analytics
Splunk, Hadoop Map-reduce, Illumina
Hardware Advances Helping the Cause
GlusterFS Upstream Innovation Pipeline
Red Hat Storage Technology Stack Libgfapi (not public)
GlusterFS Upstream Roadmap GlusterFS 3.5 (GA within a few days) Distributed geo-replication Quota Scalability Readdir ahead translator File snapshots for virtual machine image files libgfapi support for NFS Ganesha Brick failure detection Encryption at rest (Experimental) On wire Compression translator (Experimental)
GlusterFS 3.6 (June 2014) Volume Snapshot AFRv2 Data Classification SSL support Disperse Volumes (erasure coding) Heterogeneous Brick support Trash translator Better Peer Identification RDMA Improvements
GlusterFS 3.7 Predictions
Scalability for Big Data Content cloud enhancements - Sharding & Multi-protocol compatibility
Improvements for OpenStack Data Protection Improvements - Geo-replicated snapshots, bitrot detection
Support Btrfs features SSD/Flash leverage
Best open software-defined storage for file serving
Looking Back …
Red Hat Storage Server 2.0 launched June 2012 6 numbered updates released VM image store, performance & stability EOL June 2014 Red Hat Storage Server 2.1 launched September 2013 2 numbered updates released so far Quota, Geo-‐Rep, management Console, SMB 2.0 Current shipping release
Red Had Storage Server Roadmap Summary
3.0 (Denali) mid-year 2014 Theme: data protection & storage management • RHEL 6 & GlusterFS 3.6 • Volume Snapshots with user
serviceability • Monitoring Using Nagios • Hadoop Plug-In • 60 drives per server (up from 36) • Non-disruptive upgrade from previous
major version • Catalog/ID based Logging
3.1 (Everglades) 1H CY 2015
Theme: TCO reduction • RHEL 6 & GlusterFS 3.7 • 3-way replication • SSD support
• SSDs as bricks • SSDs for tiering
• Snapshot Enhancements • Support for RAID less h/w
configurations • NFSv4 full support (tentative) …
Zooming In on RHSS 3.0 (Denali)
Official Version: Red Hat Storage Server 3 Theme: Data protecQon & storage management Based on GlusterFS 3.6 Denali releases on RHEL 6 mainline not EUS Underlying filesystem: XFS Underlying Volume Management: dm-‐thinp
Snapshots
Point in Qme copy of a GlusterFS volume Create, list, status, info, restore & delete Support a maximum of 256 snapshots per volume A snapshot can be taken on one volume at a Qme Snapshot names need to be cluster-‐wide unique Management via CLI only User serviceable snapshots in scope but may need some more Qme to stabilize
Scope of Monitoring
Monitor RHSS Logical Entities – cluster, volume, brick, node Monitor Physical Entities – CPU, disk, network Alerting when critical components fails – SNMP Reporting – historical record of outages, events, notifications Trending – to enable capacity planning
Monitoring Using Nagios: Supported Use Cases
Use-‐Case 1: user has no exisQng monitoring infrastructure in place or does not use Nagios
Use-‐Case 2: user already has Nagios infrastructure in place – use plugins only
Use-‐Case 3: usage in conjuncQon with Red Hat Storage Console
Red Hat Storage as an Add-‐On – RPM based delivery Considering this packaging opQon to:
– Comply with corporate governance & security requirements
– Beber support storage co-‐resident applicaQons – Embedded use-‐cases – Ease of usage for channels & partners
GlusterFS + Other RPMs + XFS
RHEL
GlusterFS + Other RPMs + XFS
RHEL
GlusterFS + Other RPMs + XFS
RHEL
RHSS
RHSS
RHSS
RHSS ISO RHSS as an Add-ON
3-‐way replicaQon SSD support
Bricks using SSDs Tiering using RHEL’s dm-‐cache
Snapshot Enhancements Consistency Groups Snapshot scheduling
Support for RAID less h/w configuraQons NFSv4 full support
Zooming In on RHSS 3.1 (Everglades)
Bit Rot DetecQon & RestoraQon Erasure coding using the disperse translator MulQ-‐protocol support for NFS & SMB
pNFS server side support
Everglades pipeline …
The path to best storage for OpenStack
Best storage provider for OpenStack
Goal – Create the best storage offering for RHELOSP
What is needed? – Focus on requirements and feature set that are specific to OpenStack’s storage
substrate – Match product delivery model and life-‐cycle requirements to address
expectaQons of RHELOSP adopters – Clear offering for the market that tracks to the RHELOSP roadmap
Plan – Create a new product family that is exclusively targeted to address OpenStack’s
storage use cases 25
New Product Family: Red Hat Storage Server for RHELOSP Principal Product Line: Red Hat Storage
Product Family: Red Hat Storage Server for RHELOSP
Target Use-case: OpenStack Storage Provider
Delivery Model: RPM-only & CDN
Delivery Vehicle: Layered product on RHELOSP
SKU: Provides access to RHSS RPMs only
Pricing: Align with RHELOSP’s pricing model
Roadmap: OpenStack Storage & management feature set
Content
IniQally derived from Red Hat Storage Server – Start with the same core package set
Package set added/removed as needed Provisioning, configuraQon management & monitoring will be fully aligned with RHELOSP’s roadmap & capabiliQes
April 2014
Deploy & Configure with Foreman & Puppet
Post IceHouse
Cinder enhancements • Backup-Restore • Migrate • Per project user quotas Co-residency of storage & compute Native Swift
Juno
OpenStack-m • Deployment &
configuration for the undercloud
File-As-A-Service (Manila)
Red Hat Storage Server for RHELOSP Roadmap
RHELOSP + RHSS Co-‐Residency: Logical View
RHELOSP Node 1 Node 1
RHELOSP Node 2
RHELOSP Node N
RHSS for RHELOSP
Nova VMs 0..N
Local Storage
RHELOSP cinder
Nova VMs 0..N
Local Storage
Nova VMs 0..N
Local Storage
RHELOSP cinder RHELOSP cinder
What’s up with Big Data Storage?
Benefits of using RHSS for Hadoop AnalyVcs
NFS & FUSE support for data ingestion No single point of failure POSIX compliance Co-location of compute & data Multiple Volume Support Ability to partition data across multiple name spaces DR capabilities
Hadoop Plug-‐In Roadmap
Feb 2014
High Touch Beta Released on RHSS 2.1 • Single volume in a cluster with one brick per server
• Supported for HDP 2.0.6 and it’s 8 services: Pig, Hive, HBase, sqoop, Flume, Oozie, Mahout & Zookeeper
RHSS 3.0 (Denali) Promote the Hadoop Plug-In to GA status • Support multiple volumes per cluster running Hadoop
• Run in-place analytics on existing data in RHSS volumes
• Support Hadoop distro HDP 2.1 and it’s new services like Tez & Storm
Summary
Focus: File Serving on commodity hardware
New product family for RHELOSP storage
Foundational work for Big Data Workloads
Check Out Other Red Hat Storage Activities at The Summit Enter the raffle to win tickets for a $500 gift card or trip to LegoLand! - Entry cards available in all storage sessions - the more you attend, the more
chances you have to win! Talk to Storage Experts: Red Hat Booth (# 211)
- Infrastructure - Infrastructure-as-a-Service
Storage Partner Solutions Booth (# 605) Upstream Gluster projects - Developer Lounge
Thank You. Please fill out the feedback forms.
top related