ceph day london 2014 - best practices for ceph-powered implementations of storage as-a-service
DESCRIPTION
Paul Brooks and Steve L. Smith, DellTRANSCRIPT
Dell and CEPHSteve Smith: [email protected]@SteveSAtDell
Paul [email protected] @PaulBrookAtDell
Ceph Day LondonOctober 22nd 2014
agenda
• Why we are here. – we sell CEPH support
• You need hardware to sit this on. Here are some ideas
• Some best practice shared with CEPH colleagues this year
• A concept – (Research Data – would like your input)
Dell Corporation
Dell is a certified reseller of Red Hat-Inktank Services, Support and Training.
• Need to Access and buy Red Hat Services & Support?
15+ Years of Red Hat and Dell
• Red Hat 1-year /3-year subscription packages – Inktank Pre-Production subscription– Gold (24*7) Subscription
• Red Hat Professional Services– Ceph Pro services Starter Pack – Additional days services options – Ceph Training from Red Hat
3Confidential
Or…you can download CEPH for Free
Dell Corporation
Components Involved
http://docs.openstack.org/training-guides/content/module001-ch004-openstack-architecture.html
Dell Corporation
Dell OpenStack Cloud Solution
Stuff
You Get Stuff
Dell Corporation
Best Practices(well…….some)
Dell Corporation
With acknowledgement and thanks to Kyle and Mark at InkTank
Planning your Ceph Implementation
• Business Requirements– Budget considerations, organisational commitment– Replacing Enterprise SAN/NAS for cost saving– xaaS use cases for massive-scale, cost-effective storage – Avoid lock-in – use open source and industry standards– Steady-state vs. Spike data usage
• Sizing requirements– What is the initial storage capacity?– What is the expected growth rate?
• Workload requirements– Does the workload need high performance or it is more
capacity focused?– What are IOPS/Throughput requirements?– What applications will be running on Ceph cluster?– What type of data will be stored?
Dell Corporation
Architectural considerations – Redundancy and replication considerations
• Tradeoff between Cost vs. Reliability (use-case dependent)
• How many node failures can be tolerated?
• In a multi-rack scenario, should a whole rack failure be tolerated?
• Is there a need for multi-site data replication?
• Erasure coding (more capacity with the same raw disk. More CPU load)
• Plan for redundancy of the monitor nodes – distribute across fault zones
• 3 copies = 8 nines availability, less than 1 second downtime per year
• Many many things affect performance - in Ceph, above Ceph and below Ceph.
Dell Corporation
Understanding Your Workload
Dell Corporation
CEPH Architecture Refresh
Dell Corporation
Understanding Ceph (1)
Dell Corporation
Understanding Ceph (2)
Dell Corporation
Understanding The Storage Server
Dell Corporation
Multi-Site Issues
• Within a CEPH cluster RADOS enforces Strong Consistency
• The Writer process will wait for the ACK, which happens after the primary copy, the replicated copies and the journals have all been written.
• On a WAN this might extend latencies unacceptably.
• Alternatives
• For S3/Swift systems, federated gateways between CEPH clusters, RADOS uses Eventual Consistency.
• For remote backup use RBD with sync agents and incremental snapshots.
Dell Corporation
Recommended Storage Server ConfigurationsCEPH and InkTank recommendations are a bit out of date.
• CPU – 1 core GHz per OSD – so a 2 x 8-core Intel Haswell 2.0GHz server could support 32
OSDs– less for AMD
• Memory – 2GB per OSD– Must be ECC
• Disk Controller – SAS or SATA without extender for data and journal, RAID 1 for operating system disks
• Data Disks – Size doesn’t matter! Rebuilds happen across hundreds of placement groups.– 12 disks seems a good number
• Journal Disks – SSDs – write optimised
Dell Corporation
Intel Processors
Dell Corporation
Memory Considerations
C0 C2C1 C3
C4 C6C5 C7 C7C5 C6C4
C3C1 C2C0
• Always populate all channels – groups of 8• Anything less loses significant memory bandwidth
• Speed drops with 3DPC (sometimes 2DPC)• Use Dual Rank RDIMMs for maximum performance and expandability
• Important to PIN process and data to same NUMA node• But let OS processes float• Or try Hyperthreading
• Sensible memory is now 64GB (8 x 8GB RDIMMs) Dell Corporation
M
STORAGE NODE
RADOS GATEWAY
DreamObjects Hardware Specs
STORAGE NODE
STORAGE NODE
STORAGE NODE
STORAGE NODE
STORAGE NODE
x90
x4
MANAGEMENT NODE x3
LOAD BALANCER x2STORAGE NODE
Dell PowerEdge R5156 core AMD CPU, 32GB RAM2x 300GB SAS drives (OS)12x 3TB SATA drives2x 10GbE, 1x 1GbE, IPMI
Dell PowerEdge R4152x 1TB SATA1x 10GbE
MANAGEMENT NODE
Dell Corporation
Ceph Gateway Server
• Gateway does CRC32 and MD5 checksumming– Now included in Intel AVX2 on Haswell
• 64GB memory (minimum sensible)
• 2 separate 10GbE NICs, 1 for client comms, 1 for store/retrieve
• Make sure you have enough file handles, default is 100 - you should start at 4096!
• Load balancing with multiple gateways
Dell Corporation
Ceph Cluster Monitors
• Best practice to deploy monitor role on dedicated hardware– Not resource intensive but critical – Stewards of the cluster– Using separate hardware ensures no contention for resources
• Make sure monitor processes are never starved for resources– If running monitor process on shared hardware, fence off resources
• Deploy an odd number of monitors (3 or 5)– Need to have an odd number of monitors for quorum voting– Clusters < 200 nodes work well with 3 monitors– Larger clusters may benefit from 5– Main reason to go to 7 is to have redundancy in fault zones
• Add redundancy to monitor nodes as appropriate– Make sure the monitor nodes are distributed across fault zones– Consider refactoring fault zones if needing more than 7 monitors– Build in redundant power, cooling, disk
20
Dell Corporation
Networking Overview• Plan for low latency and high bandwidth
• Use 10GbE switches within the rack
• Use 40GbE uplinks between racks in the datacentre
• Use more bandwidth at the backend compared to the front end
• Enable Jumbo frames
• Replication is done by the storage not the client
• Client writes to primary and journal
• Primary writes to replicas through back end network
• Backend also does recovery and rebalancing
21
Dell Corporation
Potential Dell Server Hardware Choices
• Rackable Storage Node– Dell PowerEdge R720XD OR new 13g R730/R730xd
• Bladed Storage Node– Dell PowerEdge C8000XD Disk
and PowerEdge C8220 CPU
– 2x Xeon E5-2687 CPU, 128GB RAM
– 2x 400GB SSD drives (OS and optionally Journals)
– 12x 3TB NL SAS drive
– 2x 10GbE, 1x 1GbE, IPMI
• Monitor Node– Dell PowerEdge R415
– 2x 1TB SATA
– 1x 10GbE
Confidential22
Dell Corporation
Mixed Use Deployments
• For simplicity, dedicate hardware to specific role– That may not always be practical (e.g., small clusters)– If needed, can combine multiple functions on same
hardware
• Multiple Ceph Roles (e.g., OSD+RGW, OSD+MDS, Mon+RGW)– Balance IO-intensive with CPU/memory intensive roles– If both roles are relatively light (e.g., Mon and RGW) can
combine
• Multiple Applications (e.g., OSD+Compute, Mon+Horizon)– In OpenStack environment, may need to mix
components– Follow same logic of balancing IO-intensive with CPU
intensive23
Dell Corporation
Super-size CEPH• Lots of Disk space
• CEPH Rules apply
• Great for cold dark storage
• Surprisingly popular with Customers
• 3PB raw in a rack!
R730/R730XD or R720/R720XD
PowerVault JBOD
Dell Corporation
Other Design Guidelines
• Use simple components, don't buy more than you need. – Save money on RAID, redundant NICs, PS
and buy more disks
• Keep networks as flat as possible (East-West)– VLANs don't scale– Use Software Defined Networking for multi-tenancy in cloud
• Design the fault zones carefully for NoSPoF– Rack– Row– Datacentre
25
Dell Corporation
Research Data: Beta Slides
Dell Corporation
Concept: Get started?
Keep, Search,
Collaborate-Publish
Research Data & Publications
Digital - Pre-Publication (Any Format?)
Digital -Other (Any Format?)
Dell Corporation
Concept: Get started?
Keep, Search,
Collaborate-Publish
Research Data & Publications
Digital - Pre-Publication (Any Format?)
Digital -Other (Any Format?)
How tag metadata?
How long to store?
File types to store?
How Search?
Data Security?
How Collaborate?
Dell Corporation
Holding a tin cup below a Niagara Falls of data!"
Data keeps on coming &……. ..coming……& coming………..
Has anyone else had this problem and already solved it. ?
Open Source is best protection/longevity. “Web 2.0/Social has already solved scale-storage problem”
Dell Corporation
Solve problems one at a time
OpenStack Layer (Access)
CEPH Storage
Identity Manageme
nt
Governance
Policy & Control
PUBLISH:
Existing Publishing
routes
Dell Corporation
Solve problems one at a time
OpenStack Layer (Access)
CEPH Storage
Identity Manageme
nt
Governance
Policy & Control
PUBLISH:
Existing Publishing
routes
Start Here
Dell Corporation