vms at a tier-1 site egee’09, 21-09-2009 sander klous, nikhef
TRANSCRIPT
Contents• Introduction
– Who are we?• Motivation
– Why are we interested in VMs?– What are we going to do with VMs?
• Status– How do we approach this issue?– Where do we stand?
• Challenges
03-09-2009 BIG Grid - Virtualization working group 2
Introduction
• Collaboration between– NCF: national computing facilities– Nikhef: national institute for subatomic physics– NBIC: national bioinformatics center
• Participation from Philips, SARA, etc.
Goal:
“Enables access to grid infrastructures for scientific research in the Netherlands”
03-09-2009 BIG Grid - Virtualization working group 3
Motivation: Why Virtual Machines?• Site perspective
– Resource flexibility (e.g. SL4 / SL5)– Resource management
• Scheduling / multi-core / sandboxing
• User perspective– Isolation from environment
• Identical environment on multiple sites• Identical environment on local machine
03-09-2009 BIG Grid - Virtualization working group 4
Different VM classes• Class 1: Site generated Virtual Machines
– No additional trust issues– Benefits for system administration
• Class 2: Certified Virtual Machines– Inspection and certification to establish trust– Requirements for monitoring / integration
• Class 3: User generated Virtual Machines– No trust relation– Requires appropriate security measures
03-09-2009 BIG Grid - Virtualization working group 5
Resource management
Site infrastructure
Typical use case Class 1 VM
Torque/PBS
Box 2“8 Virtual SL4 WNs”
Box 3“8 Virtual SL5 WNs”
Virtual Machine Manager
Job queue
VMqueue
Box 1“Normal WN”
03-09-2009 BIG Grid - Virtualization working group 6
Typical use case Class 2 VM
Analysis on Virtual Machines• Run minimal analysis on desktop/laptop
– Access to grid services• Run full analysis on the grid
– Identical environment– Identical access to grid services
• No interest to become system administrator– Standard experiment software is sufficient
03-09-2009 BIG Grid - Virtualization working group 7
Typical use case Class 3 VM
Identification and classification of GPCRs• Requires very specific software set
– Blast 2.2.16– HMMER 2.3.2– BioPython1.50
• Even non-x86 (binary) applications!• Specific software for this user• No common experiment software
03-09-2009 BIG Grid - Virtualization working group 8
Project status• Working group: virtualization of worker nodes
https://wiki.nbic.nl/index.php/BigGrid_virtualisatie• Kick-off meeting July 6th 2009
– System administrators, User support, management• Phase 1 (3 months)
– Collect site and user requirements– Identify other ongoing efforts in Europe– First design
• Phase 2 (3 months)– Design and implement proof of concept
03-09-2009 BIG Grid - Virtualization working group 9
Active working group topics
• Policies/Security issues for Class 2/3 VMs• Technology study
– Managing Virtual Machines– Distributing VM images– Interfacing the VM infrastructure with ‘the grid’
• Identify missing functionality and alternatives– Accounting and fare share, image management,
authentication/authorization, etc.
03-09-2009 BIG Grid - Virtualization working group 10
The Amazon identity crisis
• The three most confronting questions:1. What is the difference between a job and a VM?
2. Why can I do it at Amazon, but not at the grid?
3. What is the added value of grids over clouds?
“We don’t want to compete with Amazon!”
03-09-2009 BIG Grid - Virtualization working group 11
Policy and security issues
E-science services and functionality• Data integrity, confidentiality and privacy• Non-repudiation of user actions
System administrator point of view• Trust user intentions, not their implementations• Incident response more costly than certification• Forensics is time consuming
03-09-2009 BIG Grid - Virtualization working group 12
Compromised user space is often already enough trouble
Security 101 = Attack surface
03-09-2009 BIG Grid - Virtualization working group 13
Available policies
• Grid Security Policy, version 5.7a• VO Portal Policy, version 1.0 (draft)• Big Grid Security Policy, version 2009-025
– Grid Acceptable Use Policy, version 3.1– Grid Site Operations Policy, version 1.4a– LCG/EGEE Incident Handling and Response Guide,
version 2.1– Grid Security Traceability and Logging Policy,
version 2.0• VO-Box Security Recommendations and
Questionnaire, version 0.6 (draft, not ratified)
03-09-2009 BIG Grid - Virtualization working group 14
Relevant policy statements
• Network security is covered by site local security policies and practices
• A VO Box is part of the trusted network fabric. Privileged access is limited to resource administrators
• Software deployed in the grid must include sufficient and relevant site central logging.
03-09-2009 BIG Grid - Virtualization working group 15
First compromise• Certified package repository
– Base templates– Certified packages
• Separate user disk– User specific stuff– Permanent storage
• At run time– No privileged access– Comparable to VO box
03-09-2009 BIG Grid - Virtualization working group 16
Licenses?
Second compromise• Make separate grid DMZ for Class 3 VMs• Comparable to “Guest networks”
– Only outbound connectivity• Detection of compromised guests
– Extended security monitoring• Packet inspection, netflows (SNORT, nfsen)• Honeypots, etc.
• Simple policy: one warning, you’re out.• Needs approval (network policy) from
OST (Operations Steering Team)03-09-2009 BIG Grid - Virtualization working group 17
Resource management
Site
Managing VMs
Torque/PBS
Box 2“8 Virtual WNs”
Box 3“8 Class 2/3 VMs”
OpenNebula
Job queue
VMqueue
Box 1“Normal WN”
Haizea
03-09-2009 BIG Grid - Virtualization working group 19
Class 2/3
upload solution
iSCSI/LVM
Distributing VM images
Box 3“8 Class 2/3 VMs”
Box 1“Normal WN”
Box 2“8 Virtual WNs”
Repository (SAN)
ImageImageImageImageImage
03-09-2009 BIG Grid - Virtualization working group 20
Cached copy-on-write
03-09-2009 BIG Grid - Virtualization working group 21
Box 1
Repository
Cache
ImageCOW
COWVM
VM
Box 2
Cache
Image
COW
COW
VM
VM
Image
Interfacing VMs with ‘the grid’
Resource management
Torque/PBS OpenNebula
Class 2/3
upload solution
Repository (SAN)
ImageImageImageImageImage
Class 2 Class 3
discussion
Grid middleware• globus-job-run• globus-gatekeeper• globus-job-
manager• contact-string
• jm-pbs-long• jm-opennebula
• qsub / opennebula
Nim
bus/OC
CI
03-09-2009 BIG Grid - Virtualization working group 22
VM contact-string• User management mapping
– Mapping to OpenNebula users• Authentication / Authorization
– Access to different VM images• Grid middleware components involved:
– Cream-CE, BLAHp, glexec– Execution Environment Service
https://edms.cern.ch/document/1018216/1
– Authorization Service Designhttps://edms.cern.ch/document/944192/1
03-09-2009 BIG Grid - Virtualization working group 23
Coffee table discussion
Parameter passing issue
Monitoring/Performance testing
Show Hosts : yes no | Experimental network_report last hour sorted descending | Columns 4 Size medium
(Nodes colored by 1-minute load) | Legend
Ganglia Web Frontend version 3.1.1 Check for Updates.
tropeRretsulClatnemirepxE::ailgnaG9002-9-3
http://ploeg.nikhef.nl/ganglia/?m=network_report&r=ho… 2/3
03-09-2009 BIG Grid - Virtualization working group 24
Performance• Small cluster
– 4 dual CPU quad core machines– Image server with 2 TB storage
• Integration with experimental testbed– Existing Cream-CE / Torque
• Testing– Network I/O, is NAT feasible?– File I/O, what is the COW overhead?– Realistic jobs
03-09-2009 BIG Grid - Virtualization working group 25
Other challenges• Accounting, scheduling based on Fair Share • Scalability!• Rapidly changing landscape
– New projects every week– New versions every month
• So many alternatives– VMWare, SGE, Eucalyptus, Enomaly– iSCSI, NFS, GFS, Hadoop– Monitoring and security tools
03-09-2009 BIG Grid - Virtualization working group 26
Conclusions
• Maintainability: no home grown scripting– Each solution should be part of a product– Validation procedure with each upgrade
• Deployment– Gradually move VM functionality in production
1. Introduce VM worker nodes
2. Virtual machine endpoint in grid middleware
3. Test with a few specific Class 2/3 VMs
4. Scaling and performance tuning
03-09-2009 BIG Grid - Virtualization working group 27