distributed (operating) systemslaurel.datsi.fi.upm.es/~jmpena/insa/ds-1-introduction.pdfdistributed...

Distributed (Operating) Systems

IntroductionIntroduction

Fernando PérezJosé María PeñaMaría S. Pérez

Distributed Operating Systems1

ScheduleSessions1. Introduction: Distributed systems

(Hardware/Software issues)2. Process management in clusters: Load

balancing and job scheduling 3. Distributed communications 4. Distributed servicesScenarios• High-performance solutions for scientific

applications (process management)• Distributed systems for transactional

services

Mon Tue8:00

9:00

10:0011:00

1-Intro 4-serv

12:0013:00

LUNCH

14:0015:00

2-proc Scenario 2

16:00

17:00Scenario 1

3-comm



Bibliography

• Distributed Systems: Concepts and DesignG. Coulouris, J. Dollimore, T. Kindberg; Addison-Wesley, 2001

• Distributed Systems: Principles and ParadigmsA. S. Tanenbaum, M. Van Steen; Prentice-Hall, 2007

• Distributed Operating Systems: Concepts & PracticeD. L. Galli; Prentice-Hall, 2000

• Distributed Operating Systems & AlgorithmsR. Chow, T. Johnson; Addison-Wesley, 1997

• Distributed Computing: Principles and ApplicationsM.L. Liu; Addison-Wesley, 2004

Distributed (Operating) Systems

Introduction andConcepts

Introduction andConcepts



Distributed System (DS)• Hardware: Network-connected processor without shared

physical memory:– Loosely-coupled system– Non-common clock– Processor-dependent I/O systems– Independent failures of system components– Heterogeneous system

• Goal of this seminar: Distributed System Software– Distributed Operating Systems (classical view)– Software interface that hide distributed system complexity:

• Single System Image



Advantages and Drawbacks• Advantages:

– Cost/performance ratio– Parallel processing: high performance– Fault tolerance: high availability– Scalable, open and heterogeneous– Most appropriate for originally distributed applications– E.g., geographically distributed enterprise

• Drawbacks:– More complex software development– Networks connection problems: latency, bandwidth and availability– Security



New Paradigms for DS• Cluster Computing:

– Dedicated systems:• High performance.• High availability.

– Homogeneous system:• Nodes.• LAN (generalist or specific).

– Open issues: Coupling degree, distributed services. • Gird Computing:

– Resource sharing and idle processor usage.– Restricted to some specific tasks.– Different scopes:

• Inter-departmental grids.• Inter-organization grids.

– Open issues: Coordination, security and dynamic changes.


Operating System Support7

Operating System Support1. OS for Distributed Systems:

• Requirements• Characteristics

2. Distributed Systems3. Parallel/Distributed OS:

• Operating Systems Parallelisation• Distributed System Services• Microkernels



Distributed ArchitecturesA distributed system is a collection of independent computers presented to the user as a single computer.

Distributed Computer Architectures:– Flynn’72: SISD, SIMD, MISD, MIMD– Johnson’88: UMA, NUMA, NORMA



Distributed System Application• Internet Services: e-mail, news, web, ...• Corporate networks or intranets.• Parallel processing:

– Massive processing (+efficiency).– Distributed topology (distributed-nature problems)

• Distributed massive data management.• High performance multimedia.• Industrial and control systems.• Real-time systems.

<and many others...>



Distributed System ProfileDistributed systems have:

1. No common clock: Message and co-ordination aspects.2. Global concurrency: Real parallel execution.3. Independent failures: Partial failures.

Distributed system usage:1. Collaborative processing: combined features and services.2. Parallel processing: massive or high-performance calculation.


Operating System Support

Parallel systems – Performance– Scalability– Reliability– Transparency– Security

11

System Requirements

Collaborative systems– Openness– Scalability– Reliability– Transparency– Security

Common characteristics but different hardware platforms and applications.All of them DISTRIBUTED



Operating System Distribution• Operating systems for multiprocessors with shared memory

(SMP):– Software tightly coupled – Hardware tightly coupled

• Distributed operating systems (DOS):– Software tightly coupled – Hardware loosely coupled

• Network operating system:– Software loosely coupled – Hardware loosely coupled



Operating Systems for SMPsArchitectures with multiple processors (2 to 8) with uniform access shared memory (SMP: Symmetric Multiprocessors)

Characteristics:– “Small” variations of the traditional OS versions.– There is only one copy of the OS.– Concurrency with real parallelism (≠ shared time).– Commercial versions (Linux, WinNT, Solaris, AIX, ...).– Different problems: kernel code running on multiple processors

(concurrent system calls), synchronisation mechanisms (spin-locks), optimisation and scheduling (processor affinity), ...



Distributed Operating Systems (DOS)A distributed operating system is a group of processor interconnected by a communication network that hides its complexity presenting to the user a “virtual uniprocessor”.

Characteristics:– It runs on a distributed systems making them appear as a

centralised system. – Transparency: Must hide complex factor of the distribution.– It is easier to say than to do.– This goal is reached partially by the experimental systems.– Failures make the users comply.



Distributed Operating Systems (DOS)Problems:

– Each node has a copy of the OS: Which tasks are performed locally and which globally?

– How mutual exclusion is achieved without shared memory?– How deadlocks are detected without global states?– Process scheduling: Each operating system copy has an own task

queue (process migration). – How a single directory tree is defined?– Problems due to no-common clock, partial failures and heterogeneity.

Main result:– New concepts have been developed and they are useful for other

domains.



DOS Evolution• First network operating systems:

– New network services in a conventional OS– E.g.: UNIX 4BSD (≈1980)

• New network functionalities:– Sun’s ONC (≈1985): includes NFS, RPC, NIS

• First DOS:– New OS based on conventional (monolithic) versions.– E.g.: Sprite, University of Berkeley (≈1988)

• DOS based on μ-kernel. E.g.:– Mach, CMU (≈1986)– Amoeba, designed by Tanenbaum (≈1984)– Chorus, INRIA, France (≈1988)



Network Operating SystemsNetwork of computers loosely coupled that share resources with no external control on the hardware/software of each node.

Characteristics:– No virtual uniprocessor vision is presented (independent nodes).– Each node runs a copy of the OS (different). – Conventional OS+ network utilities.– Communication protocols for resource sharing and high-level service

access. – From rcp/rlogin to Sun’s Open Network Computing (ONC).



Cooperative SystemsHigh-level services-oriented software systems that requires communication mechanisms to build upper level services.

Characteristics:– A grade of transparency is provided but the single-system vision is

not presented. Autonomous independent systems.– They are founded on middlewares (CORBA, DCE, COM+, ...)– These systems are designed as a combination of multiple services

offered by different network elements.



Middleware

Middleware:– Software layer over the operating system that provides standard

distributed services.– Open systems independent of the vendor.– Hardware and OS independent.

Examples:– DCE (Open Group).– CORBA (OMG).– ...

Hardware

OS

Hardware

OS

Hardware

OS

Middleware



Single System Image (SSI)The illusion, created by hardware/software, that presents a collection of resources as one.

– Hardware SSI: DEC Memory Channel or SMPs– Operating System: DOS or Gluing layer – Application and Services: Middlewares (many levels).

Every SSI has a boundary.



Why SSI is useful?• It is easy to program/use:

– Traditional programming, known interfaces.– Low-level issues hidden.

• Allows centralized and distributed management depending on task requirement.

• (Potentially) provides:– Fault tolerance.– Scalability.– Modular improvement.



Operating System Layers

A simplified vision of an Operating System has the followinglayers:

• Hardware.• Kernel.• System services.• Application programs.• Users.

Hardware

Kernel

Services

Applications

Users



Kernel Responsabilities

KernelServices

μ−KernelServices

Computer

Computer

μ−Kernel μ−Kernel μ−Kernel

Services

Monolithic Kernels:Many OS functionalities inside the kernelscheduler, memory manager, drivers, file systems...

μ−Kernels:Many OS tasks are performed outside the kernel. Remaining: (i) process communication, (ii) memory management, (iii) low-level management and scheduling y (iv) low-level i/o

Distributed Services:Distributed system structure. Depending on

the level: Distributed operating systems Network operating systems or (Cooperative).



Operating System on Distributed Systems

DistributedClustersSMPsMPPs

Size 100s – 1000s 10s 100s or less 10s – 1000s

OS N x kernels Single OS kernel N x OS platforms N x OS platforms

OS type Specific purpose Standard OS plus tools (not always)

Standard OS andspecial tools

Communic. Message / DSM Shared Memory Message passing (e.g.: MPI)

Message passing or middleware

Scheduling Single queue Single queue Multiple queues coordinated

Independent queues

Special variantsof standard OSs

Single System Image (SSI)



Tools for Distributed/Cluster Systems• Operating system:

– Modular/Layered Monolithic– Based on μ-Kernels

• Runtime systems:– Parallel file systems or I/O libaries– Distributed shared memory software

• Resource management:– Process scheduling tools– Load balancing

• Applications:– Management and administration tools.– Processing tasks and jobs


Distributed Operating Systems

Hardware andSoftware Overview

Hardware andSoftware Overview



Concept of Cluster• Alternative to traditional supercomputing facilities.• Instead of traditional systems:

– Specific hardware.– High-cost.– Slow hardware development.– Painful software development.

• the use of general-purpose systems provides:– Commodity hardware (Commercial-off-the-self: COTS).– Moderate-cost.– Fast hardware development.– Even more painful software development.



Concept of ClusterCluster: Hardware system based on commodity hardware

connected by a dedicated (high-performance) network.– Nodes: PCs or workstations (SMPs).– Network: From high-speed networks to specific hardware.

Mysterious acronyms:– PoPCs: Pile of PCs– COWs: Clusters of workstations– CLUMPS: Clusters of multiprocessors– NOWs: Networks of workstations– ....



Hardware Characteristics• Nodes:

– Processor: Intel Pentium, AMD Athlon, Compaq Alpha, IBM PowerPC, Sun SuperSparc (3-4...Ghz)

– Memory: SDRAM, DDR or similar (2-8 GB)– Storage: SCSI or RAID

• Network:– Key element. – It could cost 50+% of the system value– Cheap alternative: Ethernet (100-1000Mb/seg)



Cluster Networks (I)• General purpose network technologies:

– Improvement in network bandwidth.– Only reduced improvements in the latency Not well-suited

• Low-latency protocols:– Active Messages (Berkeley): “Zero-copy” synchronous model. GAM.– Fast Messages (Illinois): Reliable AM in order.– VMMC (Princeton): Distributed shared memory pages (DSM).– U-net (Cornell): Virtual interfaces for memory pages.– BIP (ENS Lyon): Low-latency basic interface.



Cluster Networks (II)• Cluster communication standards:

– VIA: Hardware interface (native/emulated) for communications. Mpasphysical memory regions and virtual network interfaces. MPI versions over VIA.

– InfiniBand: I/O hardware standard (2.5Gbps) using one-way connections. 6 Communication models. Using RDMA and IPv6.

• Network hardware:– Ethernet, FastEthernet, GigaEthernet: Cheap but limited. Collision

problems. VIA emulations.– Giganet (cLAN): Implementation over VIA (1.26Gbps)– Myrinet: Low-latency programmable networks. Cut-through routing

and failure detection. GM protocol.– Others: QsNet, ServerNet, SCI, ATM, FiberChannel, HIPPI, ATOLL,...



Technologies ComparativeGigabit

EthernetGiganet Myrinet QsNet SCI ServerNet2

MPI badwidth – stable(MB/sec)

35-50 105 140 208 80 65

MPI latency (μseg) 100-200 20-40 ~18 5 6 20.2

Maximum number ofnodes

1000’s 1000’s 1000’s 1000’s 1000’s 64k

VIA support Win/Linux Win/Linux Over GM NOne Software Hardware

MPI support type MPICH over MVIA or TCP

Thrirdparties

Thrirdparties

Quadrics orCompaq

Thrirdparties

Compaq orThrirdparties

© Amy Apon / Mark Baker 2000



Software Development (I)• Operating Systems:

– Linux: • Free, cheap, fast and fast-development.• e.g., Beowulf

– Solaris: • Good parallelism support and good network services.• e.g., Solaris MC

– AIX: • Powerful and well-optimized software development tools.• e.g., SP2

– Windows: • Why not?• e.g., Wolfpack



Software Development (II)• Middleware and SSI:

– SSI (Single System Image): The whole cluster is presented as a single monoprocessor.

– Layered development:• Hardware (Local).• Operating system (μkernel) or gluing level: GLUnix or MOSIX• Application, services and middleware: CODINE

– Common services (desirable):•Single access point.•Single file hierarchy.•Single management point.•Single network connection.•Single work-management service.

•Single user interface•Single I/O space•Single process space•Checkpointing.•Process migration



Software Development (III)• Programming tools:

– Thread support: Pthreads or OpenMP– Message passing in clusters:

• MPI: MPICH or LANMPI.• PVM: Worse performance but more features.

– DSM: Distributed shared memory:• Software: TreadMarks, Linda or Nanos• Hardware: DASH or Merlin

– Parallel debuggers – Instrumentation tools.



Software Development (IV)• Administration tools:

– Remote management: • Administrative commands: install software, copy files.• Process-level resource management.• User list and other system information: NIS.• e.g., SP2 tools, Cluster Command & Control (C3)

– Scheduling systems:• Work queues and workload management• Resource supervision.• e.g., CODINE, CONDORPBS (Portable Batch System)



Input/Output System• I/O Crisis:

– Exponential growth of CPUs power (Moore’s law).– I/O systems much smaller growth.– I/O phase is the actual bottleneck of high-performance systems.

• Solution based on I/O parallelism:– Parallel I/O systems: MPI I/O– Parallel filesystems: ParFiSys, GPFS– Intelligent I/O: Armada, Panda

distributed (operating) systemslaurel.datsi.fi.upm.es/~jmpena/insa/ds-1-introduction.pdfdistributed...

Documents