distributed systems - jspring.cs.herts.ac.ukjspring.cs.herts.ac.uk/distributed systems 1617.pdf ·...

Lecture - Definition and Motivation

Distributed Systems

Joseph Spring

School of Computer Science

Distributed Systems and Security


Introduction

• Distributed Systems

– Definitions

– Purpose

– Characteristics

– Challenges

– Design Requirements

• Middleware, Transparency and Heterogeneity


What is a Classical Distributed System?

• Lamport:

– A system is distributed if the message transmission delay is not negligible compared to the time between events in a single process

– It can range from:

• a single computer in which the central control unit, the memory units and the input-output channels are separate processes to

• A spatially separated collection of nodes each communicating within a message passing environment

• Coulouris, Dollimore Kindberg & Blair:

– ‘A distributed system is one in which components located at networked computers communicate and coordinate their actions only by passing messages’



• Tanenbaum & van Steen– ‘A distributed system is a collection of independent

computers that appears to its users as a single coherent system’

• Silberschatz, Galvin & Gagne– ‘A distributed system is a collection of loosely coupled

processors interconnected by a communication network’

QuestionWhat do we mean by loosely (or tightly) coupled processors?



Tanenbaum & van Steen

• Distributed Operating System– ‘A tightly coupled operating system is often referred to as

a distributed operating system

– Used for managing multiprocessors and homogeneous multi-computers

– Like traditional uniprocessor operating systems, main goal of a DOS is to hide the intricacies of managing the underlying h/w such that it can be shared by multiple processes’



Tanenbaum & van Steen

• Network Operating System– ‘A loosely coupled operating system used for

heterogeneous multicomputer systems (LAN and WAN)

– Although managing the underlying hardware is important

– for NOS distinction from traditional operating systems comes from the fact that local services are made available to remote clients

– In order to be classified as a distributed system, enhancements to the services of NOS are required such that better support for distribution transparency id required


Distributed Systems and Networks?

• Jeff Darcy– ‘A network is described as a system that creates a shared

space whilst a distributed system is one in which a shared purpose is created

Jeff Darcy ‘What is the difference between a network and a distributed system?’ www.quora.com, 2014

http://www.quora.com/


What is a Quantum Distributed System?

• Harry Buhrman and Hein Rӧhrig:

– Under the heading of Applications in Distributed

Computing three models of quantum communication are

presented:

• Communication via qubits

– BB84, B92

• Parties share EPR pairs but communication is via a classical bit

channel

– Teleportation

• Parties share EPR pairs and communicate via qubits

– Entanglement Swapping



• Rodney van Meter:

– Quantum Communication is ‘the exchange of quantum

states over a distance, generally requiring the support of

substantial classical communication’

– Quantum networks may be described as operating in at

least three modes

• The teleportation of (quantum) states

• The teleportation of (quantum) gates

• The creation of distributed quantum states



• Rodney van Meter:

– To make use of proposed h/w platforms (ion traps, quantum dots, NV

diamond) which offer ‘good optical connections … monolithic

computation’ needs to be split into ‘pieces for distributed

computation’

– Three categories of distributed quantum application:

• Distributed numeric computation (in which knowledge of input

data, algorithms used and output data are unknown by server)

• Cryptographic functions (include secret key generation, Byzantine

agreement and secret sharing)

• Sensor or cybernetic services (high precision interferometry, clock

synchronisation)


Why Build Distributed Systems?

Silberschatz, Galvin and Gagne

• Four reasons:

– Resource Sharing

– Computational Speedup

– Reliability

– Communication


Resource Sharing

• Routine Sharing

– HW Resources

• Printers, disks,

• Why? To reduce costs!

– Data Resources

• Files, shared database, shared web pages

– Resources with Specific Functionality

• Search engines, currency converter


Computation Speedup

• Computation Speedup

– Partition a computation into sub-computations

that can run concurrently

– distribute sub-computations amongst the sites

available to run concurrently (hence speedup)

• Load Sharing

– Sites with heavy workload distribute jobs to other

lightly loaded sites


Reliability

• If one site fails in a distributed system, then

remaining sites may continue operation

– Case 1 Multiple Large Autonomous Installations (General

Purpose Computers)

• Failure of one doesn’t affect others

– Case 2 System composed of small machines each

responsible for a crucial system function

• Terminal character I/O, file system, …

• Single failure may halt complete system

• Sufficient redundancy in ‘H/W and S/W’ required


Reliability

• Failure of site must be detected by system

• Appropriate action taken to recover from failure

• Site should be removed from active involvement in

system until problem resolved

• Another site to be nominated to take over role of

initial site

• System must ensure transfer of function to new site

is achieved


Communication

• Users of communication networks exchange data

• Message Passing Systems

– Higher-level functionality found in standalone machines may be

expanded to entire distributed system

• File transfer

• Login

• Mail

• web-browsing

• RPC (remote procedure calls)

• …


Communication

– Advantages of a Distributed System

• Above functions may be carried out over large

distances

• Enhances opportunities for project collaboration over

large distances

– Enables

» Transfer of files

» Log in to each others systems to run programs

» Exchange of mail to coordinate work

» Minimises problems involved in collaborating over long

distances e.g. Tele-Working


Communication

– Advantages of a Distributed System

• Companies downsizing from mainframes to networks

of workstations or personal computers

• Increased functionality per unit cost

– Increased flexibility in locating resources & expanding

facilities

– Better user interfaces

– Easier maintenance


Examples

(CDKB pp 3 - 7)

• The Internet/www and Intranets

• Mobile and Ubiquitous Computing

(CDKB pp 721 – 746)

• Distributed Multimedia Systems

(A selection of research papers)

• Cloud Computing


Characteristics of Distributed Systems

The (CDKB) definition leads us to consider the

following consequences for Distributed Systems:

• Concurrency of Components

• No Global Clock

• Independent Failure of Components


Concurrency of Components

• For a network concurrent program execution is

normal:

– We each work on our own computers at the same time

sharing resources such as

• Web pages

• Files

• …

– CDKB consider various ways in which extra resources may

successfully be deployed to a system in order to increase

its capacity to handle shared resources


No Global Clock

• Clock synchronisation

– Cooperation between programs is achieved through

passing messages in order to coordinate their actions

– Close coordination is dependent upon a shared idea of

the time at which program actions occur

– There is a limit to the accuracy that computers in a

network can synchronise their clocks

• There is no single global notion of a correct time

• This is a direct consequence of the only communication being

message passing in a network


Independent Failure of Components

• All computer systems can fail

– It is the responsibility of system designers to plan for the

consequences of possible failures

– Distributed systems can fail in new ways

• Faults in the network isolate the computers that are connected to it,

despite the fact that they may still run

• Programs on such computers may not be able to tell that the network is

unusually slow or that it has failed

• The failure of a program in the system (a crash) may not be made known

to other components in the system with which it communicates

• Each component in a system can fail independently, leaving others still

running


Challenges / Goals

• When should we build a Distributed System?

– Tanenbaum & van Steen discuss ‘four important goals that

should be met (in order) to make building a distributed

system worth the effort’

– A distributed system should

• Easily connect users to resources

• Hide the fact that resources are distributed across a network

(Transparency)

• Be open

• Be scalable


Connection

• Collaboration and exchange of information

– best illustrated by the internet

– simple protocols for exchanging

• Files – FTP

• Mail – SMTP, X.400, MIME, POP

• Documents – HTTP, HTTPS

• Digital data (text, animation) – BMP, GIF, JPEG

• Audio and Video – WAV, JPEG, MPEG, MP-3

• …

Compression Formats


Connection

• Groupware

– Software for collaborative editing,

teleconferencing, …

• Internet connectivity

– led to virtual organisations

• geographically dispersed groups

• working together through groupware

– led to e-commerce

• buying and selling achieved without going to a store


Connection

• Security issues prevail as connectivity and sharing increase– Confidentiality

• Message cannot be read by anyone else

– Authentication

• Bob knows only Alice could have sent the message

– Integrity

• The message has not been tampered with whilst in transit

– Non-repudiation

• Alice cannot deny that she sent the message

• Scenario– Alice wants to purchase some item over the internet from Bob. She

sends her order to Bob which contains credit card and payment details


Challenges / … / Approach to Design

Coulouris, Dollimore , Kindberg & Blair discuss the above goals

(and others) in terms of challenges:

• Heterogeneity– Middleware

– Heterogeneity and Mobile Code

• Openness

• Security

• Scalability

• Failure Handling

• Concurrency

• Transparency


Heterogeneity

• Variety and difference in the collection of computers and networks employed in for example the internet

• Applies to each of the following:– Networks, H/W, O/S’s, Programming Languages, Implementations by

different developers

• Networks– Differences are masked on the internet since all computers attached

to them use the internet protocols to communicate with each other

• H/W– Data types may be represented in different ways by different H/W

– Little endian v big endian byte ordering of integers

– These must be resolved in order to communicate using different H/W


Heterogeneity

• O/S’s

– All have to supply implementation of internet protocols

– Do not necessarily provide the same Application Programming

Interface to these protocols

• Calls for exchanging messages in UNIX different from those in Windows

based o/s’s

• Programming Languages

– Different programming languages use different representations for

characters and data structures

• Arrays and records

• These need to be addressed if programs written in different languages

are to communicate


Heterogeneity

• Implementations by different developers

– Programs written by different developers cannot communicate unless they use common standards

– Standards need to be agreed and adopted, as have the internet protocols


Quantum Heterogeneity

• Quantum Networks– Fibre Optic Networks

• Referred to as dark fibre

• Networks use existing unused optic fibre

– Free Space Networks• Wireless communication

• Receiver must be in ‘line of sight’ from sender

• Experiments carried out in Canary Islands between La Palma and Tenerife

– Cavity – QED Networks


Quantum Heterogeneity

• Quantum Networks– Fibre Optic Networks, Free Space Networks, Cavity – QED Networks– DARPA QKD Network (2001), – SECOQC QKD Network (Vienna) Secure Communication based on

Quantum Cryptography, (2003)– Tokyo QKD Network, (2009)– Hierarchical Network, Wuho, China, (2009)– Geneva Area Network (SwissQuantum)

• Quantum Hardware• Quantum Operating Systems

– Cambridge Quantum Computing (CQCL) new o/s t|ket>

• Quantum Programming Languages– Quantum Imperative Paradigm

• Quantum Pseudocode, QCL - Quantum Computing Language, Q Language, qGCL, LanQ

– Quantum Functional Paradigm• QFC, QPL, QML, Quipper


Classical Heterogeneity - Middleware

• The software layer that provides a programming abstraction as well as masking the heterogeneity of the underlying networks, H/W, O/S’s and Programming Languages

• Examples– CORBA

– Java RMI• Supports single programming language

• Most middleware implemented over the internet protocols which in turn mask the difference of the underlying networks

• All middleware deals with the differences in O/S’s and H/W

• Provides a uniform computational model for use by programmers of servers and distributed applications

Middleware / Design Issues IPC 35

Introduction - Middleware

• Middleware sits between ‘Applications, and services’ and the operating system:

Operating Systems

Middleware

Applications, Services

Computer and Network Hardware

Platform


Middleware

‘…a layer of software whose purpose is to maskheterogeneity and to provide a convenientprogramming model to application programmers• Represented by processes or objects

interacting in a set of computers to implementcommunication and resource sharing support fordistributed applications

• Concerned with providing useful buildingblocks for the construction of softwarecomponents that can work together

Middleware - Limitations

• Many distributed app’s rely entirely on the services provided by available middleware to support communication and data sharing needs

• Much has been achieved through the development of middleware however some aspects of the dependability of systems require support at the application level

• A similar point is made by Saltzer, Reed and Clark, regarding the design of distributed systems, the end-to-end argument


• ‘some communication-related functions can be completely and reliably implemented only with the knowledge and help of the application standing at the end points of the communication system. Therefore, providing that function as a feature of the communication system itself is not always sensible. (An incomplete version of the function provided by the communication system may sometimes be useful as a performance enhancement)’ Saltzer, Reed and Clark 1984


• The argument is counter to the view that all communication activities can be abstracted away from the programming of applications by the introduction of appropriate middleware layers

• For SR&C correct behaviour in distributed programs depend upon checks, error correction mechanisms and security measures at many levels

– Checks within communication system only will be onlypartially correct

– Same work is therefore likely to be replicated in application programs leading to wasteful programming, unnecessary complexity and computational redundancy


Middleware

See following references:

• Saltzer,J.H., Reed D.P. and Clarke, D. 1984 End-to-End Arguments in System Design, ACM Transactions on Computer Systems, Vol.2, No.4, pp. 277 –288

• http://www.reed.com


Middleware

• Middleware consists of two layers as shown:

Request – Reply Protocol

Marshalling and Data Representation

RMI and RPC

Applications, Services

UDP and TCP / Operating System

Middleware


Middleware

• RMI and RPC Layer– Concerned with integrating communication into a

programming paradigm by providing RMI or RPC

– Remote Method Invocation• Allows an object to invoke a method in an object in a

remote process

• Examples of systems for RMI are CORBA and Java RMI

– Remote Procedure Call• Allows a client to invoke a procedure in a remote

server


Middleware

• Request-Reply Protocol, Marshalling and the External Data Representation Layer– Concerned with suitable protocols that support

client-server (and group communication)– Concerned with the translation of objects and data

structures into a form suitable for sending in messages over the network• Takes into account different computers may use

different representations for simple data items• CDK consider a suitable representation for object

references in a distributed system


Heterogeneity and Mobile Code

• Mobile Code– Code that can be sent from one computer to another and run at the

destination

– For example Java applets

– Machine code suitable for running on one computer is not suitable for running on another• PC users v Macintosh

• Application sent as email attachment

– Virtual Machine • A way of making code executable on any H/W

• Compiler for a particular language generates code for a virtual machine, which needs to be implemented once for each type of H/W to enable Java programs to run

• Java solution not generally applicable to programs written in other languages


Openness

• An Open Distributed System

– A system that offers services according to standard rules describing

the syntax and semantics of the service

– That characteristic that determines whether the system can be

extended and re-implemented in various ways

– Determined primarily by degree to which new resource sharing

services can be added and made available for use by a variety of

client programs

– Specification and documentation of key software interfaces

published, made available to software developers

– Akin to standardisation of interfaces, often bypassses official

standardisation procedures which can be slow and cumbersome


Openness

• Open Distributed Systems

– Publication of interfaces seen as beginning for adding and extending

services in a distributed system

– Challenge is to tackle complexity of distributed systems composed of

many components designed by different people

– Based on the provision of a uniform communications mechanism and

published interfaces for access to shared resources

– Can be constructed from heterogeneous H/W and S/W, possibly from

different vendors

– Conformance of each component to the published standard must be

carefully tested and verified in order for system to work


Openness

• An Open Distributed System

– RFC’s (Requests for Comments)

• Series of documents introduced by designers of internet protocols

• Each RFC has a specific number

• Specifications of Internet protocols published in RFC’s during 80’s

• Forms the basis of technical documentation of internet

• Copies obtainable from www.ietf.org

• Publication of internet protocols has enabled variety of internet

applications and applications to be built

– E.g. the web, a recent addition to the services in the internet

• Not the only means of publication

– E.g. CORBA published through a series of technical documents

– Copies obtainable from www.omg.org

http://www.ietf.org/

http://www.omg.org/


Openness

• Open Distributed System

– May be extended

• at the H/W level by adding more computers to the network

• At the S/W level with introduction of new services and re-

implementation of old ones, enabling application services to share

resources

– Benefit in their independence from individual vendors


Security

Continuing from our earlier discussion, the following

two challenges have not yet been resolved:

• Denial of Service Attacks– The bombardment of a service by large number of requests that

choke availability, ensuring that serious users cannot use service

– Countermeasures based on improvements to management of networks are underdevelopment

• Security of Mobile Code– Handle with care

– Executable programs as email attachments are unpredictable

– Virus deployment


Scalability

• A system is said to be scalable if it remains effective after a significant increase in the number of resources and users

• Example

– Internet is an example of a distributed system in which number of computers and services has increased dramatically

– See info.isoc.org for information on increasing number of computers and web servers


Scalability

• Challenges to the Design of Scalable Distributed Systems:

– Controlling cost of physical resources

• As numbers of users increases so too should the availability of servers supplying service in order to avoid bottlenecks

• For a system with n users to be scalable the number of physical resources should be O(n) i.e. proportional to n

• E.g. if one server can support 20 users then 2 servers are required for 40 users


Scalability

• Challenges to Scalable Distributed Systems:

– Controlling performance loss• Consider case of managing set of data proportional to the number

of users or resources in the system

• E.g. table with correspondence between domain names of computers (e.g. www.amazon.com) and their internet addresses (e.g. 128.117.38.5) held by the Domain Name System

• Hierarchic structures fair better than linear ones however even with these, an increase in size will result in a loss of performance

• Time taken to access hierarchic structure O(logn) where n is size of set of data

• For system to be scalable the maximum performance loss should be no worse than O(logn)


Scalability

• Challenges to the Design of Scalable Distributed Systems:

– Preventing software resources running out

• E.g. for lack of scalability offered by 32 bit numbers used as internet addresses since late 70’s

• Supply of available addresses will run out in early 2000’s

• New version protocol will use 128 bit internet addresses


Scalability

• Challenges to Scalable Distributed Systems:– Avoiding Performance Bottlenecks

• In general algorithms should be decentralised to avoid performance bottlenecks

• E.g. Predecessor of Domain Name System kept name table in single master file which could be downloaded to any computer that needed it– Ok for small number of computers on internet

– Performance and administrative bottleneck with increasing size of users

– Problem resolved with DNS through partitioning of name table between servers throughout the internet and administered locally

• Some shared resources (e.g. web pages) accessed frequently causing decline in performance– Caching and replication may be used to improve performance


Failure Handling

• Detection– Some failures detectable, e.g. using check sums to detect

corrupted data in messages or files

– Detection of remote crashed server often difficult / impossible

– Challenge, to manage in presence of failures that cannot be detected but are suspected

• Masking– Some failures can be hidden or made less severe

• Messages can be retransmitted if they fail to arrive

• File data can be written to a pair of disks so that if one is corrupted, the other may be ok


Failure Handling

• Tolerating Failures

– Failures occur frequently on the internet

– Clients designed to tolerate failures, (and users),• E.g. web browser doesn’t try to invoke a web page forever, but

notifies the user of a problem so s/he can try later

• Recovery

– Involves the design of S/W to ‘roll back’ after a server has crashed, so that the state of permanent data can be recovered

– Permanent data (files and other material filed in permanent storage) can be inconsistent


Failure Handling

• Redundancy– Services can be made to tolerate failures through the use of

redundant components

• For example– There should always be two different routes between routers in the

internet

– In the Domain Name System every name table is replicated in at least two different servers

– A database can be replicated in several servers

• To ensure data accessible after the failure of any server

• Servers designed to detect faults in their peers

• When a fault is detected in a server clients automatically redirected to remaining servers


Concurrency

• Services and applications provide resources that can be shared by clients in a distributed system

• Possible that several clients will try to access a shared resource at the same time

• Services and applications generally allow multiple client requests to proceed concurrently

• Operations are synchronised in such a way as to preserve the consistency of data, using standard O/S techniques such as semaphores


Transparency

• ‘… the concealment from the user and the

application programmer of the separation of

components in a distributed system, so that the

system is perceived as a whole rather than as a

collection of independent components.’

• ANSA Reference Manual [ANSA 1989] and the

International Standards Organisation – Reference

Model for Open Distributed Processing (RM-ODP)

[ISO 1992] identify 8 forms of transparency:


Eight Forms of Transparency

• Access Transparency

• Location Transparency

• Concurrency Transparency

• Replication Transparency

• Failure Transparency

• Mobility Transparency

• Performance Transparency

• Scaling Transparency

Network Transparency



• Access Transparency

– Enables local and remote resources to be accessed using

identical operations

• Location Transparency

– The accessing of resources without knowledge of their

location

• Concurrency Transparency

– The concurrent use of shared resources by several

processes without interference between the processes



• Access Transparency Example

– To send an integer from an Intel based workstation to a

Sun SPARC machine means that we take into account that

Intel orders its bytes in little endian format (high order bit

is transmitted first) and that the SPARC processor uses big

endian format ( low order bit is transmitted first)

– Computer systems may run different O/S each having

their own name filing conventions. Differences in naming

conventions and how files can be manipulated should be

hidden from the user



• Location Transparency Example

– Naming plays an important role in achieving

location transparency

– For example:

• http://www.example.com/index.html

• The address gives no indication of location of main

server

– Unlike URL’s which end with .uk, .ie, …

http://www.example.com/index.html



• Replication Transparency

– Multiple instances of resources used to increase reliability and performance without knowledge of replications by users or application programmers

• Failure Transparency

– The concealment of faults, allowing users and application programs to complete tasks despite the failure of H/W and/or S/W components

• Mobility Transparency

– Allows movement of resources and clients within a system without affecting the operation of users or programs



• Replication Transparency Note:

– Plays an important role in Distributed Systems

– Resources may be replicated in order to• Increase availability

• Improve performance

by placing copies close to demand

– Hides fact that several copies of a resource exist

– To support replication transparency • all replicas must have same name

• System supporting replication transparency generally supports location transparency too, otherwise impossible to refer to replicas at different locations



• Failure Transparency Note:

– Masking failures

• one of hardest issues in distributed systems

– Main difficulty

• Inability to distinguish between a dead resource and a painfully

slow one

– Example

• Contacting a busy web server

• Browser will eventually time out & report web page unavailable

• User cannot conclude server is really down



• Mobility Transparency Example 1

– The reference to ‘index.html’ in the address


gives no information as to how long it has been at

this location or whether it has recently moved

This is an example of mobility / migration

transparency




• Mobility Transparency Example 2

– Where resources can be relocated whilst they are

being accessed without the user or application

‘noticing’

– Mobile users continuing to use wireless laptop

whilst moving without being disconnected

An example of mobility / relocation transparency



• Performance Transparency

– Allowing the system to be reconfigured to improve performance as loads vary

• Scaling Transparency

– Allowing the system and applications to expand in scale without change to the system structure or the application algorithms

Access and Location Transparency are often referred to collectively as Network Transparency


Transparency and Design Requirements

• Transparency

Tanenbaum and van Steen

– A distributed system that is able to present itself to users and applications as if it were only a single computer system is said to be transparent

Question - Is transparency always required?

• Design Requirements

Question - What design requirements address problems

regarding synchronisation issues?


References

• G. Coulouris, J. Dollimore, T. Kindberg & G. Blair: Distributed Systems Concepts and Design, 5th Ed. Addison Wesley, 2011

• A. Tanenbaum & M. van Steen: Distributed Systems, 2nd Ed. Maarten van Steen, 2016

• Buchanan: Distributed Systems and Networks, McGraw Hill, 2000

• Silberschatz, Galvin & Gagne: Operating System Concepts, 9th Ed. Wiley, 2013

• Stallings: Cryptography and Network Security, Prentice Hall, 2014

• Harry Buhrman and Hein Rӧhrig, Distributed Quantum Computing, pp1-20, LNCS 2747, 2003

• Rodney van Meter, Quantum Networking, John Wiley & Sons Inc., 2014