introduction to operating systems part itdde25/info/slides/tdde25_fo12_1sl.pdf(”distributed memory...

Copyright Notice: The slides partly use material from Silberschatz’s, Galvin’s and Gagne’s book (“Operating System

Concepts”, course book in TDDB68). No part of the lecture notes may be reproduced in any form, due to the copyrights

reserved by Wiley. These lecture notes should only be used for internal teaching purposes at the Linköping University.

Christoph Kessler, IDA,

Linköpings universitet.

Introduction to

Operating Systems

Part I

Christoph Kessler

IDA, Linköping University

TDDE25

2C. Kessler, IDA, Linköpings universitet.

Why should I be interested in Operating Systems?

IBM 360

DEC-20

Sun Workstations

IBM PC

Mainframe

Midframe

Linux

Windows

MS-DOS

MacOSX

Unix

OS/360

TOPS-10

MiniComputers

Workstations

?


A Whole New World!

Smart PhonesPortability

Connectivity

Networks

SensorsWearable Devices


A Whole New Future World!Smart Vehicles Smart Highways

Smart Cities Smart Homes / Smart El-Grids

The current 7-

Series BMW and

S-class Mercedes

boast about 100

processors a

piece.

A relatively low-

profile Volvo still

has 50 to 60 baby

processors on

board.


Increase in Hardware Complexity

Need for new types of

interfacing to hardware

• Multiprocessor servers

• Hardware multithreading

• Multicore CPUs

• Many-Core CPUs

• Accelerators e.g. GPGPU

• Hybrid computing

• Complex memory hierarchies

• Configurable hardware

• Clusters

• Cloud, Edge

• IoT sensors

• …

Nvidia

Tesla GPU

“Moore’s Law”(prediction 1965/75):

#Transistors / mm2

doubles every ~2 years.

Holds until ~2015 and,

with somewhat lower

exponential slope,

still today.

Source: Intel

NSC’s cluster

supercomputer

“Tetralith” (TDDC78)

Many-core

CPU


Increase in Software Complexity

Need for new types of OS Software ArchitecturesMacOS X 10.4:

86M LOC

(2005)

Linux

kernel

2015

Linux

kernel

2001


Agenda

Lecture I:

Computer Systems Overview

Building and Executing Programs

Operating System Basics

Lecture II:

Interrupts and I/O

System Calls

CPU Management

Memory Management

File System

Protection and Security






Computer Systems

Overview


Computer Systems and Environments

Stand-alone desktop computer

Client-server systems

Parallel systems

Clustered systems

Distributed systems

Real-time systems

Handheld systems


Stand-alone desktop computer

PC, workstation: Computer system

dedicated mainly to a single user.

I/O devices: keyboards, mice, screens, small printers.

Requirement: User convenience and responsiveness.

Can adopt OS technology developed for larger computer systems

May run several different types of operating systems

(Windows, MacOS, UNIX, Linux)


Client-Server Computing

Servers respond to requests by clients

Remote procedure call –also across machine boundaries via network

Client/Server are mainly software roles, but machines can be dedicated to one or few such roles, e.g.

Compute server: compute an action requested by the client

File server: Interface to file system (read, create, update, delete)


Parallel Systems (1)

Multiprocessor systems with more than one CPU in close communication.

Today the default, even for desktop machines (multi-core)

Tightly coupled system (aka. shared-memory system, multiprocessor)

processors share memory and a clock;

communication usually takes place through the shared memory.

Advantages of parallel systems:

Increased throughput

Economical

Scalability of performance (within certain limits)

Multiprocessor system vs multiple single-processor system (reduction of hardware such as disks, controllers etc)

Increased reliability

graceful degradation (fault tolerance, …)

fail-soft systems (replication, …)



Symmetric multiprocessing (SMP)

One OS controls all processors and the shared memory

Many processes can run at once without performance deterioration.

All modern operating systems support SMP

Asymmetric multiprocessing

Each processor is assigned a specific task;

a master processor schedules and allocates work to slave processors.

More common in special-purpose systems

(e.g., embedded MP-SoC, very common in mobile/smartphones)

Remark: the notion of “processor” is relative:

A traditional PC usually has one CPU, but it usually also has a graphics processor,

a communication processor etc., and this was not considered a multi-processing

system. Today, though, almost all computers have multicore CPUs that could be

used for parallel computing, and can thus be considered multiprocessors.

SMP

architecture

Processor Processor Processor


Hardware Multithreading,

Multi-Core

P0 P1

L1$ D1$ L1$ D1$

L2$

Memory Ctrl

Intel Xeon

Dualcore(2005)

Main memory

P0 P1

L1$ D1$ L1$ D1$

L2$

Memory Ctrl

Intel Xeon

Dualcore(2005)

Main memory

Hardware multithreading:

Several hardware threads time-share one core,

switched by the hardware (not the OS)

Multicore CPU:

Several processor cores on a CPU chip

Dual-core, each 2-way hardware-multithreaded

Appears to the OS like 4 standard processors

→ More in TDDD56 Multicore and GPU Programming

T0 T1 T2 T3

CPUThe first dual-core

Intel Xeon CPU (2005)



Multicomputer(”distributed memory parallel computer”)

Several stand-alone servers, each one a SMP,loosely coupled by an interconnection network for internal data exchange

More scalable than shared-memory SMPs

But harder to program (message passing)

More in TDDC78 – Programming parallel computers

NSC Tetralith, LiU, 2018

CPU CPU CPU

Memory Memory Memory

Interconnection Network

…


Parallel Computing on Parallel Systems (4)

Speed-up of a single application by parallel processing?

Requires parallelisation / restructuring of the programusing special parallel programming languages / systems

SMP: Multithreading → TDDB68, TDDC78, TDDD56

HPC clusters: MPI (Message-Passing Interface) → TDDC78

Data centers: Mapreduce / Hadoop → 732A54/TDDE31

Or new, explicitly parallel algorithms

First used in High-Performance Computing for numerically intensive applications (weather forecast, simulations, ...)

Now ubiquitous due to switch to multicore/manycore CPUs


Parallel Computing Systems


Clustered Systems

Like multiprocessor systems, but multiple systems (servers) working together

Often using a high-speed interconnect between servers (e.g. Infiniband)

Usually sharing external mass storage via a storage-area network (SAN)

Some clusters are used for high-performance computing (HPC)

For data centers and critical applications: Provides a high-availability/reliability service which survives failures

Asymmetric clustering - one server runs the application while other servers are in hot-standby mode

Symmetric clustering - has multiple nodes running the same application, monitoring each other

NSC Tetralith, LiU, 2018


Distributed Systems

Loosely coupled system

Networked stand-alone computers (nodes)

each node has its own local memory

Nodes communicate with one another by message passing through various communications lines, such as high-speed buses, telephone lines or wireless connections (LAN, WAN, MAN, Bluetooth, …)

Clustered systems are a special case, with stronger network

Distribute the computation among several physical computers

Network OS / Distributed OS: Middleware gives the illusion of a single OS controlling the network and all involved nodes

e.g. file sharing using NFS (Network File System)

Advantages of distributed systems:

Resource sharing

Computation speed up

Adaptivity: load sharing (migration of jobs)

Fault tolerance

May be either client-server or peer-to-peer systems

More in

TDDD25

Distributed

Systems


Distributed Computing Systems


Example: Cloud Systems By © Raimond Spekking / CC BY-SA 4.0

(via Wikimedia Commons), CC BY-SA 4.0,

https://commons.wikimedia.org/w/index.php

?curid=45282376

InternetThe ”cloud” = other people’s servers.

Physically, a distributed network of data centers (= huge clusters / server farms) + client-server API

Google, Amazon, Microsoft, Facebook, Dell, HPE, IBM, SAP, …

Load balancing, elastic provisioning of compute and storage capacity

Can rent your customized virtual server there

Infrastructure (e.g. compute power, special hardware) as a service

Storage (e.g. remote file system) as a service

Platform (e.g. managed server OS, desktop, database) as a service

Application (e.g. e-mail, word processing, accounting) as a service

Outsourcing the servers

Pay for service, not for ownership

$ per CPU hour, $ per Terabyte/month, or possibly with your data

Clients use service via internet and cloud API, don’t care where it is executed or data is stored

Potential security, confidentiality and privacy issues

Private cloud (e.g. of a company) as alternative to public cloud


TDDB63 – Operating System Concepts – A. Bednarski 1.8

Real-Time Systems

Often used as a control device in a dedicated application/environment, such as controlling scientific experiments, medical devices, avionics / automotive control, industrial control systems, and some display systems.

Well-defined tasks with fixed time constraints.

Hard real-time systems.

Deadlines must be met under all circumstances

Conflicts with time-sharing systems, not supported by general-purpose OSs.

Soft real-time systems

Missed deadline → degraded quality of service

Limited utility in industrial control or robotics

Useful in applications (multimedia, virtual reality) requiring advanced OS features.

More in TDDD07 Real-time Systems


Handheld Systems

Personal Digital Assistants (PDAs), iPADs etc.

Cellular telephones, smartphones

Issues:

Limited memory

Slow processors

Small display screens

Limited battery lifetime

Shipping with 2 GB RAM,

64GB Flash memory

Iphone 8

6 core CPU + embedded GPU + ...

up to 2.39GHz

4.7” (but 1334 x 750 pixels)






Before talking about

Operating Systems…


Before talking about OS, …

… a short note on programming languages suitable for system programming

Early operating systems were implemented in assembly language (and some low-level parts of modern OS still are).

Necessary for direct access to hardware devices

Since the 1970s the dominating language for system programming is C.

Also used a lot in high-performance and embedded computing

Today, no new microprocessor ships without a C compiler.

Many “high-level” programming languages are compiled to C.

You will need C for programming labs in later courses

Operating systems, Compiler construction, Parallel programming, Multicore/GPU programming, Realtime systems, Embedded systems, …


Java

For application programming only

Design goals:

Programmer productivity

Safety

Hardware completely hidden

Comfortable

E.g. automatic memory management

Protection (to some degree)

E.g., array bound checking

Slow

Time-unpredictable

Java vs. C

C

For system programming mainly

Design goals:

Direct control of hardware

High performance / real-time

Minimalistic design

Less comfortable

Little protection

”low-level”


A Short History of C

C was developed in the early 1970’s by Dennis Ritchie at Bell Labs

Objective: structured but flexible programming language

e.g. for writing device drivers or operating systems

Used for implementing the Unix OS

Book 1978 by Brian Kernighan and Dennis Ritchie (”K&R-C”)

”ANSI-C” 1989 standard by ANSI (”C89”)

The C standard for many programmers (and compilers...)

Became the basis for standard C++

Java borrowed much of its syntax

The GNU C compiler (gcc) implemented a superset (”GNU-C”)

”C99” standard by ISO, only minor changes

”C11” (ISO) – multithreading support added

…






How to build and execute

programs on a real computer


The Compilation Workflow

/* file hello.c */

#include <stdio.h>

int main() {

printf("hello, world\n");}

.file "hello.c"

.section .rodata.str1.1,"aMS",@progbits,1.LC0:

.string"hello, world"

.text

.p2align 4,,15.globl main

.type main, @functionmain:

pushl %ebpmovl %esp, %ebpsubl $8, %espandl $-16, %espsubl $16, %espmovl $.LC0, (%esp)call putsleavexorl %eax, %eaxret.size main, .-main


Preprocessing Phase

C Preprocessor (cpp):

modifies the original C source programaccording to directives that begin with the # character.

#include <stdio.h> tells the processor to read the contents of the systemheader file stdio.h and insert it directly into the program text.

Output is another C source program, typically with the .i suffix.


Compilation Phase

Compilation Phase (cc1): Translate the text file hello.i into the text file hello.s.hello.s contains an assembly-language program

Each statement in an assembly language program exactly describes

one low-level machine language instruction in a standard text form.main:

pushl %ebpmovl %esp, %ebpsubl $8, %espandl $-16, %espsubl $16, %espmovl $.LC0, (%esp)call putsleavexorl %eax, %eaxret

Assembly language

provides a common

output language for

different compilers

C

compiler

Assembly

language

Fortran

compiler


hello.s (in x86 assembler language)

.file "hello.c"

.section .rodata.str1.1,"aMS",@progbits,1.LC0:

.string "hello, world"

.text

.p2align 4,,15.globl main

.type main, @functionmain:

pushl %ebpmovl %esp, %ebpsubl $8, %espandl$-16, %espsubl $16, %espmovl $.LC0, (%esp)call putsleavexorl %eax, %eaxret.size main, .-main.section .note.GNU-stack,"",@progbits.ident "GCC: (GNU) 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)"


Assembly Phase

Assembler (as):

translates the text-based assembly program into

machine language instructions,

packages them in a form known as a relocatable object

program, and stores the result in the object file hello.o

hello.o is a binary file (object file) whose bytes encode

machine instructions and -data rather than characters.

Binary files use a system-specific binary file format, e.g. ELF, COFF.

a4 f0 dd 07

7c 65 a6 b2

06 0f c3 dd

a2 ff bd 87

…


Linking Phase

Linker (ld):

merges pre-compiled object files to a single one.

The result is an executable object file that is ready to be

loaded into memory (using the OS loader)

and executed by the system.

The hello program uses the printf function, which is part of the standard C

library. This function resides in a separate precompiled object file (e.g. printf.o or in libc.a) that has to be merged with hello.o.Also some additional code and data (program startup code, C runtime system, etc., in libc.a) is added by the linker.


Calling the entire toolchain

E.g., gcc hello.c –o hello

(here, for the GNU C compiler gcc)

calls cpp, cc1, as, ld for single-module program hello.c

For automatizing the build process of multi-module programs,

building-tools like make or IDEs like ECLIPSE are convenient.


Compiling and Linkingfor Multi-Module C Programs

glob.hxy.h

mymain.c def.cabc.c

stdio.h

#include

preprocess

#include#include

abc.o mymain.o def.o

compile + asm

a.out (executable)

link libc.a

C run-time library is

linked with the user code

cc abc.c cc mymain.c cc def.c


Running an Executable Program

/* hello.c */#include <stdio.h>

int main() {

printf("hello, world\n");}

hello

Executable Object File

(binary)

unix> ./hello

hello, world

unix>

In a Unix system, a shell is a program

which is a command line interpreter

The shell prints a prompt, waits for the

user to type in a command and then

performs the command.

If it is not a built-in shell command, then

the shell assumes it is an executable file

and that it should load and run it.

Input

OutputPreprocess

Compile

Assemble

Link

a4 f0 dd 07

7c 65 a6 b2

06 0f c3 dd

a2 ff bd 87

…


Running the hello program

Reading the hello command from the

keyboard

As the characters ./hello are typed

at the keyboard, the shell program

reads each one into a register and

then it is stored in memoryshell



Loading the executable from disk into main

memory

When the enter key is hit on the

keyboard, the shell knows that

we have finished typing the

command.

The shell then calls the OS to

allocate memory and load the

executable file.

The loader copies the codeand data in the hello object file

from disk to main memory.

Using a technique called

Direct Memory Access (DMA)

the data travels directly from the

disk to the main memory without

passing through the processor.

a4 f0 dd 07

7c 65 a6 b2

06 0f c3 dd

a2 ff bd 87

…

a4 f0 dd 07

7c 65 a6 b2

06 0f c3 dd

a2 ff bd 87

…



Writing the output string from memory to the

display

Once the code and data in the hello object file are loaded into

memory, the processor begins

executing the machine instructions in the hello program’s

main routinemain:

pushl %ebpmovl %esp, %ebpsubl $8, %espandl $-16, %espsubl $16, %espmovl $.LC0, (%esp)call putsleavexorl %eax, %eaxret

These instructions copy the bytes

in the “hello, world” string from

memory to the register file

and from there to the display

device where they are displayed

on the screen


Some questions…

How can we achieve that the shell remains active while hello is

running?

e.g. to accept input from the keyboard for aborting the started program,

or for starting further programs

How do we make sure to re-gain control over the computer

e.g. if a started program hangs in an endless loop?

How can multiple programs (or multiple copies of the same

program) run concurrently (execution overlap in time) even if there

is only one processor?

How can such processes communicate with each other?

How do we make sure that processes use system resources fairly?

…

This (and more) is solved by using an operating system.






OS Basics


What is an Operating System (OS)?

A program that acts as an intermediary

between a user of a computer and the computer hardware.

Operating system goals:

Execute user programs in a well-defined environment.

Make application software portable and not tied to a specific machine

hiding low-level and implementational detail

Make the computer system convenient to use.

Administrate system resources.

Improves overall system reliability

error confinement, fault tolerance, reconfiguration...

Enable efficient use of the computer hardware

support parallel activity, avoid wasted CPU cycles...

Provides protection mechanisms for user programs and files

and security against possible threats from within and outside the system


What is an Operating System (OS)?

An operating system provides an environment

within which other programs can do useful work;

the OS does not perform any “useful” function itself.


1%

99%

Embedded systems

Microprocessor market shares in

1999

Where are OSs found?

Real-time systems...


Operating Systems (A Selection)

General purpose operating systems

Unix

incl. variants, such as Sun/Oracle Solaris, HP-UX

Linux

Windows 95/98/2000, NT/XP/Vista/7/8/10/…

Mac OS X

…

Application specific operating systems, e.g. mobile OS, real-time OS

Android

iOS

Symbian

Windows CE, Mobile, Phone

Embedded Linux

RT-Linux

VxWorks

OSE

QNX

...


Operating System Definition

OS is a resource allocator

Manages all resources of a computer system

Decides between conflicting requests

for efficient and fair resource use

OS is a control program

Controls execution of programs

to prevent errors and improper use of the computer

No universally accepted definition

“The one program running at all times on the computer”

is called the kernel.

Everything else is either a system program (ships with the

operating system) or an application program.


Computer System Structure

Computer system

can be divided into 4 components:

Hardware

provides basic computing resources

CPU, memory, I/O devices

Operating system

controls and coordinates use of hardware

among various applications and users

Application programs

define the ways in which the system resources are used

to solve the computing problems of the users

Word processors, compilers, web browsers, database systems, games

Users

People, machines, other computers


Agenda

Lecture I:

Computer Systems Overview

Building and Executing Programs

Operating System Basics

Lecture II:

Interrupts and I/O

System Calls

CPU Management

Memory Management

File System

Protection and Security

introduction to operating systems part itdde25/info/slides/tdde25_fo12_1sl.pdf(”distributed memory...

Documents