kt6213 lecture 14: operating systems – file management & internal os computer organization and...

KT6213

Lecture 14: Operating Systems – File Management & Internal OS

Computer Organization Computer Organization and Architectureand Architecture

KT6213

Introduction to FilesIntroduction to Files Collection of (usually related) data Block

◦ Typically 256, 512, or 1024 bytes◦ Files usually require a 1-block minimum

Cluster◦ Groups of one or more blocks

Blocks or clusters correspond to one or more sectors on a disk’s single track or cylinder

File extension◦ Name of file includes identification of what type of file it is

File Association◦ The file specifies which program with which it is to be

used

KT6213

TerminologyTerminology Logical view

◦ Contents and attributes of files viewed by the user Physical view

◦ The actual way a file is stored within the computer system Sequential access

◦ Data files whose records always have to be retrieved from the beginning

Random or relative access◦ Data files whose records can be retrieved from anywhere in the

file in random sequence Contiguous

◦ The blocks that hold a particular file are stored together Noncontiguous

◦ The blocks that hold a particular file are scattered all over the device

KT6213

Database File – Table ImageDatabase File – Table Image

KT6213

Database File – Form ImageDatabase File – Form Image

KT6213

Database File – Stream ImageDatabase File – Stream Image

Closer to physical representation of file Example: YouTube video

KT6213

Logical View vs. Physical ViewLogical View vs. Physical View

KT6213

File Management SystemFile Management System Provides a logical view for the user and hides the

physical implementation Consistent set of commands that are translated to a

form appropriate for the device Consistent view of files regardless of file type, file

characteristics, or device Supports manipulation of data within the file Manages directory structures which are presented in a

logical view Command shell takes user file commands and program

file requests and translates them for the file manager Requests data transfers from I/O device drivers File security and protection of file integrity

KT6213

File Manager Request HandlingFile Manager Request Handling

KT6213

File Operations [1/3]File Operations [1/3]

File as a whole◦ Copy, Move◦ List, Print◦ Load and execute a program◦ Load file into memory◦ Store file from memory◦ Append data from memory to file◦ Compile, assemble a file

KT6213

File Operations [2/3]File Operations [2/3]

Within a file◦ Open a file◦ Read a number of bytes from file◦ Write a number of bytes to a file◦ Move the file pointer forward or backward◦ Move file pointer to beginning of a file◦ Close a file

KT6213

File Operations [3/3]File Operations [3/3]

Record Storage◦ Retrieve a record (read)◦ Store a record (write)◦ Add a record to a file◦ Delete a record◦ Modify contents of a record

KT6213

File Directory OperationsFile Directory Operations

File Directory◦ Create a new (empty) file◦ Move a file from one directory to another◦ Rename a file◦ Append one file to another◦ Delete a file

KT6213

File Management and I/O FunctionsFile Management and I/O Functions

Separation between the two allows1. I/O devices can change while keeping the

file system the same2. Redirecting of data is simple

KT6213

File Access MethodsFile Access Methods

Sequential Access◦ File is read in sequence from beginning to end◦ Majority of all files◦ Program source and binary files

Random Access◦ Assumes file is made up of fixed length logical records◦ Hashing is a common method used to calculate the

location of an internal logical record Indexed Access

◦ Additional means for accessing and viewing records in a file

◦ Key indexes◦ ISAM – indexed sequential access method

KT6213

Physical File StoragePhysical File Storage

ContiguousNon-contiguous

◦ Linked◦ Indexed

Examples◦ DOS/Windows FAT◦ UNIX i-nodes◦ Windows NTFS

Free space management

KT6213

Contiguous Storage AllocationContiguous Storage Allocation

Assign blocks (all in a row) to hold the fileAccess is simple for both sequential and

random methodsDisadvantages

◦ Space must be large enough◦ Have to take into account file growth◦ May need to be moved if it outgrows its

space◦ Fragmentation of disk

Allocation strategies to minimize fragmentation◦ First-fit, best-fit

Eventually disk becomes fragmented

KT6213

Linked AllocationLinked Allocation

Non-contiguousEach block contains a link to the next physical

blockVariant – links in both directionsAdvantages

◦ no fragmentation◦ Adding to a file is easy

Disadvantages◦ Not usable for random access◦ Additional disk head searching◦ Overhead in storing the pointers◦ Recovery of a defective block is difficult

KT6213

File Allocation

Contiguous Storage Allocation

Linked Allocation

KT6213

Windows FATWindows FAT

File Allocation Table (FAT)◦ Linked allocation with links stored in a

table◦ Table contains the first block of each file on

the disk or disk partition◦ Successive blocks contain a link to the next

blockDisadvantages

◦ Requires a tremendous amount of space◦ File integrity can be easily compromised

KT6213

File Allocation TableFile Allocation Table

Linked Allocation and File Allocation Table

KT6213

Indexed AllocationIndexed AllocationNon-contiguousAll link pointers for a file are stored together in

a single block called the index blockOne index block per fileAdvantages

◦ No fragmentation◦ Can be used for random access

Disadvantage◦ Slower due to additional access of the index

block◦ Additional disk head searching◦ Recovery of a defective block is difficult

KT6213

Indexed AllocationIndexed Allocation

Index blocks for indexed allocation of linked files

KT6213

Unix i-nodesUnix i-nodes

Indexed file allocation Index block contains

◦ File attributes◦ 10 direct blocks◦ 1 single indirect◦ 1 double indirect◦ 1 triple indirect

Advantages◦ Fast for small blocks◦ Can accommodate very large files – 100’s

of gigabytes

KT6213

Unix i-nodesUnix i-nodes

KT6213

Windows NTFSWindows NTFS

Dynamically sized volumesVolumes may be a fraction of a disk or span

many disksMaster File Table (MFT) of 1kb records

◦ First 16 records are metadata files which describe the volume

◦ First record stores the MFT attributes◦ Each file has an MFT entry

File records made up of attributes, including file information and data

KT6213

NTFS Volume LayoutNTFS Volume Layout

KT6213

Free Space ManagementFree Space Management

Bit map method◦ one bit for each block

to indicate if it is usedor free

Linked list method◦ Pointer to first free block◦ Each free block has a pointer to the next◦ Blocks are allocated from the beginning◦ Deleted files are placed at the end

KT6213

Other Secondary Storage AllocationOther Secondary Storage Allocation

Tape Allocation◦ Not practical to reallocate space in the middle

of the tape◦ Files that grow must be re-written◦ Files are stored contiguously whenever

possibleOptical drives and flash drive file allocation

◦ Similar to that of hard disks◦ UDF (Universal Data Format) supports up to 2

terabytes of data◦ Hierarchical directory format◦ Support for both HD-DVD and Blu-Ray DVD

formats

KT6213

Partitions, File Systems, Volumes, and Partitions, File Systems, Volumes, and PoolsPools

KT6213

Directory StructureDirectory Structure

Provides a means of organization so that files can be located easily and efficiently

Hide the physical devices from the logical view of the files

Partitions◦ Independent subsections of a device

Volume◦ Directory structure for a particular partition◦ Needs to be mounted to be incorporated

into the overall file system structureContain file attributes

KT6213

Tree-Structure DirectoryTree-Structure Directory Hierarchical with a top-level root directory from which all

other directories stem All directories and files have names Separator

◦ Used to indicate subdirectories and files located in a directory

◦ / UNIX◦ \ DOS, Windows

Pathname◦ Absolute – full pathname starting from the root directory◦ Relative – pathname is created starting from the current

directory Search Paths

◦ Directory locations that the operating system uses to locate files

KT6213

Tree-Structure DirectoryTree-Structure Directory

Pathname Examples:

Absolute:C:\FINANCE\QUICKEN\Q.EXE

Relative from the FINANCE directory: QUICKEN\Q.EXE

Search Path: PATH=C:\DOS;C:\FINANCE\QUICKEN◦ Now the programs in the two directories can be run by specifying the

name of the program without the absolute or relative pathnames

For our examples, root is on disk device C:

KT6213

Acyclic Directory StructuresAcyclic Directory Structures

Tree-structure that permits links between separate branches of the tree

Advantage◦ Easy user access

Disadvantages◦ Cycles and dangling links

Examples◦ Windows shortcuts◦ Unix hard and symbolic links◦ MacIntosh aliases

KT6213

An Acyclic-Graph DirectoryAn Acyclic-Graph Directory

KT6213

Graph with a CycleGraph with a Cycle

KT6213

Hard Links vs. Symbolic LinksHard Links vs. Symbolic Links

KT6213

Network File AccessNetwork File Access

FTP◦ File Transfer Protocol◦ Part of the TCP/IP protocol family

Network file systems◦ Windows

Drive letters aliased to remote file systems

◦ UNIX Network File System (NFS) Remote Procedure Call (RPC)

KT6213

Typical NFS ConfigurationTypical NFS Configuration

KT6213

File ProtectionFile Protection

Logins and passwordsRead, write, and execute protectionsACL – access control list, permissionsUNIX/Linux – owner, group, everyone

KT6213

Data Storage ApproachesData Storage Approaches

Standard client server configuration

Storage area network configuration

KT6213

File ProtectionFile Protection

Most systems provide three forms of protection on files◦ Read protection◦ Write protection◦ Execution protection

Access control list (ACL)◦ List of users who may access the file for

each of the forms of protection◦ Tremendous overhead if there are a lot of

usersOwner/Group/Everyone protection method

◦ UNIX, Linux

KT6213

File Directory Showing ProtectionFile Directory Showing Protection

ls –lF list files in directory using a long format and indicate file type

10-char code for file protection

1st char d for directory, - for file, s for symbolic link

2nd to 4th char permissions for the owner

5th to 7th char permissions for the group

8th to 10th char permissions for everyone

r - read permission, w - write permission, x - execute permission

KT6213

Journaling File SystemsJournaling File Systems

Log file records every system transaction that requires a write access to the file system

Two levels of capability1. Protect the integrity of the file system

structure only Example: Windows NTFS file system

2. Also guarantees the integrity of data that has not yet been written to the disk

Examples: Linux ext3 and ext4, IBM JFS

KT6213

Primary Kernel FunctionsPrimary Kernel Functions

File manager translates logical file requests into specific physical I/O requests

I/O Control System (IOCS) performs resource allocation and device management

Memory management determines if it is possible to load programs and data into memory and if so where in memory

Scheduler allocates time for the program to execute

KT6213

Miniature Operating SystemMiniature Operating System

Block Diagram

MemoryMap

Process Dispatch

KT6213

BootstrappingBootstrapping

Execution begins with bootstrap loader (mini-loader, IPL) stored in ROM

Looks for OS program in a fixed location (possibly on the network)

Loads OS into RAMTransfers control to starting location of OS Loader program in OS used to load and

execute user programs

KT6213

BootstrappingBootstrapping

Cold vs. warm boot (does not retest the system)

KT6213

Process [1/2]Process [1/2]

Process: basic unit of work in the OS◦ A program together with all the resources

that are associated with it as it is executed◦ Program: a file or listing◦ Process: a program being executed

Independent vs. cooperating processesPID (process ID): a unique identifier for each

process

KT6213

Process [2/2]Process [2/2]

Process creation◦ Forking, spawning, cloning a new process◦ Parent and child processes

Process context◦ All relevant register data including the

program counter◦ Allows interruption and restart invisibly

KT6213

Two Processes Sharing a Single Two Processes Sharing a Single ProgramProgram

KT6213

Process Control BlockProcess Control Block

A block of data for each process in the system

Contains all relevant information about the process

Typical process control block on the right

KT6213

Process StatesProcess States

Three primary process operating states◦ Ready state◦ Running state◦ Blocked state

Dispatching - Move from ready state to running state

Wake-up - Move from blocked state to ready stateTime-out - Move from running state to ready stateProcess completion

◦ killed, terminated, destroyedAdditional states – suspend, swapResumption – Move from suspended state to

ready state

KT6213

Process State DiagramProcess State Diagram

KT6213

ThreadsThreads

‘Miniprocess’ that can be run independent of other parts of the process◦ Event-driven programs◦ Shares resources allocated to its parent process

including primary storage, files and I/O devices◦ Each thread has its own context

Advantage of process/thread families over multiple independent processes:◦ Reduced OS overhead for resource allocation

and process management◦ Substantially less information than a normal

PCB

KT6213

Loading and Executing a ProcessLoading and Executing a Process

KT6213

CPU SchedulingCPU Scheduling

High-level scheduling Adding a program to the pool of programs to be executed

Short-term scheduling(dispatcher)

Deciding which process shall be executed next by the processor

Mid-level scheduling Swapping processes

I/O scheduling Deciding which process’s pending I/O request shall be handled by an available I/O device

KT6213

Dispatching ObjectivesDispatching Objectives

Ensure FairnessMaximize throughputMinimize turnaround timeMaximize CPU utilizationMaximize resource allocationPromote graceful degradationProvide minimal and consistent response timePrevent starvation

KT6213

Nonpreemptive DispatchingNonpreemptive Dispatching

First in, first out (FIFO)◦ Unfair to short processes and I/O based

processesShortest Job First (SJF)

◦ Longer jobs can be starvedPriority Scheduling

◦ Job with the highest priority is selected◦ If multiple jobs have the highest priority then

dispatcher selects among them using FIFO

KT6213

Preemptive DispatchingPreemptive DispatchingRound robin

◦ Inherently fair and maximizes throughput

Dynamic Priority◦ Based on ratio of CPU time to total time process has been in the system◦ Smallest ratio has highest priority◦ Linux; Windows 2000, XP, Vista, 7

KT6213

Preemptive DispatchingPreemptive Dispatching

Multilevel feedback queues◦ Favors short jobs, I/O bound jobs◦ Each level assigns more CPU time

KT6213

Virtual Memory – Basic IdeasVirtual Memory – Basic IdeasVirtual memory increases the apparent amount

of memory by using far less expensive hard disk space

Provides for process separationDemand paging

◦ Pages reside on hard disk and are brought into memory as needed

Page table◦ Keeps track of what is in memory and what is

still out on hard disk Inverted page table

◦ Representation of physical memory with the page that resides in each frame

KT6213

Pages and Frames [1/2]Pages and Frames [1/2]

Program Memory

Unit Page Frame

Address Logical Physical

Size 2 to 4KB 2 to 4KB

Amount# of bits in instruction word

Installed memory

KT6213

Pages and Frames [2/2]Pages and Frames [2/2]

Each program has its collection of pages.

The total number of pages can exceed the number of frames (physical memory).

A Simple Page Table TranslationPages and Frames

KT6213

Page Translation ProcessPage Translation Process

KT6213

Disk

Virtual Memory Pages

Page FramePages not in main memory:page fault when accessed

1 2 3 4

5 6 7 8

9 10 11

1234567891011

64

8

1012

7

Page TablePage Table

Swap space

KT6213

Mapping for Three ProcessesMapping for Three Processes

KT6213

Inverted Page TableInverted Page Table

Inverted Page Table for the previous slideThe table represents what is in physical memory

KT6213

Steps in Handling a Page FaultSteps in Handling a Page Fault

KT6213

Concept of LocalityConcept of Locality

Most memory references confined to small region

Well-written program in small loop, procedure or function

Data likely in array and variables stored togetherWorking set

◦Number of pages sufficient to run program normally, i.e., satisfy locality of a particular program

KT6213

Page Replacement AlgorithmsPage Replacement Algorithms

Page fault◦ Page is not in memory and must be loaded from disk

Algorithms to manage swapping◦ FIFO – First-In, First-Out

Belady’s Anomaly – when increasing number of pages results in more page faults

◦ LRU – Least Recently Used◦ LFU – Least Frequently Used◦ NUR – Not Used Recently

Referenced bit Modified (dirty) bit

◦ Second Chance Replacement algorithms Thrashing

◦ too many page faults affect system performance

KT6213

Frame Lookup ProceduresFrame Lookup Procedures

KT6213

SegmentationSegmentation

KT6213

Virtual Memory TradeoffsVirtual Memory Tradeoffs

DisadvantagesSWAP file takes up space on diskPaging takes up resources of the CPU

AdvantagesPrograms share memory spaceMore programs run at the same timePrograms run even if they cannot fit into

memory all at onceProcess separation

KT6213

Virtual Memory vs. CachingVirtual Memory vs. Caching

Cache speeds up memory accessVirtual memory increases amount of

perceived storage ◦ Independence from the configuration and

capacity of the memory system◦ Low cost per bit compared to main

memory

KT6213

Secondary Storage SchedulingSecondary Storage Scheduling

FCFS - First-Come, First-ServedShortest Distance First

◦ Indefinite postponement problemScan

◦ Middle of disk gets serviced twiceN-Step C-Scan

◦ Disk seek in only one direction◦ Return after last request in queue served◦ Two queues

Queue of requests being processed Queue of new requests

KT6213

Scan Scheduling AlgorithmScan Scheduling Algorithm

KT6213

Comparison of Different Disk Comparison of Different Disk AlgorithmsAlgorithms

KT6213

Network OS ServicesNetwork OS Services

File transfer programs Access to data files on other computers on the network

◦ Computer naming scheme is required for some network systems

Print services◦ Print requests are redirected by the OS to the

network station that manages the requested printer Other peripheral sharing services Web services Messaging services API network services Security and network management services Remote processing services

KT6213

Access for a Networked OSAccess for a Networked OS

KT6213

Other OS IssuesOther OS Issues

Deadlock◦ Two or more processes simultaneously

have resources that are required by one another in order to proceed

◦ Prevention◦ Avoidance◦ Detection and recovery

Process Synchronization◦ Required by cooperating processes when

one process is dependent on the other

KT6213

Virtual MachinesVirtual MachinesVirtualization

◦ Using a powerful computer to simulate a number of computers

Virtual machines◦ A simulated computer

Hypervisor◦ Layer of software and/or hardware that separates

one or more operating systems from the hardware

◦ Type 1 (native) – hypervisor software interfaces directly with the computer hardware

◦ Type 2 (hosted) – hypervisor software runs as a program on the operating system

KT6213

Virtual Machine ConfigurationVirtual Machine Configuration