solaris fundamentals

Thoughtful Solutions Ltdhttp://very.thoughtful-solutions.info/

Solaris Fundamentals Training

Abstract

Operations Group Solaris Fundamentals Training

http://very.thoughtful-solutions.info/

Attribution-ShareAlike 2.0 United Kingdom

You are free:

• to copy, distribute, display, and perform the work• to make derivative works

• to make commercial use of the work

Under the following conditions:

Attribution. You must give the original author credit.

Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a licence identical to

this one.

• For any reuse or distribution, you must make clear to others the licence terms of this work.

• Any of these conditions can be waived if you get permission from the copyright holder.

Your fair use and other rights are in no way affected by the above.

This is a human-readable summary of the Legal Code (the full licence).

Disclaimer

http://creativecommons.org/licenses/disclaimer-popup?lang=en_GB

http://creativecommons.org/licenses/by-sa/2.0/uk/legalcode

http://creativecommons.org/licenses/disclaimer-popup?lang=en_GB

Unix History and Lore ................................................................................................ 9 Introduction .......................................................................................................... 10 What is Unix? ....................................................................................................... 11 Distributed Computing with Unix ........................................................................ 12 Aims ..................................................................................................................... 13 Why Unix? ........................................................................................................... 14 Unix Philosophy ................................................................................................... 15 Major Features ...................................................................................................... 16 Unix Lore ............................................................................................................. 17 Potted History ....................................................................................................... 18 Unix Flavours ....................................................................................................... 20

Key Concepts ............................................................................................................ 26 The Business of Computing ................................................................................. 27 Files & Processes .................................................................................................. 28 Organisation of Files ............................................................................................ 29 Organisation of Processes ..................................................................................... 30 Login .................................................................................................................... 31 Login File & Process ............................................................................................ 32 What are Users? .................................................................................................... 33 Unix Structure ...................................................................................................... 34 Kernel Responsibilities ......................................................................................... 35

Getting Started: Login .............................................................................................. 36 CDE Login Screen ............................................................................................... 37 Command Line Login Screen ............................................................................... 39 Usernames ............................................................................................................ 40 Valid Usernames .................................................................................................. 41 Good Passwords ................................................................................................... 42 Passwords ............................................................................................................. 43 Changing Passwords [passwd] ............................................................................. 44 Login Failures ....................................................................................................... 45 Why login fails ..................................................................................................... 46 Interruptions ......................................................................................................... 47 Ending the sessions [logout] ................................................................................. 48 /etc/passwd : home of identity .............................................................................. 49 /etc/shadow : Password vault ................................................................................ 50 /etc/group: where groups live ............................................................................... 51 Summary .............................................................................................................. 52

File Hierarchy ........................................................................................................... 53 The Unix File system ............................................................................................ 54 Hierarchical Structure ........................................................................................... 55 Directory Paths ..................................................................................................... 56 File names ............................................................................................................. 57 What is a file? ....................................................................................................... 58 Varieties of Files .................................................................................................. 59 Traversing the File System : cd ............................................................................ 60 Examining Directory Contents : ls ....................................................................... 61 Shell Wildcards .................................................................................................... 62 Building the File System : mkdir/rmdir ................................................................ 63 Copying Files : cp ................................................................................................. 64 Copying with Wildcards ....................................................................................... 65 Moving Files : mv ................................................................................................ 66 Deleting Files : rm ................................................................................................ 67

Linking Files ......................................................................................................... 68 Linking Files : ln .................................................................................................. 69 Hard links : ln ....................................................................................................... 70 Symbolic links : ln ................................................................................................ 71 Why create links? ................................................................................................. 72 Pathname abbreviations ....................................................................................... 73 Shell Meta-characters ........................................................................................... 74 * - Asterisk ........................................................................................................... 75 ?- Question Mark .................................................................................................. 76 [ ] – Square Brackets ............................................................................................ 77 ` ` - Backticks ....................................................................................................... 78 Quoting in Shells [ ” ’ ] ........................................................................................ 79 Escaping in shells [ \ ] .......................................................................................... 80

Getting Help ............................................................................................................. 81 Getting help : man ................................................................................................ 82 man : Output ......................................................................................................... 85 man : example ....................................................................................................... 88 man page sections ................................................................................................. 92 manual entries in other sections ........................................................................... 93 Other useful man options ...................................................................................... 94 Google .................................................................................................................. 95 Google: Example .................................................................................................. 96 SunSolve ............................................................................................................... 97

File Types and Listing ............................................................................................. 98 Varieties of Files .................................................................................................. 99 What sort of file [file] ......................................................................................... 100 Strings in a file [strings] ..................................................................................... 101 Contents of a file [cat] ........................................................................................ 102 Listing contents of files [more] .......................................................................... 103 Listing contents of files [less] ............................................................................. 104 Top few lines [head] ........................................................................................... 105 Bottom few lines [tail] ........................................................................................ 106 Fields in a file [cut] ............................................................................................. 107 Counting things [wc] .......................................................................................... 108 Finding the differences [diff] .............................................................................. 109 Differences of 3 files [diff3] ............................................................................... 110 Sorting files and output [sort] ............................................................................. 111 Handling duplicate lines [uniq] .......................................................................... 112

Redirection ............................................................................................................. 113 Why ? ................................................................................................................. 114 Standard Files ..................................................................................................... 115 Redirection : < > 2> ........................................................................................... 116 Redirection : >> .................................................................................................. 117 Redirection : Merging ........................................................................................ 118 Redirection: Multiple redirections ...................................................................... 119 Redirection : Examples ....................................................................................... 120 More redirection: File creation ........................................................................... 121 Pipelines : | ......................................................................................................... 122 Pipelines : examples ........................................................................................... 123 Named Pipes ....................................................................................................... 124

File Security ............................................................................................................ 125 Security Overview .............................................................................................. 126

Permissions ......................................................................................................... 127 Finding the permissions [ls –l] ........................................................................... 128 Types of users ..................................................................................................... 129 Access Permissions ............................................................................................ 130 How is access decided ? ..................................................................................... 131 Controlling access [chmod] ................................................................................ 132 Controlling access [chmod]: The modes ............................................................ 133 Octal Modes ....................................................................................................... 134 Octal Modes : example ....................................................................................... 135 User ID and Group ID ........................................................................................ 136 Real & Effective User/Group ID ........................................................................ 137 Who am I [id] ..................................................................................................... 138 Ownership .......................................................................................................... 139 Changing Ownership: chown and chgrp ............................................................ 140 setuid .................................................................................................................. 141 setgid .................................................................................................................. 142 setuid and setgid with directories ....................................................................... 143 Sticky bit ............................................................................................................. 144 Default Permissions [umask] .............................................................................. 145

Searching and Translating ...................................................................................... 147 Why? .................................................................................................................. 148 find : Finding files .............................................................................................. 149 find : Conditions ................................................................................................. 150 find : Examples ................................................................................................... 151 Regular Expressions ........................................................................................... 152 Regular Expressions : Meta-characters .............................................................. 153 Regular Expressions: Character Classes ............................................................. 154 Regular Expressions : Two warnings ................................................................. 155 Regular Expressions : Examples ........................................................................ 156 Translate [tr] ....................................................................................................... 157 Finding things [grep] .......................................................................................... 158 Stream Editor [sed] ............................................................................................. 159 sed : Substitution Examples ................................................................................ 160 sed ....................................................................................................................... 164 Who is using the system ? [who] ........................................................................ 165 Quicker who [w] ................................................................................................. 166 Find a spelling [look] .......................................................................................... 167

Processes and Control ............................................................................................. 168 Process Overview ............................................................................................... 169 Generating Processes .......................................................................................... 170 Understanding Processes .................................................................................... 172 Displaying the Processes [ps] ............................................................................. 173 Displaying the Processes [prstat] ........................................................................ 174 prstat : column headings ..................................................................................... 175 Managing Processes ........................................................................................... 177 Background a command [ & ] ............................................................................ 178 nohup .................................................................................................................. 179 Listing the jobs [jobs] ......................................................................................... 180 Foreground a command [fg] ............................................................................... 181 Killing processes [kill] ....................................................................................... 182 Job Management Commands ............................................................................. 183 /proc .................................................................................................................... 184

vi: Visual Editor ..................................................................................................... 185 The Visual Editor ............................................................................................... 186 Modal Editor ...................................................................................................... 187 Editing a file ....................................................................................................... 188 Leaving vi ........................................................................................................... 189 Entering Text ...................................................................................................... 190 Insert / Replace mode ........................................................................................ 191 Command Mode ................................................................................................. 192 Cursor Modes ..................................................................................................... 193 Moving around in vi ........................................................................................... 194 Deleting Text ...................................................................................................... 195 Searching in vi .................................................................................................... 196 Replacing Text .................................................................................................... 197 More text replacement ........................................................................................ 198 Substituting text ................................................................................................. 199 Coping text ......................................................................................................... 200 Retrieving text .................................................................................................... 201 Miscellaneous tricks ........................................................................................... 202 ex commands ...................................................................................................... 203 ! (bang) ............................................................................................................... 204 set commands ..................................................................................................... 205 Typical Set Commands ....................................................................................... 206 .exrc : making set commands stick ..................................................................... 207 Final Thoughts .................................................................................................... 208

Shells and Variables ............................................................................................... 209 What is an Environmental Variable ? ................................................................. 210 Where do they come from ? ............................................................................... 211 Example variables : Which shell? ...................................................................... 212 Example variables : Process ID [$$] .................................................................. 213 Example Variables : Exit Status [$?] .................................................................. 214 Creating local variables ...................................................................................... 215 Removing environmental variables .................................................................... 216 Making variables global [export] ....................................................................... 217 Displaying the environment [set] [env] .............................................................. 218 Search paths [PATH] .......................................................................................... 219 Searching the PATH [which] ............................................................................. 220 Making the environment stick ............................................................................ 221 Initialisation file sequences ................................................................................ 222

More about Shells ................................................................................................... 223 Quoting in Shells [ ” ’ ‘ ] .................................................................................... 224 Escaping in shells [\] .......................................................................................... 225 Custom Prompts ................................................................................................. 226 Aliases [alias] ..................................................................................................... 227 Removing Aliases [unalias] ................................................................................ 228 Command recall [history] ................................................................................... 229 Command line editing: the vi way ...................................................................... 230 Command line editing: the emacs way ............................................................... 231

Shell Scripting ........................................................................................................ 232 What is a command ? ......................................................................................... 233 Shell Scripts ........................................................................................................ 234 Flow control ....................................................................................................... 235 Testing conditions [test] ..................................................................................... 236

Flow control: if ................................................................................................... 237 Flow control: else ............................................................................................... 238 Flow control: elif (i.e.“else if”) .......................................................................... 239 Flow control: nested if statements ...................................................................... 240 Flow control: while / until .................................................................................. 241 Flow control: while / until examples .................................................................. 242 Flow control: for ................................................................................................. 243 Positional Parameters ......................................................................................... 244 Positional Parameters: continued ....................................................................... 245

Networking ............................................................................................................. 246 Networking and Distributed Systems ................................................................. 247 Basic Network Services ...................................................................................... 248 Distributed File Systems ..................................................................................... 249 Distributed Window Systems ............................................................................. 250 Protocol Layers ................................................................................................... 251 The OSI 7 Layer Model ...................................................................................... 252 The TCP/IP Protocol Stack ................................................................................ 253 Concepts ............................................................................................................. 254 Connectionless protocols .................................................................................... 255 Connection-oriented protocols ........................................................................... 256 Ports + Sockets ................................................................................................... 257 Important Port Numbers ..................................................................................... 258 UDP - Fundamentals .......................................................................................... 259 UDP - Frame format ........................................................................................... 260 UDP - Issues ....................................................................................................... 261 TCP - Fundamentals ........................................................................................... 262 TCP - Frame format ............................................................................................ 263 TCP - A Sample Session .................................................................................... 264 TCP - Options ..................................................................................................... 265 Name Services .................................................................................................... 266 Domain Name Service (DNS) ............................................................................ 267 Domain Name Service: Examples ...................................................................... 268 Structure of DNS name space ............................................................................. 269 LDAP .................................................................................................................. 270 /etc/nsswitch.conf ............................................................................................... 271 Connect to another host [telnet] ......................................................................... 272 Fetching remote files [ftp] .................................................................................. 273 Inside ftp ............................................................................................................. 274 Secure remote connections [ssh] ........................................................................ 275 Remote login without passwords [ssh] ............................................................... 276

Startup and shutdown ............................................................................................. 277 Boot Stages ......................................................................................................... 278 Firmware ............................................................................................................ 279 POST output ....................................................................................................... 280 Open Boot PROM (OBP) ................................................................................... 282 OBP: boot command .......................................................................................... 283 Kernel Initialisation ............................................................................................ 284 The /etc/system file ............................................................................................. 285 Boot output ........................................................................................................ 286 init and run levels ............................................................................................... 287 Predefined run levels for Solaris ........................................................................ 288 The /etc/inittab file ............................................................................................. 289

Run Control Scripts ............................................................................................ 291 A Typical Run Control Script ............................................................................. 292 The /etc/init.d directory ...................................................................................... 293 Run level commands .......................................................................................... 294 Run level commands: shutdown ......................................................................... 295 Run level commands: halt .................................................................................. 296 Run level commands: reboot .............................................................................. 297 Run level commands: telinit ............................................................................... 298

Devices ................................................................................................................... 299 Local Devices ..................................................................................................... 300 Device Naming Conventions .............................................................................. 301 Device aliases [devalias] .................................................................................... 302 Partitions ............................................................................................................. 303 Why Partition? .................................................................................................... 304 Partition Tables ................................................................................................... 305 Listing partitions [prtvtoc] .................................................................................. 306 prtvtoc output ..................................................................................................... 307 Creating partitions [format] ................................................................................ 308 The format Main Menu ...................................................................................... 309 The partition Menu ............................................................................................. 310 Partitioning a disk using modify ......................................................................... 311 File Systems ....................................................................................................... 313 Network File Systems ......................................................................................... 314 Media errors ....................................................................................................... 315 Checking and fixing file systems [fsck] ............................................................. 316 Backup: Principles .............................................................................................. 317 Backup Commands: tar ...................................................................................... 319 Backup Commands: tar examples ...................................................................... 320 Backup Commands: cpio .................................................................................... 321 Backup Commands: cpio examples .................................................................... 322 Backup Commands: dd ....................................................................................... 323 Backup Commands: mt ...................................................................................... 324

Unix History and Lore

Unix History and Lore

“The wonders of the ages assembled for you edification, education, and enjoyment – for a price”

P.T. Barnum

Introduction

Introduction

What is Unix?

Distributed Computing with Unix

Why Unix?

Unix Philosophy

Unix Major Features

Unix Lore

A Potted History

Unix Flavours

The Evolution of Unix

What is Unix?

Multi-user, multi-process, multi-access operating system

Allows users to run programs, manage their own files and use devices

What is Unix?

Unix

Unix™ is a multi-user, multi-process, multi-access operating system. This means that it can support multiple simultaneous users, each executing multiple programs.Unix provides an operating environment for users to run programs, manage files, accessing devices, communicate with each other and coordinate their activities.Unix is commonly used in networking environments, allowing data and resources to be shared amongst the connected machines.


Users share resources around a distributed network of machines


multiplexer / hub

terminator

ethernet segment

repeater

router

transceiver

workstations

Distributed environments consist of machines interconnected through some networking mechanism. Machines are able to access each others disks, peripherals and CPUs.Individual users connected to a workstation have the resources of the entire network at their disposal. Access to the network is often transparent; for example, an application may physically reside on one machine, be executed on a user’s local machine and make use of configuration and data files from a third machine. A distributed network of computers and devices is usually presented as an integrated, homogenous, environment to users.

Aims

Aims

Design Goal“... to create a computing environment where they themselves

(the staff of CSRG) could comfortably and effectively pursue their own work - programming research.”

not so much goals in early design OS for programmer, not for production

- simple and easy (not performance and functionality)- interactive (no batch processing)

C form the original programmers’ preference- easy porting -> wide spread of Unix

small hardware platform

- entails elegance

Why Unix?

Why Unix?

Highly portable, extensible & configurable- written largely in C- modular kernel and command set- configuration soft coded

An open operating system- vendor independent

Relatively standardised- IEEE POSIX, X-Open XPG- Open Software Foundation

Provides a productive development environment- software tools- file processing tools

Over 90% of the Unix operating system is written in the C high-level programming language. This makes the system relatively easy to port to new hardware. Unix is based on a few simple concepts and is structured in a highly modular fashion. It has been able to evolve as new technologies become available, but has remained relatively standardised. Such groups as the IEEE POSIX committee, the X/Open Forum and the Open Software Foundation (and previously Unix International) have strived to keep Unix developments open and standardised. Unix is not tied to any particular vendor, being perhaps the first true open operating system. Implementations of the system exist on some of the smallest machines in the world (such as PCs) and also on the biggest machines (such as super-computers).

Unix Philosophy

Unix Philosophy

Write programs that do one thing and do it well

Write programs to work together

Write programs to handle character streams, because this is the universal interface

Unix has often been described as a toolbox. Rather than being an entity in itself, it is a collection of tools which work together. The notion of command pipelines, the ability to feed the output of one command directly into the input of another, led to this philosophy.Doug McIIroy, responsible for writing pipes, diff (and reportedly herding cats) was one of the first to articulate the above philosophy.Writing programs to do one thing and do it well was a departure from other systems which tended to bundle heaps of functionality into one place. As a consequence, such software was often large, expensive to run, and difficult to adapt to changing requirements.The problem with writing many simple programs comes when it is necessary to achieve a complex task. The programs must be arranged to work together. Unix achieves this through command pipelines, because most of the utility programs in Unix adhere to the universal interface of the character stream. That is, commands produce as their output and receive as their input a stream of characters. Such notions as data or record format (and ideas of fixed and padded records) are largely meaningless.

Major Features

Major Features

Simple, powerful, user interface

Complex commands are madefrom simple ones

Hierarchical file system

Consistent file format, the byte stream

Simple, consistent, peripheral interface

Hides machine architecture from user

Unix provides hundreds of commands each designed to do one thing well. Through a Unix shell (command line interpreter) collections of such commands are combined to perform complex tasks. In Unix, files on disk, devices and the input and output of running programs are considered files. All physical devices have filenames, and behave as ordinary files. The fundamental component of information in Unix is the byte stream. It allows files, devices and even programs to be used interchangeably as the source or destination of data; and thus allows the underlying machine architecture to be hidden from the user.

Unix Lore

Unix Lore

Intensely loyal defenders acclaim- extremely rich and powerful command set- clean and terse command syntax- ability to do things in beautiful and simple ways

Intensely critical detractors profane- terse syntax and strange command names- complex administration- unfriendly to novice users

Unix is an old operating system (though it has changed considerably from its earlier implementations). It started life as a support environment for a computer game, and graduated into a teaching and research platform for Universities. As a consequence, it has acquirednment, to others (especially novice users) the sheer size and complexity of the command set make it difficult to use.

Potted History

Potted History

1965 Bell,GEC and MIT start MULTICS1965 Bell,GEC and MIT start MULTICS

1982 Unix System IIIAT&T combined several variants

1982 Unix System IIIAT&T combined several variants

1983 UNIX System VSystem IV was an internal version

1983 UNIX System VSystem IV was an internal version

1981 BSD 4.11981 BSD 4.1

1973 Unix coded in C1973 Unix coded in C

1969 Space Travel conceivedUnix built on PDP-7

1969 Space Travel conceivedUnix built on PDP-7

Unix started life as the support environment for a Space Traveller game developed at Bell Laboratories. It was originally coded in PDP assembler, some parts were developed in B (an interpreted language influenced by BCPL) and later the entire system was recoded in C. C was developed by Dennis Ritchie for Unix to overcome the performance limitations of B. Unix was named by Brian Kernighan as a pun on an earlier system called ‘Multics’.Once Unix had become a stable product within Bell Laboratories, consideration was given to selling it. However, due to a Consent Decree Bell had signed with the US Federal government in 1956, it was forbidden to market computer products. Instead, Bell laboratories (specifically, the Unix Systems Group of AT&T) gave the system to Universities for educational purposes. Research at the University of California at Berkeley lead to the development of a variant of the Unix system. Most commercial flavours of Unix are based upon AT&T system V Unix, or BSD (Berkeley Software Distribution) 4.x Unix.

Potted History

1983 BSD 4.21983 BSD 4.2

1984 SunOS derived from BSD1984 SunOS derived from BSD

1985 BSD 4.31985 BSD 4.3

1991 Convergence on SVR41991 Convergence on SVR4

1989 SunOS 4 (Solaris I)1989 SunOS 4 (Solaris I)

1992 Solaris II (SVR4)1992 Solaris II (SVR4)

SunOS™ is Sun Microsystems flavour of Unix. It is based largely upon BSD Unix, with NFS and NIS network extensions to allow files to be shared and managed around a network.In the interest of standardisation, Sun Microsystems, together with a number of other major workstation manufacturers, have converged on SVR4 Unix. System V Release 4 incorporates many of the features of earlier System V versions of Unix, in addition to many other facilities. Sun’s SVR4 product is Solaris II, and all preceding SunOS products are now referred to as Solaris I.

Unix Flavours

Unix Flavours

Workstation UnixesSun Microsystems SunOS-4 (Solaris I) BSD + SVR3Sun Microsystems SunOS-5 (Solaris II) SVR4

Silicon Graphics IRIX 4 SVR4IBM AIX-3 SVR3 + ?Hewlett-Packard HP-UX - 8 SVR3 + BSD

Personal Computer UnixesSanta Cruz Operation SCO Unix V SVR4IBM AIX-1 SVR3

Interactive Unix SVR4Public Domain Linux SVR4

Mini and Super Computer Unixes

DEC Ultrix BSD + DECnetMeiko Meikos SVR3

The popularity of Unix is demonstrated by the variety of platforms on which it is available. From PCs to workstations, and mini-computers to super-computers, Unix provides a standard operating system.The various flavours of Unix descend either from AT&T Unix or from Berkeley Unix. AT&T System V has been particularly influential on the development of PC based Unix systems, whilst BSD Unix (Berkeley Software Distribution) has influenced mini and workstation Unixes.One of Unixes strengths is that it has been in the public domain for much of its life. Consequently, it has evolved to take advantage of new technology and has become an integral part of many embedded, control and general purpose systems. This strength is also a weakness, however, since it has allowed many diverse versions of Unix to evolve. Standards activities promoted by the X-Open Forum, the IEEE POSIX committees and the Open Software Foundation have helped to pull the systems together. Currently, most of the important versions of Unix are converging towards System V Release 4 (SVR4).

This page intentionally left blank

The Evolution of Unix

1965 MulticsA joint project started between AT&T Bell Laboratories, project MAC of MIT and General Electric. The aim was to develop MULTICS (MULTIplexed Information and Computing System) an interactive, multi-user, multi-access operating system.1969 Unix begins ...Bell Lab’s pull out of the MULTICS project because it failed to meet its design goals. Ken Thompson develops the initial basis for Unix on a discarded PDP-7, including a filesystem, basic process model and an interactive shell. The system was developed to support a Space Travel program, which simulates the motion of planetary bodies. The system was later named Unics and then Unix, as a pun on the original MULTICS project by Brian Kernighan.1971 Unix ported to PDP-11The first real project supported by Unix was to provide a text processing system for the patent department at Bell Laboratories. In order to achieve this, the system was ported to a PDP-11, providing 16K for the system, 8K for user programs, a disk of 512K bytes and a limit of 64K bytes per file.Ken Thompson, in an attempt to implement a FORTRAN compiler for the new system, came up with the language B. This was an interpreted language used in the development of some Unix tools. B was developed from BCPL, and named after the first initial.1972 C DevelopedInterpreted languages tend to realise poor performance. In an attempt to overcome the performance limitations of B, Dennis Ritchie developed the now popular language C. C was named from the second letter of BCPL. It provides for declaration of data types, definition of data structures, low level machine access and allows the direct generation of machine code.1973 Unix coded in CA major step in the acceptance of Unix was its recoding in C. This allowed it to be easily ported and extended, and lead to the formation of the Unix Systems Group to provide internal support within Bell Laboratories.1974 Unix distributed to universitiesAT&T were prohibited from marketing computer products because of a 1956 Consent Decree it had signed with the US government. However, it provided Unix to universities who requested it for educational purposes. AT&T neither advertised, marketed nor supported the system in accordance with the terms of the Consent Decree. As a consequence, the system was distributed with all of the source code. This version of Unix was known as 5th edition.


1975 Thompson attends UCBIn 1975 Ken Thompson took a sabbatical to the University of California at Berkeley (UCB). This marked the start of Unix development at UCB within the Computer Systems Research Group (CSRG). Unix software released from Berkeley is released in Berkeley Software Distributions; for example, 4.3 BSD.1977 Unix ported to non PDP machineBy 1977 the number of Unix sites had grown to about 500, of which about one quarterty of Unix began to grow, as marked by the first port of Unix to a non PDP machine and the establishment of the first Value Added Reseller (VAR) of a Unix system. The current version was the 6th edition.1978 7th Edition UnixThe first Unix system with portability as a specific goal was Unix Time-Sharing System, Seventh Edition (V7). The system ran on the PDP-11, the Interdata 8/32 and had a DEC VAX 11/780 port called Unix/32V. The VAX version provided 4Gb virtual address space.After the release of V7 the external distribution of Unix systems was turned over to the Unix Support Group (USG).1979 3 BSDThe first significant Berkeley Unix work was an enhanced version of VAX Unix/32V. In this system, known as 3 BSD, support for virtual memory, demand paging and page replacement was added. The large virtual memory of 3 BSD was motivated by large program development at Berkeley.1980 4 BSD4 BSD was produced between 1979 and 1980 for the American Defense Advanced Research Projects Agency (DARPA). An important goal of the projecttly general to allow communication among wide and local area networks.4 BSD was the operating system of choice for VAXes from when it first became available until the release of System III. Due to licensing limitations, organisations would buy a 32V licence from AT&T, and then order 4 BSD from Berkeley. Eventually, DEC produced ULTRIX, its own version of 4 BSD (which included support for DECs proprietary network DECNET) and sold this directly with VAX machines as an alternative to VMS.1981 /usr/group formedThe first independent organisation for the standardisation of Unix systems was introduced. Originally known as /usr/group, the organisation was later renamed UniForum.1982 Unix System IIIIn the period between 1977 and 1982 several variants of the AT&T Unix system were combined to produce Unix System III. The system included some features from BSD Unix, including terminal independent support for screen based applications (curses and termcap) and the C-shell.


1983 Unix System V & 4.2 BSDAT&T Bell Laboratories further develop Unix System III and call the new product System V (System IV was an internal evolutionary version). System V was released by the USG and for the first time official support for the system was announced.In addition, and of great significance to the future development of Unix, the US court-ordered divestiture of Bell Operating Companies from AT&T left AT&T in a position to market System V aggressively.The culmination of the original Berkeley DARPA Unix project was released as 4.2 BSD.Sun Microsystems released SunOS 1.0, a version of Unix derived from BSD 4.2. Sun’s reputation is based around networked workstations and its enhancements to BSD were to support this kind of environment. Sun introduced a window based user interface, called Sun Tools, and a distributed Network File System (NFS).1984 Unix System V Release 2After a transformation of the USG into the Unix System Development Laboratory (USL) Unix System V Release 2 (SVR2) was released. The system introduced paging and shared memory.Microsoft produced Xenix, the first significant, commercially licensed, derivative of AT&Ts Unix Version 7. Xenix was designed to run on PC architecture machines, but was also ported to larger machines. The system was eventually acquired by the Santa Cruz Operation (SCO).By this time there were in excess of 100000 Unix installations around the world running on machines from microprocessors to mainframes. No other operating system has ever been so prolific.The US ‘/usr/group’ published a document defining the Unix operating system. Although the report was too brief to provide a complete description of the system, it did form the basis of the future IEEE POSIX standardisation work.1985 System V definition and verificationAs several manufacturers bundled Unix with there systems, AT&T attempted to enforcee Definition (SVID) was produced to unambiguously define System V’s call interface, and the System V Verification Suite (SVVS) was produced to check that a Unix system was compliant.Microsoft released Xenix 3, an extension of their previous product based on System III and 4.2 BSD.1986 4.3 BSDBerkeley software distribution 4.3 was released to primarily fix bugs in the 4.2 BSD product. In particular, there were problems with the TCP protocol implementation and performance problems with new features such as symbolic links.The Santa Cruz Operation (SCO) took over distribution of Xenix from Microsoftily a European initiative to define a Common Applications Environment (CAE) independent of hardware platform. The group consists of a consortium of manufacturers, including AT&T, DEC, ICL and Sun.


1987 Unix System V Release 3The Unix System Development Laboratory was succeeded by AT&T Information Systems (ATTIS) which then distributed Unix System V Release 3 (SVR3) The system included the STREAMS generalised network interface, shared libraries and a mechanism for interprocess communication.IBM released the Advanced Interactive Executive (AIX-1) as an enhanced version of Unix for its 6150 PC-RT range of machines.The X-Open group produced the X-Open Portability Guide (XPG) to promote the portability of applications between diverse Unix implementations. The guide provides information relating to kernel and system interfaces, and utility programs.In late 1987 AT&T announced that it had purchased a sizeable portion of Sun Microsystems. Following this Sun announced that its next operating system would not be a further extension of SunOS (BSD derived) but would be based on Unix System V Release 4 (now known as Solaris II). This raised some degree of horror in other parts of the Unix community which felt that this special relationship between AT&T and Sun would threaten their market.1988 IEEE POSIX & OSFThe IEEE created the POSIX standards as an independent family of standards for software and hardware manufacturers. The POS in POSIX stands for Portable Operating System, the IX is borrowed from Unix. The standards are produced by committeeerating system interface functions, interactive shells, basic commands and real-time processing.Sun Microsystems released SunOS 4.0, a major version of the SunOS family. This was the last major release based on BSD, since SunOS 5 is derived from SVR4.In May 1988 the Open Software Foundation (OSF) was formed. This is an independent organisation for the further development and provision of computing environments, based on current standards and technologies. Perhaps more significantly for the members of OSF, the aim was to produce a standard Unix environment independent of AT&T licensing. The group of companies pulled together to make OSF included Apollo, DEC, Hewlett-Packard, Honeywell-Bull, MIPS, NCR, Silicon Graphics and IBM. The formation of OSF was the direct result of AT&Ts procurement of a stake in Sun Microsystems, and the consequential dangers to their market. At the time, it was noted that the OSF had the distinction of getting IBM and DEC into the same bed! As a consequenceun formed a counter-consortium called Unix International, dedicated to SVR4.The American National Standards Institute (ANSI) completed the standardisation process for C. However, since the committee was in progress for five years, most of the features of the standard were already supported by modern compilers.1989 OSF MotifThis year saw the growth of Unix International and the Open Software Foundation; companies and private members joined both. By the end of the year, OSF released its Motif user interface. The distinction between UI and OSF was, however, relatively small and even at this stage there was talk of a merger.


1990 AIX-3 for the RS/6000IBM released AIX-3 for their RS/6000 Unix workstation. AIX-3 is a radical rewrite of the earlier AIX-1 product. Whilst maintaining some conformance to its System V ancestor, the system includes a number of IBM specific extensions. 1991 HP-UX Version 8.0Hewlett-Packard released HP-UX version 8.0. The system is based on SVR3 with extensions from 4.2 BSD and 4.3 BSD. Emphasising standardisation, the system is compliant with SVID2, POSIX and relevant XPG interfaces.1992 Sun released Solaris IIThe first release of Solaris II was distributed by Sun’s software company, SunSoft. The product is also known as SunOS 5.0, and is Sun’s first implementation of SVR4 (and marks, therefore, Sun’s departure from BSD Unix).The wrangling between the OSF and UI decayed, and UI was disbanded. Moreover, Sun (formerly of UI) adopted OSFs Motif as its user interface for Solaris. OSF continued development of DCE, the Distributed Computing Environment. DCE represents a significant effort to link diverse computers into a single, productive network. DCE systems should be able to: support the creation of a single network filesystem (similar to NFS), make network services transparently accessible, balance computing load around a network, provide network management facilities, and provide a flexible network security mechanism.1993 Common Open Software EnvironmentA proposal by leading hardware and software companies for a common desktop environment was presented to X-Open for incorporation into the X-Open Portability Guide (XPG). The Common Open Software Environment (COSE) should provide a consistent computing experience, with a single set of programming interfaces, for HP, IBM, SCO, SunSoft and AT&T platforms.Sun continued work on DOE, the ‘Distributed Objects Everywhere’ project. DOE is indicative of many object-oriented operating system projects currently underway. In such systems, the entities that make up a distributed computing environment are considered to be autonomous objects, and the networked system is an amalgam of interacting objects.1994 SunSoft & NeXT ...SunSoft and NeXT announced a series of agreements designed to establish a comprehensive set of object-oriented application development standards. This is another significant thread in the development of distributed object-oriented environments, which is becoming the distinctive mark of the new generation of operating systems.

Key Concepts

Key Concepts

Great is the art of beginning, but greater is the art of ending

Lazurus Long

The Business of Computing

The Business of Computing

Computer programs process data

There are only two entities in Unix

inputdata

outputdataprocessing

files provide input and receive output

processes manipulate data to produce new data

The business of computer programs is to process data. To read data from some source, apply some computation, and generate result data. This model is supported directly by Unix.In Unix there are only two entities: files and processes. Files represent the data being read or written, processes are the active entities reading, processing and writing data.

Files & Processes

Files & Processes

Everything is a process or a file

Files are passive entities- streams of bytes stored on disks- interfaces to devices- the input and output streams of running programs

Processes are active entities instances of running programs

- instructions for the CPU

Every process starts life as a file- Unix commands are stored as program files on the disk

In Unix, everything is a process or a file. No other entities exist, not even disks, printers, terminals or networks---in Unix all of these things appear as files. The idea of using file names to represent devices saves introducing another concept. When a devices file is read or written, Unix ensures that the interactions are propagated to the particular device which the file represents.Files are passive entities, unable in themselves to do anything. Processes are active entities, in some sense they have life. Consider a human as a process, and a suitcase as data. The suitcase cannot move itself, because it does not have life. A human must be applied to the suitcase in order for it to move. Likewise a process is applied to a file in order for it to be processed.Unix is unusual compared with many operating systems in that process creation is relatively inexpensive. As a consequence, each command is executed as a single process. This differs from other systems which often run commands as procedure or function calls within a central command process.In Unix, each invocation of a command gives rise to a new process. The process is created from the command’s program file, executes and then dies. It is not unusual for one command to be executed by one or more users simultaneously, giving rise to many independent processes (instances) each performing the same task.Note that the instructions as to what a process should do are stored in a corresponding program file on disk. When a command is issued, the text of its program file is used to make the process. The CPU interprets each instruction within the process to carry out the work.

Organisation of Files

Organisation of Files

Files are organised as an inverted treeThe Unix File System consistsof a single root directory whichcontains files.

Files may be data, programsdevices or directories.

This logical file system may becomposed of many physical devices and networks.

Both process and file entities in Unix are organised as trees. The tree used to hold files is called the Unix File System (UFS). The tree used to hold processes is simply called the process tree.Unix File SystemThe Unix File System is organised as an inverted tree; the root is at the top, and branches and leaves in the form of directories and files grow down. A directory is a special file which can hold other files. Since these files may themselves be directories, a tree structure is formed.Unix systems contain only one logical file system. The file system may span multiple partitions and disks, cross networks and exist in multiple physical forms. However, the illusion maintained by the operating system, is that the file system is one, coherent, tree. As a user moves around the file system, Unix ensures that the physical joins between disks or networks remain hidden.Since there is only one file system, all users, programs and devices exist in the same name space. By convention, programs and devices exist in their own sub-directories, and users exist in their own sub-directories.

Organisation of Processes

Processes are organised into a process tree

Organisation of Processes

All processes have a parent process (save the first) and may have child processes.

Each command is executedas a new process, and is thechild of the process which invoked it.

Unix does not discriminate between the execution of system related programs and user programs. Both give rise to processes. The first process in a Unix system is called init and it runs as part of the procedure of making the machine ready for users. init creates child processes which setup the machine and ultimately prompt the user to login. Once a user has logged in, a new child process is created to enable the user to enter commands. This process is called a shell.The shell prompts the user to enter a command. For each command that the user enters, the shell spawns a new processes. The shell is therefore the parent of these processes. The shell is itself the child of init, since this gave rise to its creation when the user logged in. Many Unix commands also give rise to child processes, which may in turn give rise to new generations of processes. Thus a tree of processes is seen to have been spawned, starting from the great grand parent of all processes, init.

Login

Login

Multi-user OSs require users to login- validate username- associate with ‘account’

The login process establishes the user’s- initial process (shell)- initial directory (home directory)- Identity (User and Group)

Users view Unix through- interactive shells- custom menus- windowing environments- applications

In order to use a Unix machine users must login. In this process the system determines who the user is (the username), verifies this information (by requesting a password), and then associates with the user their file and process resources. Specifically, the user is associated with a sub-directory of the filesystem (their home directory) and an initial process (usually an interactive shell).Humans tend to use names to distinguish similar objects, machines tend to prefer numbers. During the login process the system associates a UID (user identity number corresponding to the username) and GID (group identity number corresponding to the user’s default group) with the user. This is subsequently used to label all files and processes created by the user.

Login File & Process

Login File & Process

Home Directory and Login Shell

init/

cat file

shhome

fredThe home directory isa sub-directory which contains files which belong to the user

The login shell is acommand line interpreter,invoking commands onbehalf of the user

Once a user has successfully logged-in to the system, the user is placed in their home directory and is given an initial shell from which they may invoke commands.The home directoryThe home directory is simply a sub-directory within the single, logical tree structure of the Unix file system. Since this structure may be composed of both remote and local disks, it is likely that the actual physical location of the home directory resides over the network on a carefully managed file server machine. Each user usually has their own home directory and uses this to store personal files, such as programs, letters and data. Home directories may themselves be broken down into sub-directories, to help organise the users files.The home directory is said to be the top of the users file tree. Note, however, the user may still go to any other part of the tree provided they have suitable access rights.The Login ShellThe login shell is a process executed on behalf of the user when they login. The command giving rise to the process may be any Unix command. Often it is an interactive shell, allowing users to enter their own command. Sometimes it is a specific application, locking the user within the application and hiding Unix itself.The login shell is said to be the top of the users process tree. The user may logout by terminating this process.

What are Users?

What are Users?

Users are the owners of files and processes

In Unix everything carries a UID and GID

process process

file

fred george

file

Users are not first class entities within Unix, only files and processes can claim this status. Users are simply attributes of files and processes. In Unix, every file and every processes must be owned by someone and exist in a group.When users first login their initial process (the shell) and file (their home directory) belongs to them. Every subsequent file or process created by a user is stamped with the users identity (UID) and the users current group identity (GID). The UID is usually unique and has a one-to-one mapping with the users username; the GID is shared by users working together. The GID provides a means by which users can gain joint access to shared files and commands.

Unix Structure

Unix Structure

kernel

cat

ls

sh

ksh

sort

vi

more

grep

Unix has a highly modular,layered, structure

Utilities are layered aroundthe kernel

The structure of the Unix system is different from most other large operating systems. At its centre is a core of functionality known as the kernel. This provides the system resources, such as the management of process and file trees.Around the kernel are layered the programs which make up a users perspective of the operating system. Hundreds of individual commands use the kernel in order to achieve their tasks. The commands physically reside in the filesystem as program files. However, once executed they are loaded into the systems memory and executed as processes.In the above picture, some of the standard commands are shown around the kernel. The commands marked sh and ksh are interactive user shells. They are usually automatically executed when a user logs into the machine and invoke commands on the user’s behalf. Its interesting to note, that the shells in themselves are no different from other programs. Although they form the layer between the user and the kernel, they reside in the filesystem and are executed as processes.

Kernel Responsibilities

System calls provide hooks into the kernel

Kernel Responsibilities

kernel

sh who

vi

The kernel is responsible for

process managementmemory managementfilesystem managementdevice management

The kernel is responsible for providing the resources needed by the programs wishing to run. There are primarily four resources which the kernel manages.Process ManagementUnix systems are multi-user, multi-process environments. This means that at any point in time there may be several processes running (between 15 and 60 is not unusual). Most of the hardware on which Unix is run only has one processor. The process management part of the kernel is responsible for scheduling each of the processes able to run onto the CPU. This is referred to as time-slicing or multi-tasking. Provided the kernel can switch from one process to another fast enough (called a context switch), the illusion to the user is that all of the processes are running simultaneously.File System ManagementThe notion of a tree structure is maintained by the kernel. It allows users to manage parts of the tree, hides the division between multiple disks and networks, and protects one users files from another.Memory & Device ManagementUnix employs a virtual memory management policy in which the amount of memory available to processes is greater than is physically available in RAM. This is achieved by swapping (or paging) processes from RAM to disk (the swap region on the disk) when the process is not needed, and swapping it back when it is.Device Management refers to the maintenance of the file interface to devices. When users interact with device files, the file like interactions are mapped by the kernel onto the corresponding device.

Getting Started: Login

Getting Started: LoginGetting Started: Login

“My voice is my passport, verify me”

V. Brandis

CDE Login Screen

CDE Login Screen

Menu of login options

Hostname of remote machine

CDE, the Common Desktop Environment, is a Graphical User Interface, or GUI, which is provided as standard across a number of different Unix platforms.

It was designed to provide end users with a consistent graphical user interface across workstations and PCs. The project was intended to:

• Reduce learning time by providing the same appearance and behavior across multiple operating systems.

• Increase productivity by helping system administrators and end users customize the desktop environment to fit individual work styles and preferences.

• Make learning easier by providing a consistent, rich, and easily accessible context-sensitive on-line help system for help whenever and wherever the user needs it.

• Provide a common set of desktop and application development tools. • Ease the porting of many existing X Window applications to a new

environment. Applications should be easy to run across many different platforms thus reducing the costs of moving to a new environment and helping to protect investments in software.

Core components of the environment include: • A login manager (shown in slide)

– A graphical login screen and manages user access to the system. • A file manager

– An on screen graphical file representation where users can directly manipulate icons associated with files to organize the file system and launch applications.

• An application manager

– The application manager is similar to the file manager except that it is intended to be a user specific list of files.

• A session manager – Users can easily customize their environment.

• The CDE window manager – The control mechanism for the visual user interface, or desktop, of a

session. – The CDE window manager includes a FrontPanel and a workspace

manager. • A user can manage all aspects of a session (except the initial

login) through objects on the FrontPanel. • An inter-application messaging system (similar to the Windows

Clipboard)– This aims to provide facilitates for the seamless interaction between

applications. • A desktop tool set

– A comprehensive set of productivity tools including multimedia-enabled mail, text editor, calendar, clock and icon editor, are provided with the CDE.

• Application development tools – A comprehensive set of development tools including debuggers,

application manager, application (Motif GUI) builder are provided with the CDE.

• Application integration components – Aplications written on any X Window system or toolkit should be

easy to integrate with the CDE tools provided.

Command Line Login Screen

Command Line Login Screen

Trying trinity.matrix.com...Connected to 10.1.5.1.Escape character is '^]'.

SunOS 5.9

login:

The CDE command line login screen is only accessible from the CDE login manager’s options menu, when using a local login session.

Command line login can be achieved remotely using tools such as telnet and ssh.

Usernames

Usernames

Should be at least 6 characters long

Only the first 8 characters are significant

Case sensitive as in all Unix

Spaces are not allowed

Control characters and punctuation are permitted Most people avoid them

Control-s and control-g are not permitted

Best login names contain both characters and digits

A login must exist on the system prior to attempting to log in as that user Only the Super-User may issue logins

Valid Usernames

Valid Usernames

Valid Entries Invalid Entries

cowboy the_black_marauder >8 characters

marauder john smith Contains space

johnny sarah^g Contains ctrl-g

johnnyQ sam Too short

cee3po

MyLoGiN

it’s_me

Best to choose a system and stick to it, e.g.• first name + first letter of family name• department code + initials + serial number• initials and year of entry

Think of a way of dealing with conflicting names before they occur

Good Passwords

Good Passwords

Must be at least 6 characters long

Only the first eight characters are significant

Case sensitive

Spaces, punctuation, and control characters are permitted (but not control-s and control-g)

Should be different than login name

Must differ from the old password

Before picking a password, it is important to consider security. A good password is easy for you to remember and hard for someone else to guess. "since the password protects your work and the entire system from unauthorized and careless users, and since it is relatively easy for other users (all throughout the networked world no less!) to move through Unix systems, you should carefully guard your password." Passwords should be at least six characters long. They can be up to eight characters long, including numbers and special characters. Using eight characters makes your password more secure. Try not to use a word in a dictionary or a proper noun because they are so easy to guess just by going through the dictionary. Personal information, like your social security number, phone number or birthday is also easy to guess (as you know from all those spy movies you have been seeing).In particular, do not use your username. Mix letters, numbers, and punctuation. Mix lower and upper case letters, too. A good way of getting a secure password is to think of a phrase and then take the initial letters of the words in the phrase, randomly capitalizing and adding punctuation. License plate combinations also work well, however, don’t use your own. Keep several passwords in mind, so you can change your password regularly (every few months or any time you think someone else knows it).

Passwords

Passwords

Any user may change their own password with the passwd command

Super-User cannot decode your password Though they can be brute force decoded using crack

If old password is forgotten, a new one must be issued

Passwords never appear on the screen

Keep in mind that your password is your own. No one else should know it. Do not share your account with other users by giving your password to them. Depending on the conditions under which your account was given to you, sharing it may be like sharing your driver's license, it is often grounds for revocation. Please do not do it. Keep your password secure. Do not keep it in a computer file, send it by email, write it on a post-it note or your white board.

Changing Passwords [passwd]

Changing Passwords [passwd]

System prompts for old password

System then prompts for new password

System then prompts for new again, to confirm

If passwords do not match, system prompt for new password again.

trinity$ passwdpasswd: Changing password for morpheusNew Password: passwordRe-enter new Password: passwordpasswd: password successfully changed for morpheus

Take care when setting your password. You will not be shown your password as you type it in.

The system will perform some basic checks on your password as defined by the operating system vendor when you enter it. You may be warned or prevented from changing your password, if your password does not meet these checks.

Login Failures

Login Failures

A login failure will generate a login error which must be acknowledged

If login does not exist Screen will be cleared and a new login screen

generated

If login does exist, but password is incorrect Only the password entry is cleared The cursor is positioned for a new password attempt.

After a certain number of tries, the entire screen is cleared

Why login fails

Why login fails

Incorrect username or password Frequently a login will look correct, but is not

- It may contain a leading space, which is difficult to see

Name service failure Name service process isn’t running Network failure between login machine and name service

provider

/etc/nologin File created by super-user to prevent non-root logins, e.g.

during maintenance

If login continues to fail after the second try Contact the Super-User.

If usernames and passwords are managed by a central networked name service, such as LDAP or NIS, it will not be possible to log in as a user to a machine that uses this service when it is disconnected from the network.

The super-user may also set a machine to refuse logins from ordinary users when maintenance work needs to be performed. This is done by creating an empty file in the /etc directory, called “nologin”.

Interruptions

Interruptions

To stop a command from executing

To stop things from scrolling

To restart things scrolling

To stop input and close a session

To erase the current line

Ctrl–c

Ctrl–s

Ctrl–q

Ctrl–d

Ctrl–u

Ctrl-c terminates a process. Ctrl-d asks it to stop running in a more graceful fashion.

Ending the sessions [logout]

Ending the sessions [logout]

$ logout

$ Ctrl-d (hold down the Ctrl key and D) Sends the End-Of-File Character

$ exitend the current shell

Click on the EXIT button on the Front Panel

/etc/passwd : home of identity

/etc/passwd : home of identity

/etc/passwd file

Generic Format

login the username x a place holder for the password, kept in /etc/shadow uid a unique number to identify each user gid a unique number to identify the user’s primary group gecos a comment field used to identify the user shell the program executed on completion of the login

root:x:0:0:root:/root:/bin/shdaemon:x:1:1:daemon:/usr/sbin:/bin/shbin:x:2:2:bin:/bin:/bin/shsys:x:3:3:sys:/dev:/bin/shsync:x:4:100:sync:/bin:/bin/syncbill:x:1002:100:Bill_Smith,,,:/home/bill:/bin/ksh

login:x:uid:gid:gecos:home directory:shell

NAME passwd - password file

SYNOPSIS /etc/passwd

DESCRIPTION The file /etc/passwd is a local source of information about users' accounts. The password file can be used in conjunction with other password sources, such as the NIS maps passwd.byname and passwd.bygid and the NIS+ table passwd. Programs use the getpwnam(3C) routines to access this information.

Each passwd entry is a single line of the form:

username:password:uid:gid:gcos-field:home-dir:login-shell

Password is generally substituted by an “x” to show that the actual password is found in the /etc/shadow file.

/etc/shadow : Password vault

/etc/shadow : Password vault

/etc/shadow file

Generic format

username The user's login name (UID) password A 13-character encrypted password for the user or NP lastchg A number of days indicating when the password was last modified. min The minimum number of days required between password changes max The maximum number of days the password is valid warn The number of days before expiry that the user is warned inactive The number of days of inactivity allowed for that user expire An absolute date specifying when the login may no longer be used flag Reserved for future use, set to zero. Currently not used

root:zoJMzm8KNESF6:6445::::::daemon:NP:6445::::::bin:NP:6445::::::bill:4sEixbwp7aoak:12925::::::

username:password:lastchg:min:max:warn:inactive:expire:flag

NAME shadow - shadow password file

DESCRIPTION /etc/shadow is an access-restricted ASCII system file that stores users' encrypted passwords and related information. The shadow file can be used in conjunction with other shadow sources, including the NIS maps passwd.byname and passwd.byuid and the NIS+ table passwd. Programs use the getspnam(3C) routines to access this information.

The fields for each user entry are separated by colons. Each user is separated from the next by a newline. Unlike the /etc/passwd file, /etc/shadow does not have general read permission.

Each entry in the shadow file has the form:

username:password:lastchg: min:max:warn: inactive:expire:flag

/etc/group: where groups live

/etc/group: where groups live

/etc/group file

Generic format

group : name of group x : password placeholder gid : a unique number to identify the group user-list : list of users belonging to the group

root:x:0:daemon:x:1:bin:x:2:sys:x:3:adm:x:4:tty:x:5:admin:x:1000:morpheus,tank,cypher,neo

group:x:gid:user-list

NAME group - group file

DESCRIPTION The group file is a local source of group information. The group file can be used in conjunction with other group sources, including the NIS maps group.byname and group.bygid and the NIS+ table group. Programs use the getgrnam(3C) routines to access this information.

The group file contains a one-line entry for each group recognized by the system, of the form:

groupname:password:gid:user-list

The password is generally replaced by an “x”

Summary

Summary

Usernames 6 characters or greater 8 characters are significant Created in /etc/passwd or name service

Group Primary group is in /etc/passwd or name service All groups are in /etc/group or name service

Login might fail Bad username/password /etc/nologin

name service failures.

File Hierarchy

File HierarchyFile Hierarchy

“Just because something doesn’t do what you planned it to do, doesn’t mean it’s useless”

T. Eddison

The Unix File system

The Unix File system

An organised way of storing files

The structure of the file system can be thought of as an inverted tree

Unix employs a tree structure to store files. Starting from an initial top-level directory (the root directory) sub-directories successively organise information into categories, and then sub-catagories. There are no limits on the depth to which the tree structure can grow. Unix differs from other hierarchical file stores (such as those provided in DOS and VMS) in that there is only one tree. The single tree structure hides multiple disks, partitions and even the network when NFS (the Network File System) is employed.

Hierarchical Structure

Hierarchical Structure

Directories are files which hold information on other files

Directories can be viewed as branches and files as leaves

Since directories are just other files, they can also be stored inside directories

The Unix file system is organised into a hierarchical tree structure in which directories are branches and files leaves. The purpose of directories is to group together related files. However, since files may themselves be directories, it follows that directories may contain sub-directories.

Directory Paths

Absolute path names start from root (/)

Relative path names start from the current directory

/

Directory Paths

mbox News report

format text style

letter book

note

A list of directories forms a path

/report/style/book

style/book

Path names describe routes through the file system. A relative path name is a route from the current working directory; an absolute path name is the route from the top of the file system.Absolute path names begin with a leading slash and are unique. Relatively path are not unique since they depend on the directory in which the path is specified.

File names

File names

Case sensitive

letters A-Z or a-z

numbers 0-9

underscore _

period .

comma ,

root /

Other characters allowed but discouraged

There are no limits to the length of a file name.

What is a file?

What is a file?

A file consists of an inode data

The inode contains information about the file file type, mode, ownership, creation date/time, etc. pointers to physical location of data on device

Data is stored across a physical device

A file does not know its name names are associated with inodes in directories

The file hierarchy is provided to offer an logical way of accessing data from physical media.

Files consist of an inode and the actual data. Each file has an inode.The inode contains information about the file. For Solaris, this is ususally:

the type of file (regular, directory, device, pipe, link, socket)the mode of the file (read, write and execute permissions)the number of hard links to the filethe user id of the file ownerthe group id of the group to which the file belongsdates and times of creation, last modification and last accesspointers to where the data is stored physically on the media

The inode does not contain the name of the file.Other operating systems and file systems put different information in their inodes.

Varieties of Files

Varieties of Files

Ordinary Files ASCII text or binary data, formatting left to user Subclass of ordinary is hidden (name starts with “.”)

Directory Directory provides a connection between the names

of the files and the files themselves. They impose a structure on the file system

Device Files Often located in the /dev directory Writing to the device file transfers data to the device.

A directory is a file whose data is a list of file names and the locations of the inodes that describe those files.

Device files are used to access hardware. They come in two types, block and character.

Traversing the File System : cd

Traversing the File System : cd

Every directory contains two special directory files

- “ . ” current directory- “ . . ” parent directory

trinity% cd ../style

trinity% pwd/report/style

trinity% cd ..trinity% pwd/report

trinity% cd –trinity$ pwd/report/style

trinity% cd /report/style

trinity% pwd/report/style

trinity% cd /report/texttrinity% pwd/report/text

trinity% cd .trinity% pwd/report/text

The directories “.” and “..” are convenient names for the current and parent directories. “..” allows path names to traverse back up through the tree.Use the cd command to ‘change directory’ and the pwd command to print the current working directory. Note that cd without any arguments takes the user to their home directory.

Examining Directory Contents : ls

Examining Directory Contents : ls

The contents of a directory can be listed using:

trinity% pwd/report

trinity% lsformat note style text

trinity% ls -Fformat* note style/ text/

trinity% ls -aF./ format* style/../ note text/

trinity% ls ..mbox Newsreport

trinity% ls /report/stylebook letter

trinity% ls -l-rwx------ 1 bill 100 Mar 21 20:01 format-rw------- 1 bill 873 Mar 21 17:59 notedrwx------ 2 bill 512 Mar 21 17:58 styledrwx------ 2 bill 512 Mar 21 17:58 text

trinity% ls /mbox News report

trinity% ls stylebook letter

trinity% ls -d stylestyle

trinity% ls -ld styledrwx------ 2 greg 512 Mar 21 17:58 style

ls [-aAcCdfFgilLqrRstu1] filename

The ls command is used to display the contents of the specified directory. It takes a variety of options that affect which files are displayed and the way their information is formatted.

-l long listing-a all files, including those beginning with ‘.’-g used with -l for group ownership-d the directory file not its contents-F show file type

Shell Wildcards

Shell Wildcards

Shell wildcards used for file name expansion

Note that the shell expands wildcards

trinity% ls p*pub pintplastered

trinity% ls /etc/*/m*/etc/adm/messages/etc/dp/modem/etc/openwin/modules

trinity% echo hello worldhello world

trinity% echo p*pub pint plastered

The Unix shell provides wildcard expansion to generate filenames for commands. To list all filenames that being with p (as above), then “*” is used to tell the shell to generate the filenames automatically. The shell searches the specified directory to find the files.

* matches zero or more characters? matches exactly one character[ABC] matches either A or B or C[A-Za-z] matches any single letter in upper or lower case

Note that wildcard expansion is different in Unix than for DOS. In DOS each individual utility interprets * and may associate a different meaning to the symbol. This is not possible in Unix, since the shell interprets the *, generates an argument list, and then calls the specified command. In the above, echo simple writes to the display its list of arguments; the list of arguments beginning with p were generated by the shell prior to invoking echo.

Building the File System : mkdir/rmdirBuilding the File System : mkdir/rmdir

mkdir creates directoriesrmdir removes them

trinity% rmdir bibliotrinity% ls -Fformat* note style/ text/

trinity% rmdir stylermdir: style: Directory not empty

trinity% pwd/report

trinity% ls -Fformat* note style/ text/

trinity% mkdir bibliotrinity% ls -Fbiblio/ note text/format* style/

mkdir [-p] dir1 [dir2 ...]rmdir [-ps] dir1 [dir2 ...]

The mkdir command is used to create new directories, and rmdir to remove directories. In keeping with most Unix commands, the commands may be supplied as many filename arguments as is required. In the following

mkdir one two three four five six /tmp/seven

six directories are created within the current directory, and a seventh is created beneath /tmp. Notice, however, that a minimum of one directory must be supplied to the command. Using the -p option, mkdir is able to create missing parent directories as needed

mkdir -p first/second/third

will create the missing parent directories first and second if they do not already exist.Note that it is not possible to remove a directory with rmdir if it contains other files. To remove the directory, first remove all the files and sub-directories which it contains. The powerful (and somewhat dangerous) rm -r command is useful for this.

Copying Files : cp

Copying Files : cp

cp copies files and directories around the filesystem

trinity% ls -Fmbox News/ report/

trinity% ls -F report/stylebook letter

trinity% cp report/style/book .trinity% ls -Fmbox News/ report/ book


trinity% cp -r report/style .trinity% ls -Fmbox News/ report/ style/

trinity% ls -F stylebook letter

cp [-ip] f1 f2cp [-ip] f1 f2 ... fn dcp -r [-ip] d1 d2

cp is used to copy files and directories around the filesystem. Note that copy means duplicating the bytes on disk representing the contents of the files being copied.cp is used with two arguments when copying from one file to another and with many arguments when copying a collection of files into a directory. In the case of the latter, the directory must exist and be the last argument. cp may also be used to copy the contents of one directory to another. In this case the -r (recursive) option must be supplied. When copying directories, if the target (d2) exists, then the source (d1) is created within it. A file f1 within d1 may now also be accessed as d2/d1/f1. If, however, the target does not exist, then it is created and the actual contents of d1 are copied into it. Therefore, a file f1 within d1, may now also be accessed as d2/f1.By default, the copy command overwrites any files which already exist with the target name. The -i (interactive) option may be used in order to get cp to prompt prior to overwriting any existing files.To preserve a file’s modification time and permission bits, use the -p option. If it is also necessary to preserve the file’s ownership, then the cpio command should be used.

Copying with Wildcards

Copying with Wildcards

Wildcards may be used when copying multiple files

Wildcards cannot be used for names which don’t exist

In DOS, the copy command does the expansion

trinity% cp *.c /home/george/src

trinity% lschapter1.txt chapter2.txttrinity% cp *.txt *.bak

copy *.c *.bak

Wildcards may be used with the cp command to generate a list of filename arguments to be copied. However, it is important to realise that the cp command is not involved in the expansion of the wildcard symbols. The wildcards are replaced by the shell with any matching filenames, prior to invoking the command. Since the shell generates the filename lists, based upon the files which currently exist in the directory, it is impossible for the shell to generate names which do not already exist. Therefore, the semantics of the DOS copy command do not apply to the Unix cp command.In the example given above, the user attempts to backup two chapters of a book, by copying them to new names with the extension .bak. In DOS this would be successful, in the older Unix systems it would be catastrophic. The shell would replace *.txt with the two files and *.bak with nothing. It would then invoke the cp command as followscp chapter1.txt chapter2.txtwhich is most unfortunate since the purpose of the activity was to backup not destroy the files!Unix is not wrong in its behaviour, it is in fact entirely consistent. The semantics of the DOS copy command are, however, somewhat strange. In DOS, the * in the first argument is used as a wildcard and the * in the second as a place holder indicating where to substitute the first part of the filename, and what to tack on the end.Fortunately, recent shells pass the * to the application if they are unable to perform an expansion and this gives rise to a cp error. The problem would be solved in Unix by copying the files to a backup sub-directory.

Moving Files : mv

Moving Files : mv

Files and sub-directories can be movedtrinity% ls -Fmbox News/ report/

trinity% ls -F report/stylebook letter

trinity% mv report/style/book .trinity% ls -Fmbox News/ report/ book

trinity% ls -F report/styleletter

trinty% ls -Fmbox News/ report/

trinity% mv report/style .trinity% ls -Fmbox News/ report/ style/

trinity% ls -F stylebook letter

mv [-i] f1 f2mv [-i] f1 f2 ... fn dmv [-i] d1 d2

mv is used to rename files and directories. It does not cause the contents of the file to be physically moved, only the file’s name is changed in its directory.The new name may be a path to another directory, so mv can in fact move a file from one place to another. Note that there is no need for a recursive option when moving a directory since files contained within the directory don’t care what it is called. More specifically, the contents of a directory file are the files stored within it, and mv does not effect file contents. The -i option may be used if there is a danger of overwriting existing files.

Deleting Files : rm

Deleting Files : rm

rm deletes files and directory structures

rm -i requests confirmation before deletion

trinity% ls -Fmbox News/ report/ booktrinity% rm booktrinity% ls -Fmbox News/ report/

trinity% ls -Fmbox News/ report/ style/trinity% rm -r styletrinity% ls -Fmbox News/ report/

trinity% rm -i bookrm: remove book? n

trinity% ls -Fmbox News/ report/ book

trinity% rm -i bookrm: remove book? y


The rm command deletes files and directories. Beware that in Unix a deleted file is lost forever. There is no mechanism to allow a file to be un-deleted since the disk space associated with the file may immediately be re-used by some other process. To recover a deleted file, the administrator must be asked to restore it from a system backup. It is unlikely that the restored file will contain recent changes made to the file.The rm command also has the -i and -r options provided with cp. -i allows interactive use of the command so that a use may stop the command from accidently deleting a file. The -r option is necessary if directory structures must be deleted.

Linking Files

Unix directories have multiple names

It is also possible to create multiple names for files

Linking Files

/

News report

. style

. ..

All directories have at leasttwo names, their name in the parent directory and ‘.’ inthemselves.

With each sub-directory, anew name is created for theparent, ‘..’

The Unix file system is held together through links. Each file is identified by a link name, or file name as it is usually called. Every file in the file system must have a name (a link), however it is possible (and often necessary) that some files have multiple links.The above example shows the multiple names associated with directories. All directories have at least two names, their name in the parent and ‘.’ in themselves. Should they have sub-directories, then a new name is generated for them in each sub-directory, ‘..’ The ‘.’ and ‘..’ directory links are created automatically when a new sub-directory is made. They are used as a short-hand notation to refer to the current or parent directories.

Linking Files : ln

Linking Files : ln

Use the ln command to create links

trinity% echo hello > afiletrinity% ls -l afile-rw-r--r-- 1 fred 6 Jan 14 23:16 afiletrinity% ln afile newNametrinity% ls -li afile newName

384927 -rw-r--r-- 2 fred 6 Jan 14 23:16 afile384927 -rw-r--r-- 2 fred 6 Jan 14 23:16 newName

trinity% cat newNamehello

ln [-s] f1 f2

The ln command enables users to create their own links.

In the above example, the names newFile and afile are linked to the same file. The fact that afile existed first is not relevant, both are equal. Changes made to the file through the name afile would be the same if made through the file name newFile. The two names refer to exactly the same area on disk.

Hard links : ln

Hard links : ln

By default, ln creates hard links A hard link is simply another directory entry pointing

at the inode of the file or directory- Hard links cannot be made across different partitions

or physical disks- It does not matter if the “original” file or directory is

deleted, as this simply removes one of the hard links to the file

$ ln original new

The number of hard links to a file is recorded in the inode of the file, and can be seen with ls.

trinity$ ls -li /usr/bin/vi 401851 -r-xr-xr-x 5 root bin 227828 Jun 19 2002 /usr/bin/vi

The number of hard links to /usr/bin/vi is “5”.

ex is one of the commands that is a hard link to the vi command

trinity$ ls -li /usr/bin/ex 401851 -r-xr-xr-x 5 root bin 227828 Jun 19 2002 /usr/bin/ex

You can see that the inode number of the two files is the same.

When the number of hard links to a file or directory is “0”, then there are no longer any directory entries for the file, so it ceases to be available, i.e. it has been deleted.

Symbolic links : ln

Symbolic links : ln

With the -s option, ln creates symbolic links A symbolic link is a file that contains as its data, the

absolute path to another file- Symbolic links, unlike hard links, can cross file system

boundaries - The link will break if the file or directory to which it

points is removed

$ ln –s original new

In general, it is preferable to use symbolic links when creating links. This is partially because they can cross over different partitions and physical disks, but also because they are easier to see in listings.

Symbolic links can be seen using the ls command. Symbolic links will show their file name and then the location to which they are linked.

e.g.trinity$ ls -li bin 4 lrwxrwxrwx 1 root root 9 May 13 15:06 bin -> ./usr/bintrinity$ ls -dli /usr/bin 401792 drwxr-xr-x 4 root bin 9728 May 13 15:38 /usr/bin

Note that since links are files in themselves, they will have their own inode number, separate from that of the thing they point at.

Why create links?

Why create links?

Commands can be given different names to indicate different behaviour The vi editor has multiple names to provide different

functionality depending on how it is started

Files may be linked to appear in multiple directories A file can be linked to appear in several places in the

file hierarchy and can be edited from any of them, however, the data appears the same in all places.

Backwards compatibility Allows users to use an old name for a renamed file

It is sometimes useful for files to have multiple names.

For example, the standard Unix editor, vi, has several names. The different names for vi include ed, view and vipw. The purpose of the multiple names is to give the illusion that there are many programs providing vi like editing facilities rather than simply one.When invoked, the vi program checks which name was used to invoke it and changes its behaviour accordingly. This saves users having to remember large number of command line options

A useful application of links is to give the illusion that a file exists in multiple directories. This is achieved by specifying path names with the ln command. In the event that the directories physically exist on different partitions, the -s option (for symbolic) must be used to establish the link.

Pathname abbreviations

Pathname abbreviations

$HOME and ~ The home or login directory of the user

~username The home or login directory of the user username

$PWD and . The present working directory

.. The parent directory

“ ~” is only available in ksh, csh and bash

Shell Meta-characters

Shell Meta-characters

Characters with special meanings to the shelle.g. * ? [ ] ` < > | ! ~ $ %

The dot “.”, underscore “_” and hyphen “-” are not meta-characters

It is advisable not to use these meta-characters in directory or file names

Meta-characters have special meanings to the shell.

However, as there are a limited number of characters available, they may also have special meanings to other programs.

You will need to be aware of what is interpreting the character to be sure of what meaning is assigned to it.

* - Asterisk

* - Asterisk

“*” represents zero or more of any character

trinity% lsdante dir dir2 dir3 eat fruit fruit2 zilogtrinity% ls d*dante dir

dir2:beans coffee nuts

dir3:mango peach peartrinity%

?- Question Mark

?- Question Mark

“?” represents any single character

trinity% lsdante dir dir2 dir3 eat fruit fruit2 zilogtrinity% ls dir?

dir2:beans coffee nuts


[ ] – Square Brackets

[ ] – Square Brackets

[] defines an occurrence of a range

trinity% lsdante dir dir2 dir3 eat fruit fruit2 zilogtrinity% ls [e-z]*eat fruit fruit2 zilogtrinity% ls dir[0-5]dir2:beans coffee nuts


Ranges are matched using ASCII

Each character is translated to the equivalent ASCII value and then the match is performed mathematically.

Using decimal ACSII values, the range [e-z] would be translated as any character with an ASCII value between 101 and 122, i.e. 101 < character value < 122

It is not possible to specify a range of [z-e] as this would translate to a mathematical match of “less than 122 and greater than 101” which is not possible to represent in one mathematical expression.

` ` - Backticks

` ` - Backticks

` ` - backticks (or graves) force the command enclosed in them to be executed and its output substituted into command before the remainder of the line is evaluated

trinity$ rm `cat obsolete_files.txt`

trinity$ YEAR=`date +%Y`trinity$ echo $YEAR2005

trinity$ echo the hostname is ‘uname -n‘the host name is trinity

Backticks are particularly useful in shell programming.

Quoting in Shells [ ” ’ ]

Quoting in Shells [ ” ’ ]

Use single quotation marks to identify text literally

Use double quotation marks to identify text literally, while enabling variable and command expansion

trinity$ echo ’the path is $PATH’the path is $PATH

trinity$ echo ”the path is $PATH”the path is /bin:/sbin:/usr/ucbbin

Why quote?We have seen that the shell has meta-characters which have special meanings. We have also seen that some of these characters have meanings to commands. We can use quoting to force meta-characters to be dealt with in the way we expect, and by the command we expect.

Escaping in shells [ \ ]

Escaping in shells [ \ ]

\ - The back-slash escapes the the meaning of the following character so that it the shell does not interpret it, if it normally would

trintiy$ echo \”the path is \$PATH\””the path is $PATH”

The backslash - \ - allows single characters to be escaped so that they are not interpreted by the shell. This can be particularly useful for escaping quotes or single characters that would still be interpreted inside quotes.

Getting Help

Getting Help

“Good information is hard to get. Doing anything with it is even harder !”

L. Skywalker

Getting help : man

Getting help : man

man displays sections of the Unix manual

You must know the name of the command you want help with

Displays the manual pages on the command “head”

trinity% man head

trinity%

The standard way for getting help in Unix is the man command.

Getting help : man

If you do not know quite what the command is

provides a list of command which are related to print

apropos does the same

Both assume the System Administrator has builtwhatis database

trinity$ man –k print

trinity$ apropos print

trinity$ catman -w

The standard way of using man requires that you know the name of what you want help about.

Getting help : man

To move around the man page, use the following commands Spacebar moves you a single screen full b moves back a single screen f moves forward a single screen q leave the manual page display h lists all the scrolling facilities /string searches forward for string n finds the next occurrence of string

man : Output

man : Output

Header line

The section of the manual the command belongs to

Name

The name of the command and and an outline

Synopsis

The command and its options and parameters

Description A brief description of what the command does

User Commands head(1)

head - display first few lines of files

head [ -number | -n number ] [ filename... ]

man : Output

Options A detail explanation of each of the options

supported by the command

Operands A list of parameters, such as files, this command can

manipulate

Usage How the command is used and what problems are

know about it

Examples A simple example

man : Output

Environment Any environmental variables which the application can

or does use.

Exit Status What error levels or exit status are given when the

program has run[very useful when writing scripts]

Attributes Attributes of the command

See Also commands associated with this one

man : example

man : example

User Commands head(1)

NAMEhead - display first few lines of files

SYNOPSIShead [-number | -n number] [filename...]

DESCRIPTIONThe head utility copies the first number of lines of eachfilename to the standard output. If no filename is given,head copies lines from the standard input. The default valueof number is 10 lines.

When more than one file is specified, the start of each filewill look like:

==> filename <==

Thus, a common way to display a set of short files, identi-fying each one, is:

example% head -9999 filename1 filename2 ...

man : example

OPTIONSThe following options are supported:

-n numberThe first number lines of each input file will becopied to standard output. The number option-argumentmust be a positive decimal integer.

-numberThe number argument is a positive decimal integer withthe same effect as the -n number option.

If no options are specified, head will act as if -n 10hadbeen specified.

OPERANDSThe following operand is supported:

file A path name of an input file. If no file operands arespecified, the standard input will be used.

man : example

USAGESee largefile(5) for the description of the behavior of headwhen encountering files greater than or equal to 2 Gbyte ( 2

**31 bytes).

EXAMPLESExample 1: Writing the first ten lines of all files

To write the first ten lines of all files (except those witha leading period) in the directory:

example% head *

ENVIRONMENT VARIABLESSee environ(5) for descriptions of the following environmentvariables that affect the execution of head: LANG, LC_ALL,

variables that affect the execution of head: LANG, LC_ALL,LC_CTYPE, LC_MESSAGES, and NLSPATH.

man : example

EXIT STATUSThe following exit values are returned:

0 Successful completion.

>0 An error occurred.

ATTRIBUTESSee attributes(5) for descriptions of the following attri-butes:

____________________________________________________________| ATTRIBUTE TYPE | ATTRIBUTE VALUE ||_____________________________|_____________________________|| Availability | SUNWcsu ||_____________________________|_____________________________|| CSI | enabled ||_____________________________|_____________________________|| Interface Stability | Standard ||_____________________________|_____________________________|

SEE ALSOcat(1), more(1), pg(1), tail(1), attributes(5), environ(5),largefile(5), standards(5)

man page sections

man page sections

1 User Commands

2 System Calls

3 C library functions

4 File Formats

5 Standards, Environments and Macros

6 Games and Demos

7 Device and Network Interfaces

8 System Administration

9 Driver Entry points

Some sections have modifiers which group similar pages.e.g.1M System maintenance and administration commands3LIB Libraries implemented a shared object

manual entries in other sections

manual entries in other sections

The SEE ALSO part at the bottom of a man page lists alternate references.

If the command is followed by a number other than 1 in parentheses, it indicates a section of the man pages

To view this section, use

This will display the format of the password file

passwd(4)

trinity$ man –s 4 passwd

Other useful man options

Other useful man options

Most sections of the manual have an introduction

If a keyword appears in multiple sections of the manual, these can be listed

man -s# Intro

man -l keyword

trinity$ man -l passwdpasswd (1) -M /usr/share/manpasswd (4) -M /usr/share/man

Introductory pages explain what the section of the manual contains, any subsections and often list the pages within the section.

Google

Google

Help can often be found on-line through search engines such as Google

Searches can be restricted using limiters site

filetype

Logical constructs can be used to improve results + forces the inclusion of common words

- forces the exclusion of a word

OR either one search term OR a different search term

http://www.google.com/

Google: Example

Google: Example

SunSolve

SunSolve

SunSolve is Sun's official online help source

Available to the general Sun Community:- Security Information- Resolved Sun Alerts- Patch Descriptions (a.k.a. Patch ReadMe documents)- Archived SunSolve content- Limited Access to the SunSolve Knowledgebase- Sun Support Forums- Big Admin

Extras available to Service Plan or Contract customers:- Full Access to the SunSolve Knowledgebase

http://sunsolve.sun.com/

File Types and Listing

File Types and Listing

“The human animal differs from the lesser primates in his passion for lists of ‘Ten Best’.”

H. Allen Smith

Varieties of Files

Varieties of Files

Ordinary Files ASCII text or binary data, formatting left to user Subclass of ordinary is hidden (name starts with “.”)

Directory Directory provides a connection between the names

of the files and the files themselves. They impose a structure on the file system

Device Files Often located in the /dev directory Writing to the device file transfers data to the device.

A directory is a file whose data is a list of file names and the locations of the inodes that describe those files.

Device files are used to access hardware. They come in two types, block and character.

What sort of file [file]

What sort of file [file]

$ file filename classifies the named files several tests performed on file

- if plain text, tries to determine programming language used

- if binary, compares the “magic number”, if available, with contents of /etc/magic to determine type

trinity$ file dantedante: English text

trinity$ file /usr/bin/cat/usr/bin/cat: ELF 32-bit MSB executable SPARCVersion 1, dynamically linked,stripped

The file command is used to provide information about the contents of the file passed as the parameter.It performs a number of tests on the data in the file, including whether the file starts with a particular “magic” number.The file /etc/magic contains a plain text listing of these magic numbers and what they mean about the file.

Strings in a file [strings]

Strings in a file [strings]

$ strings filename Show any ASCII strings within a file

trinity$ strings /usr/bin/catSUNW_OST_OSCMDusvtebnusage: cat [ -usvtebn ] [-|file] ...cat: Cannot stat stdoutcat: cannot open %scat: cannot stat %scat: input/output files `%s' identicalcat: close error

The strings command can be used to find ASCII strings within a binary file.This can be useful for finding error messages in binary files, or identifying random binaries.

Contents of a file [cat]

Contents of a file [cat]

$ cat filename Displays the contents of the files to stdout, usually, the screen

trinity$ cat danteHalf way along the road we have to go,I found myself obscured in a great forest,Bewildered, and I knew I had lost the way.

It is hard to say just what the forest was like,How wild and rough it was, how overpowering;Even to remember it makes me afraid....

The cat command displays the contents of a file to the standard output file handle. This is usually the screen.

If given more than one file name, cat will display the files sequentially. This can be used to join, or concatenate, files.

Listing contents of files [more]

Listing contents of files [more]

$ more filename Display the contents of a file a small quantity at a time

to stdout

<cr> moves you forward a single linef moves you forward a single screenspace moves you forward a single screen fullq leave the page display immediatelyh lists all the scrolling facilities/string searches forward for stringn finds the next occurrence of string

The more command is one of a group of commands known as pagers, i.e. they display data a page at a time.An alternative to more is the less command.

Listing contents of files [less]

Listing contents of files [less]

$ less filename Display the contents of a file a small quantity at a

time to stdout less is the opposite of more

Allows backwards movement through file Doesn’t read whole of file before displaying

- Useful for large files

NAME less - opposite of more

SYNOPSIS less -? less --help less -V less --version less [-[+]aBcCdeEfgGiImMnNqQrsSuUVwX] [-b bufs] [-h lines] [-j line] [-k keyfile] [-{oO} logfile] [-p pattern] [-P prompt] [-t tag] [-T tagsfile] [-x tab] [-y lines] [-[z] lines] [+[+]cmd] [--] [filename]... (See the OPTIONS section for alternate option syntax with long option names.)

DESCRIPTION Less is a program similar to more (1), but which allows backward movement in the file as well as forward movement. Also, less does not have to read the entire input file before starting, so with large input files it starts up faster than text editors like vi (1).

Top few lines [head]

Top few lines [head]

$ head [-#] filename Display the first 10 lines of a file by default to stdout [-#] change the number of lines displayed

trinity$ head –3 danteHalf way along the road we have to go,I found myself obscured in a great forest,Bewildered, and I knew I had lost the way.

NAME head - display first few lines of files

SYNOPSIS head [-number | -n number] [filename...]

DESCRIPTION The head utility copies the first number of lines of each filename to the standard output. If no filename is given, head copies lines from the standard input. The default value of number is 10 lines.

Bottom few lines [tail]

Bottom few lines [tail]

$ tail [-#] [-f] filename Display the last 10 lines of filename by default to

stdout [-#] change the number of lines [-f] forever loop displaying end of file

trinity$ tail –3 danteThat are in the heavens, through a round opening;

And then we emerged to see the stars again

NAME tail - deliver the last part of a file

DESCRIPTION The tail utility copies the named file to the standard output beginning at a designated place. If no file is named, the standard input is used.

Fields in a file [cut]

Fields in a file [cut]

$ cut -b list [-n] [file...]$ cut -c list [file...]$ cut -f list [-d delim] [-s] [file...] -b list refers to bytes

-c list refers to characters

-f list refers to fields

list comma or space separated list of numbers indicating the required items

[-n] do not split multi-byte characters

[-d delim] field delimiter character, default is tab

[-s] suppress lines with no delimiter characters

NAME cut - cut out selected fields of each line of a file

SYNOPSIS cut -b list [-n] [file...] cut -c list [file...] cut -f list [-d delim] [-s] [file...]

DESCRIPTION Use the cut utility to cut out columns from a table or fields from each line of a file; in data base parlance, it implements the projection of a relation. The fields as specified by list can be fixed length, that is, character positions as on a punched card (-c option) or the length can vary from line to line and be marked with a field delimiter character like <TAB> (-f option). cut can be used as a filter.

Counting things [wc]

Counting things [wc]

$ wc [-c] [-l] [-w] filename Counts contents of a file [-c] characters [-l] lines [-w] words.

trinity$ wc dante33 223 1320 dante

trinity$ wc -l dante33

NAME wc - display a count of lines, words and characters in a file

SYNOPSIS wc [-c | -m | -C] [-lw] [file...]

DESCRIPTION The wc utility reads one or more input files and, by default, writes the number of newline characters, words and bytes contained in each input file to the standard output.

The utility also writes a total count for all named files, if more than one input file is specified.

wc considers a word to be a non-zero-length string of characters delimited by white space (for example, SPACE, TAB). See iswspace(3C) or isspace(3C).

Finding the differences [diff]

Finding the differences [diff]

$ diff [-i] [-c] file1 files2 Compares contents of two files [-i] case insensitive comparison [-c] position contextual differences

trinity$ diff /etc/hosts hosts20c20< 192.168.1.2 tiny--->

NAME diff - compare two files

SYNOPSIS diff [-bitw] [-c | -e | -f | -h | -n | -u] file1 file2 diff [-bitw] [-C number | -U number] file1 file2 diff [-bitw] [-D string] file1 file2 diff [-bitw] [-c | -e | -f | -h | -n | -u] [-l] [-r] [-s] [-S name] directory1 directory2

DESCRIPTION The diff utility will compare the contents of file1 and file2 and write to standard output a list of changes necessary to convert file1 into file2. This list should be minimal. Except in rare circumstances, diff finds a smallest sufficient set of file differences. No output will be produced if the files are identical.

Differences of 3 files [diff3]

Differences of 3 files [diff3]

$ diff3 file1 file2 file3 Displays the differences between 3 files

trinity$ diff3 /etc/hosts /etc/inet/hosts hosts====1:20c192.168.1.2 tiny2:20c

3:20c192.1.6.1 unplugged

NAME diff3 - 3-way differential file comparison

SYNOPSIS diff3 [-exEX3] filename1 filename2 filename3

DESCRIPTION diff3 compares three versions of a file, and publishes disagreeing ranges of text

Sorting files and output [sort]

Sorting files and output [sort]

$ sort [OPTIONS] files Ordering options

- -d dictionary order- -f fold lower case into upper case

i.e. a follows A- -n numerically- -M compares months

i.e. Jan before Feb before Mar- -r reverses order

Use +n to sort on a particular column, where n is the column number

multiple columns may be sorted at the same time- ordering options can be specified for each column

sorts the file numerically by the sixth column and in reverse order by the third column

sort +6n +3r file

Sort can be used to sort multiple files into one sorted file.

Sort is often used as part of pipelines to sort the output of one command before it is input into another.

Handling duplicate lines [uniq]

Handling duplicate lines [uniq]

$ uniq [input_file] [output_file] Reports or filters repeated lines in a file Only works if lines are adjacent

- Run sort first on input

trinity$ cat file26465trinity$ sort -n file | uniq2456

NAME uniq - report or filter out repeated lines in a file

SYNOPSIS uniq [-c | -d | -u] [-f fields] [-s char] [ input_file [output_file]]

uniq [-c | -d | -u] [-n] [ + m] [ input_file [output_file]]

DESCRIPTION The uniq utility will read an input file comparing adjacent lines, and write one copy of each input line on the output. The second and succeeding copies of repeated adjacent input lines will not be written.

Repeated lines in the input will not be detected if they are not adjacent.

Redirection

Redirection

“The trail's got to be 'round here somewhere!”

D. Boone

Why ?

Why ?

For useful work to be performed data must be transformed Most work requires data to be transformed in a

number of different ways

Unix commands designed to do one task very well

Redirection allows commands to be joined together in an efficient way

Redirection is an important part of the Unix philosophy.

Commands are designed to do one job extremely well. However, most real tasks require more than one transformation to be performed on the input data in order to get out the required information.

Standard Files

Standard Files

stdin aka file descriptor 0 This file normally be referenced by scanf(); Typically the keyboard

stdout aka file descriptor 1 This file normally referenced by printf(); Typically the display

stderr aka file descriptor 2 This file to which run time errors are sent; Typically the display

Every process is assigned three files on creation; stdin, stdout and stderr.

These files are used to accept input, display output and display errors.

Redirection : < > 2>

Redirection : < > 2>

stdin, stdout, and stderr may be redirected use file descriptors and the “<” and “>” symbols

Run the command but get input from the fi le file instead of the keyboard

Run the command but put the output in the file file.outinstead of on the display

Run the command and put any errors produced from the command into errors.lst

$ command < file

$ command > file.out

$ command 2> errors.lst

stdin is redirected using the “less than” symbol - <This allows commands to receive input from files, rather than the keyboard

stdout is redirected using the “greater than” symbol - >This allows the output produced by commands to be sent to a file, rather than the display

stderr is redirected using its file descriptor “2” and the “greater than” symbol - 2>This allows any errors output by the running command to be sent to a file, rather than the display.This can be particularly useful in scripts.

Redirection : >>

Redirection : >>

> Causes a new fi le to be created each time the redirection occurs

>> Causes the text to be appended to an existing file or creates a

new file if one does not exist

Creates a new fi le or overwrites fi le called names

Adds Smith:Bubba:Seymour to the end of the file names

trinity$ echo ”Smith:Bubba:Seymour” >>names

trinity$ echo ”Smith:Bubba:Seymour” >names

Single redirection arrows are used to create new files. If the file already exists, it is overwritten by the new output.

Double redirection arrows are used to append data to files. If the file does not exist, it will be created. However, if the file already exists, the output will be added to the end of the file.

Redirection : Merging

Redirection : Merging

Sometimes useful to merge stderr and stdout for scripts, where we want to capture the output and

any errors in the same file

Merging is simply a redirection of stderr to stdout stdout is referenced by its file descriptor (i.e. 1)

2>&1

Redirection: Multiple redirections

Redirection: Multiple redirections

Multiple redirections can be used with a single command

Error output merged with standard output Standard output to file called report Input from file called monthly_stats

Order is important with multiple redirections Read the redirections from right to left

Only out.a will contain the usage message from the cpcommand

trinity$ command < monthly_stats > report 2>&1

cp 2>&1 > out.bcp > out.a 2>&1

A very powerful way of using redirection, is to use more than one redirection with a single command.

It is important to note that redirections are performed in a particular order, and should be read from right to left.

Thus, in the example shown:cp > out.a 2>&1This reads as: redirect stderr to stdout (i.e. merge stderr and stdout) then redirect stdout to the file out.a

cp 2>&1 > out.bThis reads as: redirect stdout to out.b, then redirect stderr to stdout.However, as stdout has already been redirected, it is not possible to redirect stderr to stdout, so the usage message from cp will go to the display.

Redirection : Examples

Redirection : Examples

Takes the output from cat, i.e. the contents of the file dante, and places it in a new file newdante- Effectively, this is a replacement for the cp

command

Will append onto the end of the /etc/passwd file a new username bin- This is a quick way to add new users to the

system

trinity$ cat dante > newdante

trinity$ echo bin:x:2:2:bin:/bin:/bin/sh >> /etc/passwd

More redirection: File creation

More redirection: File creation

Creates a file myworld and stores “Hello World”

Creates a file myfile by entering text via the keyboard until the Ctrl-D is entered when the file is closed and saved

trinity$ cat > myfile“And if anyone knows anything about anything,”said Bear to himself, “it’s Owl who knows some-thing about something,” he said, “or my name’snot Winnie-the-Pooh,” he said. “Which it is,” he added. “So there you are.”Ctrl-D

trinity$ echo Hello World > myworld

Another useful method of file creation using redirection is called the “Here Document”.This is a specialised case of the second example shown, where text read from the following lines until an end of file marker (typically the string EOF, but may be anything) is entered.

cat > newfile <<EOF“And if anyone knows anything about anything,”said Bear to himself, “it’s Owl who knows some-thing about something,” he said, “or my name’snot Winnie-the-Pooh,” he said. “Which it is,” he added. “So there you are.”EOF

As in the example in the slide, a file called new file would be created containing the data, however, this method is useful for scripts where one may wish to create long files, e.g. containing configuration data, without using multiple echo statements

Pipelines : |

Pipelines : |

Pipes are command line FIFO buffers redirects the stdout of one file to the stdin of

another the pipe is used to supply input to another command redirection will direct output to other than a

command

The output of command1 is used as the input of command2

$ command1 | command 2

Pipelines : examples

Pipelines : examples

Lists the current directory and searches for the string morpheusfind files named morpheus in current directory

Lists the mail servers belonging to the domain matrix.net, searches for the number 5 and returns the seventh field from any matching line, using spaces as the field delimiter

trinity$ ls . | grep morpheus

trinity$ host -t MX matrix.net | grep 5 | cut -d" " -f7

It can be seen that very complex transformations can be built using pipelines.

Named Pipes

Named Pipes

Named pipes, or FIFOs, are used to facilitate communications between a sending process and a receiving process.

$ mknod name p$ mkfifo [-m mode] path Both commands will create a named pipe, either

called name or located at path

A FIFO is a queue where the first item in, is the first item out.

Named pipes can be used instead of temporary files in scripts where it is necessary to pass data between processes.

File Security

File Security

“Anyone who uses the phrase ‘as easy as taking candy from a baby’, has never tried taking candy from a baby”

R. Hood

Security Overview

Security Overview

The primary function of a system's security feature is to deny access to unauthorized users

Unix’s primary security features User passwords File and directory protection with permissions Files that control remote logins and commands

on individual workstations Logs and audit files for System Administrators

to check for unauthorized usage

The requirement for security comes directly from the multi-user, networked nature of Unix.

By requiring users to log into the system, we can restrict access to systems to only those people who need to access them. It also helps provide an audit trail as to who has done what on a system. This is an important tool for administrators and is a very good reason why you shouldn’t allow other people to use your account.

Since the system has a concept of identity, security can be further refined, by assigning ownership to files, then defining how this affects a user’s ability to read, write or execute the file.

Permissions

Permissions

Unix has two default levels of security Username and passwords to access the workstations File access permissions

Files and directories are automatically protected by default system permissions when they are created

Usernames and passwords limit access to a system and provide identity once logged in.

File and directory permissions provide access control within the file hierarchy, limiting what users can see and do on a system.

Finding the permissions [ls –l]

Finding the permissions [ls –l]

File type - Kind of file- d for Directory - l for Symbolic Links - - for ordinary fi les- p for named pipes

User (owner)- The user who created the file or directory

Group- Class of users defined by the system administrator

Others (public)- All other users

trinity$ ls –l thisfile-rwxrw---x morpheus users 34342 Jan 14 1999 thisfile

The mode of the file details its type and the permissions of the file, as stored in the inode. It is listed in the first 10 characters of the output of the ls command when the -l parameter is used.

The first character may be one of the following:d The entry is a directory.D The entry is a door.l The entry is a symbolic link.b The entry is a block special file.c The entry is a character special file.p The entry is a FIFO (or "named pipe") special file.s The entry is an AF_UNIX address family socket.- The entry is an ordinary file.

The remain nine characters should be examined as three groups of three and show the permissions granted to the user who owns the file, the group of users the file is associated with, and any other user who does not fall into one or other of the previous categories.

Types of users

Types of users

User Owner of a file may manipulate a file in any manner, except

transferring ownership

Only the super-user may change the user who owns a file

Group A list of users, as in a class, all of whom have equivalent

permissions

To find who is in a group, look in the /etc/group file

Others Public with a login on the system may access the file

The super-user, root, is a special case, and ignores all permissions on all files. This can be very dangerous.

Access Permissions

Access Permissions

Access permissions affect files and directories in different ways

readable Files: May be viewed internally by user with proper

status Directory: Contents may be listed

writable Files: May be overwritten by a user with proper status Directory: May have new files added and removed

executable Files: May be executed by user with proper status Directory: Allows directory to be traversed by users or

processes

Directories are files whose data consists of a list of file names and pointers to the inodes of those files.If a user has read access on a directory, then they may list the contents of that directory, i.e. the data of the file is visible.If a user has write access on a directory, they they may create and delete files in the directory, i.e. modify the data in the directory file, by adding or removing links between file names and inodes.If a user has execute permissions on a directory, they are allowed to “enter” the directory or pass through it.

How is access decided ?

How is access decided ?

If the UID of the user is that of the Super-User, ignore all permissions and grant full access

If the process UID and the file UID match Use the User Permissions bits

If the process GID and the file GID match Use the GID Permissions bits

If neither UIDs nor GIDs match Use the Other permissions

For users other than root, access is processed using the most restrictive match first.

Controlling access [chmod]

Controlling access [chmod]

$ chmod [-R] mode file Sets the access rights on file [-R] recursive mode contains those access permissions

- may be symbolic or octal

trinity$ chmod o=rwx newfile trinity$ chmod –R 755 newdir

As with rm, it can be dangerous to use chmod recursively.

Controlling access [chmod]: The modesControlling access [chmod]: The modes

File access modes can be specified in two ways Symbolic

Octal- group of three numbers- position specifies who permissions is defined for

- U## : user, #G# : group, ##O : others- value specifies permission

- 4 : read, 2 : write, 1 : execute- multiple permissions may be set by summing values

- remove

x executeo other

+ add= setHow

w writer readWhata allg groupu userWho

When using symbolic modes, a is used to set a permission for all of the user, group and others. If who the permission is for is omitted, then a is assumed.

For example, to specify that a file may be read and modified by its owner, and the group, but only read by anyone else use the following modes

Symbolic: chmod ug=rw,o=r myfile or chmod +r,ug+wOctal: chmod 664 myfile

It is often easier to set permissions initially with an octal mode and then make changes using the symbolic notation.

Octal Modes

Octal Modes

001010100execute

002020200write

004040400read

othergroupuser

Octal Modes : example

-rwx rw- --x root other 34342 Jan 14 1999 thisfile

Octal Modes : example

0+0+1=14+2+0=64+2+1=7othergroupuser

=761100011111xwrxwrxwr

In this example, we can see that the file has been given the following permissionsUser (owner): may read, modify and execute the fileGroup: may read and modify the fileOther: may execute the file

The octal mode which matches this permission set is 761

User ID and Group ID

User ID and Group ID

Every user has a unique number that identifies files and processes belonging to them The number is known as a UID (user ID)

UIDs are mapped to names in the /etc/passwd file

Every group is also identified by a unique number The number is known as a GID (group ID)

GIDs are mapped to names in the /etc/group file

A user's primary group often has the same name as the user and the UID and GID are usually the same.

UIDs and GIDs can also be mapped to names using naming services, such as LDAP or NIS. These are commonly used in large networked environments so that users and groups are provided consistently across the entire network but can be managed from a central location.

Real & Effective User/Group ID

Real & Effective User/Group ID

real user and group id from /etc/passwd

effective initialised as same but can be changed

real user/group is who is actually running

effective is for determining permissions

Why would you want them to be different? Non-privileged users accessing privileged information

We have seen that ordinary users do not have access to the shadow file where passwords are stored.How then can a user change their own password?

Who am I [id]

Who am I [id]

$id [user] Display the UID, GIDs and corresponding user names

and group names for the current, or specified, user If real and effective UID and GID are not the same,

both will be displayed

trinity$ iduid=115(guest) gid=10(staff)

The id command is particularly useful for finding the effective UID and GID of a user.

Ownership

Ownership

All files have a UID and GID associated with them determining the ownership

All Unix processes have a UID and GID associated with them

UIDs and GIDs are mapped to names by the /etc/passwd and /etc/group files or naming services, such as LDAP or NIS

trinity$ ls –an thisfile-rwxrw---x 1002 100 34342 Jan 14 1999 thisfile

trinity$ iduid=1002(morpheus) gid=100(users)

trinity$ ps -ef | grep sshroot 315 1 0 10:28:25 ? 0:00 /usr/lib/ssh/sshdroot 327 315 0 10:28:48 ? 0:01 /usr/lib/ssh/sshd

By default, processes are created with the UID and GID of the owner and group owner of the executed file.

Changing Ownership: chown and chgrpChanging Ownership: chown and chgrp

$ chown [-fhR] owner [ : group] file$ chgrp [-fhR] group file [-f] forces changes, no errors reported [-h] if the file is a link, change the ownership of the link,

not the referenced file [-R] make changes recursively owner UID or name of user to own file group GID of name of group to own file

Only the super-user may change the owner of a filetrinity$ ls –l drwxr-xr-x 2 root other 117 Apr 19 13:04 newdir-rw-r--r-- 1 root other 0 Apr 19 13:04 newfile

trinity$ chown morpheus newfile ; chgrp users newdirtrinity$ ls –ldrwxr-xr-x 2 root users 117 Apr 19 13:09 newdir-rw-r--r-- 1 morpheus other 0 Apr 19 13:09 newfile

chown can be used to change the ownership and group ownership of a file. Generally, it is only used by the super user.chgrp can be used to change the group ownership of a file.

As with rm, it can be dangerous to use chown or chgrp recursively.

Standard users may only have one real user id, but may be members of many groups.The super-user is not restricted by file permissions or ownership.

setuid

setuid

Program file owned by one user but run by another

Program runs as though by file owner

root owns the file, so allow root to run program on your behalf (e.g. change your password)

Routine in program can now make setuid call

chmod +s filename

-r-sr-sr-x 1 root sys 21964 Apr 7 2002 /usr/bin/passwd

passwd command runs as root, no matter which user invokes it. This allows it access to the shadow file, which is not accessible by non-root users.

setuid should be used with care on root owned files as it removes some of the protection granted by ownership and access permissions.

setgid

setgid

setgid operates in a similar fashion to setuid

When program is run it takes the permissions of the program's group rather than the group(s) of the user

chmod g+s filename

If a program has the has the set-group-ID bit set, then when any user runs the program it will be run with all the access privileges of the program's group rather than the group(s) of the user running the program. A program can give up this privilege after starting (which is often done for security reasons).

setuid and setgid with directories

setuid and setgid with directories

Directories can also have the setuid or setgid bits set

In this case, files created in the directory will be owned by the user who owns the directory, or be owned by the group the directory belongs to, or both rather than being given the UID or GID of the

creating user

If a directory has these bits set, everything created under that directory will be owned by the directory's user or in the group of the directory, regardless of who creates the file.

Sticky bit

Sticky bit

On a directory Restricts who may rename or delete files in directory Used on /tmp and /var/mail to prevent users

deleting files which belong to other users

On a file Used to prevent swapping out of executable No longer used on modern operating systems

chmod +t file

ls -ld /tmp /var/maildrwxrwxrwt 5 root sys 312 May 15 16:57 /tmpdrwxrwxrwt 3 root mail 512 May 13 15:06 /var/mail

ls displays a set sticky bit as a "t" in the permissions of a file or directoryIf a directory is writable and has the sticky bit set, files within that directory can be removed or renamed only if one or more of the following is true: o the user owns the file o the user owns the directory o the file is writable by the user o the user is a privileged user

/tmp is owned by root and

If set for an executable, the sticky bit tells the kernel to keep the code loaded in swap space even after it has finished executing on the assumption that it is likely to be used again soon. This performance optimisation was included in some early versions of Unix to save reloading frequently used programs such as the shell or vi from disk.

Default Permissions [umask]


$ umask [mode] Displays or sets the umask, default file creation value [mode] contains a 3 byte octal value

The operating system sets a default umask Users can override this umask in their .profile

trinity$ umask022 trinity$ umask 027trinity$ umask027

The usual default umask is 022.This means that files are created readable and writable by their owners and only readable to the group and the rest of the world.Owners of directories created with this umask may list the files in the directory and create and delete new files. Users in the group and other users may only list the files, change into the directory or traverse through it.


Default object creation permissions are set by the current umask

umask shows the permissions NOT granted on the file or directory For file creations, subtract the umask value from

666. For directory creations, subtract the umask value

from 777trinity$ umask022trinity$ touch newfile; mkdir newdirtrinity$ ls –ldrwxr-xr-x 2 morpheus users 117 Apr 18 10:43 newdir-rw-r--r-- 1 morpheus users 0 Apr 18 10:43 newfile

The maximum permissions that can be set on a file are all of the user, group and other having read and write permissions. The octal mode which corresponds with this permission set is 666.Files are not created executable, by default, for security reasons. Most files will never be executed.

The maximum permissions that can be set on a directory are all of the user, group and other having read, write and execute permissions. The octal mode which corresponds with this permission set is 777.Directories are created executable, by default, so that they may be traversed.

Searching and Translating

Searching and Translating

“Were you looking for me?” Dr. Livingstone

Why?

Why?

Useful work transforms data

This is often achieved using some form of translation

It is useful to be able to find data So that we can transform the right thing So that we only transform what is necessary

find : Finding files

find : Finding files

$find path condition Descends the file system tree beginning at path

locating files which matches the conditionspecified in condition

Finds a file whose name is fruit in this directory and any below and displays the name of that file

$find . –name fruit

find : Conditions

find : Conditions

$ find path condition [-user uid] file whose owner is uid

[-group gid] file whose group is gid

[-mtime [+|-] n] file modified in the last n days [-atime [+|-] n] file accessed in the last n days [-perm nnn] file with permissions matching

octal nnn

[-inum n] file with i-node number of n

[-size [+|-] n] file of size bigger or smaller than n blocks

[-exec command {}\;] run command on each match with the current match replacing {} in the command to run.

Block size is dependant on file system, however, it is often 512bytes

Numeric user and group ids and user and group names can both be used with the -user and -group conditions

find : Examples

find : Examples

Find all files who share the same i-nodei.e hard links

Find all core files and remove themi.e. remove all crash dumps

Find all files larger than 400 blocks

$find / –inum 767

$ find / –name core.0 –exec rm {}\;

$ find / –size +400

We have previously seen that we can find all files whose listing contains the string “morpheus” with by using grep on the output of ls -lSimilar, but more refined, searches can be performed using find.e.g.To find all files in the current working directory owned by “morpheus” use:find . -user morpheus

To find all files in the current working directory whose group ownership is morpheusfind . -group morpheus

To find all files called “morpheus”find . -name morpheus

Regular Expressions

Regular Expressions

Regular Expressions - “regexp” - are a powerful method for finding things A Regular Expression is pattern of characters used to

match against the same characters in a search. They usually include meta-characters, which

represent things other than themselves, to refine the search.

Some of the most powerful UNIX utilities, such as grep, sed and awk, use regular expressions

The command evaluates text against the pattern to determine if the text and the pattern matchIf they match, the expression is true

Regular Expressions : Meta-charactersRegular Expressions : Meta-characters

Positional meta-characters

anchors match to end of line$

anchors match to start of line^

escapes following character\

exactly one of the preceding pattern?

1 or more of the preceding pattern+

0 or more of the preceding pattern*

Modifying meta-characters

a range of characters to exclude[^ ]

a range of characters[ ]

any character, except newline.

Character patterns

Regular Expressions are built up from text and meta-characters.Common meta-characters and their meanings are listed in the slide.Not all meta-characters can be used with all commands that use regular expressions.

The backslash - \ - character works in two ways.If a character has a special meaning, then preceding it with a backslash will cause it to be matched as the literal character.i.e. to match a literal dot, the Regular Expression would include \.

The backslash can also be used to give a special meaning to an otherwise normal character.

Regular Expressions: Character ClassesRegular Expressions: Character Classes

Printable characters (includes whitespace)[:print:]

Hexedecimal digits[:xdigit:]Uppercase characters[:upper:]Whitespace characters[:space:]

Punctuation characters[:punct:]

Lowercase characters[:lower:]Printable and visible (non-space) characters[:graph:]Numeric characters[:digit:]Control characters[:cntrl:]

Space and tab characters[:blank:]Alphabetic characters[:alpha:]Printable characters (including whitespace)[:almun:]Matching CharactersClass

These character classes are defined by the POSIX standard and may or may not be available in commands which implement Regular Expressions.

Regular Expressions : Two warnings

Regular Expressions : Two warnings

Regular Expressions use similar meta-characters to the shell However, not all have the same meaning in both

cases.

The shell will expand bare meta-characters it recognises before passing them to the command To be sure your Regular Expression is interpreted in

the way you expect, it is best to quote it- .* will be expanded by the shell to a list of all

hidden files in the current working directory- “.*” is a Regular Expression meaning 0 or more

of any character except newline

Regular Expressions : Examples

Regular Expressions : Examples

Finds the pattern of a dollar sign preceded by at least a space

Finds the pattern of a dollar sign preceded by at least one character

Finds the pattern of a line which starts with the word “The”

Finds the pattern of a line which ends with the word “End”

” \$”

”.\$”

”^The”

”End$”

Translate [tr]

Translate [tr]

Translates the characters in set1 to set2- Each input character found in the range specified by set1 is replaced by the character in the same relative position in the range specified by set2.

tr [set1] [set2]

trinity$ cat dante | tr [a-z] [A-Z]HALF WAY ALONG THE ROAD WE HAVE TO GO,I FOUND MYSELF OBSCURED IN A GREAT FOREST,BEWILDERED, AND I KNEW I HAD LOST THE WAY.

IT IS HARD TO SAY JUST WHAT THE FOREST WAS LIKE,HOW WILD AND ROUGH IT WAS, HOW OVERPOWERING;EVEN TO REMEMBER IT MAKES ME AFRAID....

The tr command takes two ranges and replaces every character in the input that matches the characters in the first range, by every character in the same relative position in the second range.

The POSIX character classes mentioned previously, are particularly useful with the tr command.

For example, the translation shown in the slide, could be rewritten as followstrinity$ cat dante | tr [:lower:] [:upper:]

Finding things [grep]

Finding things [grep]

$ grep [-i] [-v] [-c] pattern filename Global Regular Expression Pattern matcher

finds pattern in filename

[-i] ignore case

[-v] match everything but pattern

[-c] counts matches

Finds the pattern “root” in /etc/passwd

Counts how many warning messages were logged on May 13

trinity$ grep root /etc/passwdroot:x:0:0:root:/root:/bin/sh

trinity$ grep "^May 13" /var/adm/messages | grep -c warning3

The grep command is particularly useful for finding simple strings in files or output.

Stream Editor [sed]

Stream Editor [sed]

$ sed [option] file Editor for files [option] take a look at the man page Common usage is to substitute a pattern for another pattern

Substitutes all occurrences of “Truman” with “Clinton” in infileand send the changes to stdout

trinity$ cat infileTruman was a Democrat President of the United States of America.trinity$ sed -e 's/Truman/Clinton/g' infileClinton was a Democrat President of the United States of America.

sed -e ‘s/pattern1/pattern2/flags’ file

sed : Substitution Examples


sed works on a file one line at a time

By default, only the first match on a line is substituted

substitutes the first occurrance of yes on a line with the string no

To match all occurrences of a pattern, use the g flag

substitutes all occurrances of yes

s/yes/no/g

s/yes/no/

Another useful flag that can be used with substitutions is:n a number (between 1 and 512) indicating that the replacement should occur for only the nth occurrence of the pattern


Leaning Toothpick Syndrome

often occurs when searching for unix paths

can be avoided by using an alternative pattern delimiter, for example “:”

if the delimiter stil l occurs in the pattern it must be escaped

Delimiters occur three times in each substitution statement Once to start the expression, once between the two patterns

and at the end to close the expression

s/\/usr\/bin\/ksh/\/usr\/bin\/sh/

s:/usr/bin/ksh:/usr/bin/sh:

The name Leaning Toothpick Syndrome (LTS) was coined by Larry Wall, the creator of the perl programming language, which makes heavy use of Regular Expressions in its syntax.


Deleting characters

finds all occurrences of the ranges and subsitutes them with nothing, i.e. deletes them

[a-z]- any lower case alphabetic character

[^a-zA-Z0-9\-]- any character that isn’t an upper or lower case alphabetic,

numeric or hyphen character- removes whitespace- hyphens have a special meaning within ranges, so need to

escape the hyphen

s/[a-z]//gs/[^a-zA-Z0-9\-]//g


Substrings  used to create a substring pattern \n references the nth matched substring

where n is a single digit

Matches a line and adds “dnl ” to the front of the line

Matches a line and removes “dnl ” from the front of the line

s/^$DAEMON_OPTIONS.*$/dnl \1/s/dnl $define(\`SMART_HOST$/\1/

sed

sed

sed is an extremely powerful tool for non-interactive editing particularly useful in scripting

Common usage is to substitute strings in text files

By default, sed outputs to stdout can test substitutions without changing file on disk

Handy one liners for sed can be found here: http://sed.sourceforge.net/sed1line.txt

Who is using the system ? [who]

Who is using the system ? [who]

$ who [-H] [-q] [ am i] [-r] Display who is currently logged into the system [-H] put a header line on the display [-q] just list the users [ am i] tell me who I am [Real User-Id] [-r] shows the current run level

trinity$ who -H NAME LINE TIMEmorpheus console May 24 10:17 trinity pts/4

May 24 17:36 (trinity.matrix.com)trinity$ who am imorpheus

trinity$ who -r. run-level 3 May 15 10:28 3 0 S

The who command is useful for the administrator to find out which users are logged into a system and from where.The who am i version of the command is useful for finding the real UID of the current user

Quicker who [w]

Quicker who [w]

$ w [-h] [username]

Display who is currently logged in [-h] suppress the header line [ username ] just show username

trinity$ w 12:52pm up 59 day(s), 20:57, 1 user, load average: 0.00, 0.00, 0.01User tty login@ idle JCPU PCPU whatroot pts/0 12:52pm w

w is similar to the who command

Find a spelling [look]

Find a spelling [look]

$ look word Checks the system dictionary for a word

- /usr/share/lib/dict/words

trinity$ look fredFredFreddieFreddyFredericFredericaFrederickfrederikfredricite

look is a dedicated command for searching the system dictionary which is usually located in /usr/share/lib/dict/wordsThe system dictionary is a plain text file, so is also searchable with grep

Processes and Control

Processes and Control

“Careful Planning is the key to safe and swift travel”

Ulysses

Process Overview

Process Overview

All tasks within Unix begin as processes

Processes are running files

Process Identification Numbers (PIDs) are used to keep track of separate processes

All processes have a UID and a GID associated with them

One process may have many Light Weight Processes, or threads, associated with it

Generating Processes


Parent process runs fork() to generate an identical copy of itself except it is given a new process ID

If fork() succeeds child process inherits parent’s environment space STDIN, STDOUT and STDERR redirected to parent

fork() returns value of child PID to parent uses exec() to run command when command finishes, returns ONLY the exit status of

command ($?) to parent

If fork() fails returns exit status of -1 to parent


CHILD

STDIN STDOUT

STDERR

ENV

fork()

successChild PID

CHILD

STDIN STDOUT

STDERR

ENV

exec()

PARENT

STDIN STDOUT

STDERR

ENV

status code ($?)

fork()

PARENT

STDIN STDOUT

STDERR

ENV

Child NOT created

Out of memory

Out of process slots

failure(i.e. -1)

Incorrect permissions

Understanding Processes

Understanding Processes

The cd command must be built into the shell

Why? cd changes directory

- i.e. modifies the $PWD environment variable If cd were an external command, it would follow the fork()/exec() model- child processes cannot change the environment space

of the parent- child processes can only return numeric exit status of

command exec()-ed

The current working directory is defined by the PWD environment variable.To change directory, this value held by this variable must be changed.If the cd command were implemented as a command external to the shell, what would happen?When the command was invoked, a new process would be created by fork() with a copy of the parent’s (i.e. the shell’s) environment, which would include the PWD variable.The code to change the PWD variable would then be exec()-ed in the child process, and then the command would end.When the child process ends, it returns only the exit status code of the process that has been run. So, if the variable had been changed successfully, the parent shell would receive 0.Has this affected the PWD variable in the parent process? No. Child processes cannot affect their parent’s environment.So, have we changed directory? No.Thus the cd command must be an internal process to the shell.

Displaying the Processes [ps]

Displaying the Processes [ps]

$ ps[-e] [-f] [-u username] Displays the processes running on a host [-e] every process [-f] full listing [-u username] list the processes with UID of

username

$ ps –ef | more UID PID PPID C STIME TTY TIME CMDroot 0 0 80 16:46:40 ? 00:01 schedroot 1 0 80 16:46:40 ? 00:40 /etc/init -root 2 0 27 16:46:40 ? 00:00 pageoutroot 3 0 80 16:46:40 ? 04:33 fsflushmorph 126 1 80 18:13:12 console 00:01 –ksh---More---

The ps command prints information about active processes. Without options, ps prints information about processes that have the same effective user ID and the same controlling terminal as the invoker. The output contains only the process ID, terminal identifier, cumulative execution time, and the command name.Otherwise, the information that is displayed is controlled by the options.

Displaying the Processes [prstat]

Displaying the Processes [prstat]

$ prstat [OPTIONS]

By default, shows an iterating list of processes sorted by CPU usage

[OPTIONS] can be found in the man page and define alternative methods of displaying and sorting processes

trinity$ prstat -a PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP509 root 4568K 4344K cpu7 59 0 0:00:00 0.0% prstat/1504 root 328K 256K sleep 59 0 0:00:00 0.0% sh/1508 root 2576K 1888K sleep 59 0 0:00:00 0.0% bash/1502 root 1872K 1288K sleep 59 0 0:00:00 0.0% in.telnetd/162 root 2784K 1952K sleep 59 0 0:00:00 0.0% picld/426 root 1120K 376K sleep 59 0 0:00:00 0.0% dhcpagent/1

NPROC USERNAME SIZE RSS MEMORY TIME CPU40 root 87M 55M 7.6% 0:00:04 0.1%1 smmsp 4312K 1416K 0.2% 0:00:00 0.0%

Total: 42 processes, 104 lwps, load averages: 0.00, 0.00, 0.01

prstat is a useful way to see which commands are using the most CPU. This can be particularly useful if a machine appears loaded unexpectedly.It is also useful to see which users are using a machine most intensively.prstat has many options which are listed in the man pageThe -a option shown in the slide, shows user information at the same time as process information

prstat : column headings


Column headings read as follows PID process ID username username of process owner size total virtual memory size of process in kilobytes

(K), megabytes (M) or gigabytes (G)Includes the heap and stack assigned to the process

RSS resident set size, as above without the heap and stack

state current state of the process- cpun - running on cpun- sleep - process is sleeping- run - process is runnable- zombie - process is terminated but parent has not waited

for return value- stop - process is stopped

A zombie process is a process that has completed execution but the exit status of which has not been read by the parent process yet. A process becomes a zombie process when it issues the exit subroutine and the following circumstances occur: Its parent process is not running a wait subroutine and has not notified the operating system that it does not intend to wait for its children to finish. The it is said that process has died, but has not yet been reaped.


pri process priority- most user processes run with priorities between 0 and

59- OS processes run with priorities between 60 and 99- real time processes run with priorities between 100

and 129 (these are extremely rare) nice a modification value for process priority time time running on CPU cpu percentage of CPU used process name of process NLWP number of Light Weight Processes

(threads) associated with process NPROC number of processes associated with a

user

Nice values range from 0 to 39 and are used to bias the priority of a process. A higher number results in a lower priority.There are commands to change nice values, however, users can only affect the nice value of processes which they own. They also cannot start processes with nice values less than 20, nor can they lower the nice values of their processes after they've raised them.

Managing Processes

Managing Processes

Commands executing take over the current display until they complete This is called foreground processing

In general, we want the ability to have many commands running at once without needing a separate console for each one

Command executed in the background leave the current display free to run others jobs

Processes exist in one of three states: background, foreground, or stopped.By default, all processes run in the foreground. However, it is possible to start processes in the background or change processes from foreground to background processing.

Background a command [ & ]

Background a command [ & ]

$ command [options] [parameters] & Execute the command with its options and parameters in the

background

The command will be placed in background for execution, and a new command may be issued immediately on return of the system prompt whether or not the first command has completed

3342 is the process-id of the job.

When it is finished, [DONE] 3342 will be displayed

trinity$ find . –name ”*.gif” &[1] 3342 &trinity$

A command run with a trailing ampersand (&) will be executed in the background.It will be assigned a job number, separate from its process ID and both the job number and process ID will be printed to the display.When the command has finished running a done message and the process ID are printed to the display on which it was started.If the controlling terminal is killed, all running jobs in that terminal are also killed.The terminal is the parent process for the jobs. When a parent process is killed, all its children are also killed.

nohup

nohup

$ nohup command [&] When a command is invoked by the nohup command

it will ignore hup signals In practice, this means that the command is not

killed if the parent terminal dies nohup can also be applied to processes after they

have been started using the -p PID option

trinity $ nohup find / -name binSending output to nohup.out

nohup is a simple shell command that tells the program it is told to run not to exit when the controlling terminal is killed.It is particularly useful for leaving long running processes in the background without having to keep a shell open - i.e. keeping them running in case you need to, or accidentally, log out (or if the windows computer on which you've logged in freezes and needs to be restarted.) Example:nohup long_running_command & The '&' tells the shell to not only nohup the command, but to also run it in the background, letting you continue using the terminal as usual.By default, nohup sends all output (both stdout and stderr) to a file called nohup.out in the current working directory. However, if this directory is not writable, the file will be created in the user’s home directory. This file can be very useful for examining if there are problems with the job.

Listing the jobs [jobs]

Listing the jobs [jobs]

$ jobs[-l] Displays the processes which were placed in the

background from this shell [ -l ] display the process ID and not just the

job-id

trinity$ jobs[1] + Running find . –name ”*.gif” &

jobs will only show the jobs started in the terminal in which it is run. This can be a problem if you have started jobs in multiple terminals or have used nohup and closed the controlling terminal.

Foreground a command [fg]

Foreground a command [fg]

$ fg %job-id Moves the process running with job-id from the

background to the foreground

trinity$ jobs[1] + Running find . –name ”*.gif” &trinity$ fg %1

Processes running in the background may also be stopped with stop %job-idStopped jobs may be restarted with bg %job-id

Killing processes [kill]

Killing processes [kill]

$ kill [-signal] [%job-id | PID] Ends a processes specified by either job-id or PID [-signal] how badly you want it to end

[see man –s 3HEAD signal ]

Ends a processes giving it it chance to close files

Kill a process with extreme prejudice. Just end.

$ jobs[1] + Running find . –name ”*.gif” &$ kill %1[1] + Terminated find . –name ”*.gif”

$ kill –9 %1[1] + Terminated find . –name ”*.gif”

The default signal sent by kill is 15, SIGTERM.

Useful signals include:Name Value DefaultEventSIGHUP 1 Exit Hangup - causes some programs to re-read their configuration filesSIGKILL 9 Exit KilledSIGTERM 15 Exit Terminated - exits gracefully

One sometimes sees programs fail with the following signals:SIGSEGV 11 Core Segmentation Fault - program dies and produced a core dumpSIGPIPE 13 Exit Broken Pipe - a command in a pipeline failed, breaking the pipeline

Job Management Commands

Job Management Commands

Suspend the current job, ready to be sent into the background

Ctrl-Z

Abort the current jobCtrl-C

Abort a jobkill %n

Background a jobbg %n

Foreground a jobfg %n

Display which jobs are current runningjobs

FunctionCommand

Job management commands only show or operate on jobs started in the terminal in which they are run.It is not possible to background a process in one terminal and then foreground it in another.

/proc

/proc

Special filesystem

Contains information about all running processes memory segments (as)

current working directory (cwd)

light weight processes (lwp)

ps and prstat are ways of displaying some of the contents of the /proc filesystem

Most files within /proc are plain text and can be read by the user, however, there is generally no explanation of the format

/proc is a file system that provides access to the state of each process and light-weight process (thread) in the system. The name of each entry in the /proc directory is a number corresponding to a process-ID. These entries are themselves subdirectories. Access to process state is provided by additional files contained within each subdirectory.The owner of each /proc file and subdirectory is determined by the user-ID of the process.

vi: Visual Editor

vi: Visual Editor

“Tell you what. Let me sweeten the deal for you a bit…”

Beelzebub

The Visual Editor

The Visual Editor

A screen oriented text editor with many enhancements to greatly speed up the entry of program source files

Well adapted to and created for the C language

At best, very powerful but very cryptic in use of commands

At worst, an abomination from the sons of dogs

Often, the only editor available on a system

Modal Editor

Modal Editor

Two fundamental modes of operation Insert/Replace Mode

- for entering new text or overtyping text

Command Mode- for all other functions –

deleting text, moving cursor, saving work, copying text, etc.

Editing a file

Editing a file

To edit a file using the vi editor, just type

To enter the vi editor without specifying the file name in advance, type:

$ vi filename

$ vi

Leaving vi

Leaving vi

To exit, you must be in Command mode

To verify this mode, press <ESC> until you hear a beep :q! exit immediately and discard all changes :w save file only, remain in vi with current file :wq write the file and exit back to the shell ZZ write the file, and exit back to the shell

Entering Text

Entering Text

To enter insert mode from Command Mode, press i insert text before the cursor a append text; insert after the cursor o insert a new line after cursor line and remain in

insert mode O insert a new line before the cursor line

To exit back to command mode, press the <ESC> key

Insert / Replace mode

Insert / Replace mode

As you type, follow each line by pressing the <Return> key

The <backspace> key, denoted by the following:

moves cursor back and erases the previously typed characters, while remaining in insert mode

Cursor keys may not do what you expect while in this mode

Command Mode

Command Mode

To verify you are in Command mode, press <ESC> until you hear a beep

Two forms of commands direct commands

- executed immediately

ex commands- are always preceded by a colon “:”- followed by the command

Cursor Modes

Cursor Modes

The cursor keys may move one character in the direction of the arrow

More certainly, you should use the following keys: j move cursor down one line k move the cursor up one line h move the cursor to the left on character l move the cursor to the right one character space move the cursor to the right one character

Moving around in vi

Moving around in vi

To move further, you may use the following keys: Ctrl-f move forward one screen Ctrl-b move backward one screen G move to end of the file 3G move to line 3 of the file 1G move to line 1 of the file 0 move to the beginning of the line $ move to the end of the line

Deleting Text

Deleting Text

Removing text by the cursor: dd deletes the cursor line 5dd deletes the current line and 4 more d$ deletes from the cursor to the end of the line dw deletes from cursor to next white space x deletes the character under the cursor 5x deletes the character at the cursor and 4

more U undoes the last change, and only the last.

Searching in vi

Searching in vi

Text can be searched in the following ways: /pattern advance the cursor from present

position to the next occurrence of pattern

?pattern advance cursor from present position to previous occurrence of pattern

n repeat the previous search in same direction as original search

N repeat the previous search but in the opposite direction to original search

% Find the matching delimiter - with cursor on one of the characters {, [, (

find the matching },], ) or vice-versa

Replacing Text

Replacing Text

To replace text from Command Mode, enter one of the following: rd replace the character under the cursor with

the letter ‘d’. No <ESC> necessary R global replace mode. Text under cursor will

be replaced with typed text until <ESC> is pressed

cw Change the word from current cursor until and not including the next white space with

the typed text until <ESC> is pressed. A ‘$’ symbol marks the end of the text to be

replaced

More text replacement

More text replacement

Modifying text continued: 3cw change the next three words C$ or C change the text from the cursor until end

of line ~ map uppercase to lower or lower case to

upper for the character under the cursor

Substituting text

Substituting text

Substitute the first_exp with the replacement_exp

The [g] at the end means global to the line.

For the entire file, replace the first occurrence in a line of the string “first” with the string “second”

From the present cursor line to the end of the file, substitute all occurrences of the string “meat” with the string “bones”

:s/first_exp/replacement_exp/[g]

:1,$s/first/second/ or :%s/first/second/

:.,$s/meat/bones/g

Coping text

Coping text

Text can be copied with the following commands yy yank (copy) the current line to the temporary

buffer 3yy yank the current line plus two to the

temporary buffer xp delete the character under the cursor and

retrieve the deleted character after the cursor. (Interchanges two characters)

J join the current line with the next line

Retrieving text

Retrieving text

Yanked and deleted text can be inserted using: p pull (insert) the contents of the temporary buffer

following the cursor P pull the contents of the temporary buffer before

the cursor

Miscellaneous tricks

Miscellaneous tricks

A few extras: Ctrl-l Redraw the screen contents

(lowercase L) Ctrl-r Redraw the screen contents Ctrl-G File status

ex commands

ex commands

ex commands in fact invoke the ex line editor on the file for more information, see man ex

All ex commands are executed from Command Mode are prefaced by “:” are terminated by a carriage return

Range of command may be restricted through line numbers and patterns

! (bang)

! (bang)

:q! In this context the command is to be executed

without error reporting

:!ls In this context, it means to execute and display the

results of the shell command ls

:r !ls Perform the shell command ls and insert the results

into the present file following the cursor

set commands

set commands

The set command allows modification of the environment

The environment variables which can be accessed through the set command can be seen through the ‘:set all’ command

Typical Set Commands

Typical Set Commands

:set ai Auto indent; causes the cursor to automatically indent

to match the previous line in Insert Mode

:set noai no-auto-indent; turns ai mode off

:set nu Number; causes line numbers to be displayed

:set nonu no-number; causes line numbers to be turned off

:set sm showmatch: causes a momentary display of the

matching delimiter whenever a delimiter [], {}, ()is typed. Very handy in C

.exrc : making set commands stick

.exrc : making set commands stick

vi customisation start-up file checked for in the current working directory, then $HOME directory

enable auto-indent mode enable showmatch mode set tabstop to 4 spaces

trinity$ cat ~/.exrcset ai sm tabstop=4

Final Thoughts

Final Thoughts

All work is done on a temporary file

No changes to the original file occur until a command which causes the write has been executed

User must have write permission to modify the file

User needs only read permission to view the file

All files may be edited, but binary files are unintelligible

Shells and Variables

Shells and Variables

One cannot collect all the beautiful shells on the beach. One can collect only a few, and they are more beautiful if they are few.

Anne Morrow Lindbergh

What is an Environmental Variable ?

What is an Environmental Variable ?

A variable is a placeholder for information to be used by a process

There are two categories of variables Local (shell) Global (environmental)

Environmental variables include PATH controls where to look for commands to

execute PWD absolute path of current working directory

i.e. defines where you are currently in the file hierarchy

TERM describes the type of terminal in use

Other environmental variable hold information about the current user, the name of the machine, the local time zone, the machine architecture, process ids and the exit status of the last command executed.

Where do they come from ?

Where do they come from ?

From the shell

Initially, when a variable is created, it is local and so only available to the shell of its origin

To make it global, it must be exported

Can be customized or pre-defined by the user

By convention, Bourne and Korn shells use capital letters for pre-defined, built-in variables.

Example variables : Which shell?

Example variables : Which shell?

Look at the prompt $ Standard Bourne Shell (sh), Korn Shell (ksh) % Standard C Shell (csh) # This prompt is usually reserved for the root

(Super-User)

Examine the shell environmental variable $SHELL sh, ksh, bash $shell csh

sh is the standard command interpreter for Unix systems.

ksh, the Korn shell, csh, the California shell and bash, the Bourne again shell are alternatives that extend the functionality of the shell environment.

ksh is the default user shell in Solaris.

Example variables : Process ID [$$]

Example variables : Process ID [$$]

When scripting, it is often useful to know the Process ID of the script itself Useful for naming temporary files

$$ variable contain the PID of the script being run

When invoked in a shell, it gives the PID of the shell

trinity$ cat testscript.sh#!/bin/shecho “the PID of this script is $$”trinity$ sh testscript.shthe PID of this script is 3829

trinity$ echo $$417trinity$ ps –ef | grep 417root 417 413 0 13:02:51 pts/3 0:00 ksh

Example Variables : Exit Status [$?]

Example Variables : Exit Status [$?]

Numerical value that indicates whether an executed command was successful Zero Value = Successful Non-Zero Value = Not Successful

$? Exit status of last executed command

trinity$ touch newfiletrinity$ echo $?0trinity$ mkdir newfilemkdir: Failed to make directory “newfile"; File existstrinity$ echo $?2

Creating local variables

Creating local variables

$ variable_name=value Creates a local variable variable_name and assigns

to it the value value NOTE: there are no spaces around the “=”

NOTE: this only affects the current shell

trinity$ ZERO=1trinity$ echo $ZERO1

Variables can be viewed with the echo command and the name of the variable, prefaced by a $ symbol to denote to the shell that it should treat the name as a variable name and expand it before displaying it.

Removing environmental variables

Removing environmental variables

$ unset variable_name removes variable from the environment

trinity$ unset ZEROtrinity$ echo $ZEROtrinity$

It can be problematic to remove some built-in variables. For example, without a PATH variable you will have to type the full path to every command you want to run.

Making variables global [export]

Making variables global [export]

$ variable=value; export variable Creates a variable variable and assigns to it the

value value Then, places it within the global environment

available to this shell and any child processes of this shell

Note: there are no spaces around the “=”

trinity$ ZERO=1; export ZERO

Export should be used with the bare variable name. A common mistake is to use export $variable which the shell will expand to the contents of the variable before attempting the export.

Displaying the environment [set] [env]Displaying the environment [set] [env]

$ set Displays the current shell variable space (local)

$ env Displays the current environment space (global)

trinity$ setERRNO=1FCEDIT=/bin/edHOME=/IFS=''LINENO=1LOGNAME=rootMAILCHECK=600OPTIND=1PATH=/usr/bin:/bin:/usr/sbin:/sbin

trinity$ env_=/usr/bin/envSSH_TTY=/dev/pts/1PATH=/usr/bin:/bin:/usr/sbin:/sbinLOGNAME=rootUSER=rootSHELL=/sbin/shHOME=/SSH_CLIENT=172.16.0.100 34415 22TERM=screenPWD=/

Some variables appear in both listings.

Search paths [PATH]

Search paths [PATH]

$ PATH=directory [ :directory... ]$ export PATH; The shell uses the PATH variable to locate commands

in directories Directories are searched in the same order as they

are specified in the PATH statement Each directory is searched for the existence of the

command which is to be executed The first command found is executed

Search paths may contain an entry for dot, i.e. the current working directory. This is considered unsafe for users and dangerous for the super-user as it can allow the running of arbitrary code by accident.

Searching the PATH [which]

Searching the PATH [which]

$ which command The which command displays the pathname leading

to an accessible command based on your search path If a particular command isn’t found, you may need to

modify the PATH variable in order to add the directory in which it is located

trinity$ which passwd/usr/bin/passwd

which takes a list of names and looks for the files which would be executed had these names been given as commands.Each argument is expanded if it is aliased, and searched for along the user's PATH. Both aliases and PATH are taken from the user's shell.

Making the environment stick

Making the environment stick

Initialisation files System-Wide

User specific, in $HOME

/etc/profile/etc/.login

~/.profile~/.kshrc ~/.login~/.cshrc

It is sometimes useful to create shell variables which are available every time a new session is started without needing to be hand set.e.g. in an Oracle environment, the ORAHOME variable is commonly set to last between sessions.

In general, it is good practice to leave global shell configuration files as set up by the operating system vendor and make changes only in the user-specific files.

Initialisation file sequences

Initialisation file sequences

~/.bashrc/etc/profile~/.bash_profileor ~/.bash_loginor ~/.profile

bash

~/.kshrc/etc/profile~/.profile~/.kshrc

ksh

~/.cshrc/etc/.login~/.cshrc~/.login

csh

/etc/profile$HOME/.profile

sh

on new shellon loginshell

Typically the ~/.bash_profile contains if [ -f ‘~/.bashrc’ ];

then source ‘~/.bashrc’; fias the last entry to get ksh like behaviour

More about Shells

More about Shells

Try to be like the turtle -at ease in your own shell.

Bill Copeland

Quoting in Shells [ ” ’ ‘ ]

Quoting in Shells [ ” ’ ‘ ]

Use single quotation marks to identify text literally

Use double quotation marks to identify text literally, while enabling variable and command expansion

Backticks, or graves, ` `, force the command enclosed in them to be executed and its output substituted into command before the remainder of the line is evaluated

trinity$ echo ’the path is $PATH’the path is $PATH

trinity$ echo ”the path is $PATH”the path is /bin:/sbin:/usr/ucbbin

trinity$ echo the hostname is ‘uname -n‘the host name is trinity

Why quote?We have seen that the shell has meta-characters which have special meanings. We have also seen that some of these characters have meanings to commands. We can use quoting to force meta-characters to be dealt with in the way we expect, and by the command we expect.

Escaping in shells [\]

Escaping in shells [\]

The back-slash \ escapes the the meaning of the following character so that it the shell does not interpret it, if it normally would

trintiy$ echo \”the path is \$PATH\””the path is $PATH”

The backslash - \ - allows single characters to be escaped so that they are not interpreted by the shell. This can be particularly useful for escaping quotes or single characters that would still be interpreted inside quotes.

Custom Prompts

Custom Prompts

$ PS1=value PS1 is a predefined prompt variable that

users can change NB there are no spaces around the “=”

$ PS1=”‘uname -n‘:\$PWD $”trinity:/home/morpheus $

It can be useful to set the prompt in your shell to display certain information.For example, if you are connected remotely to many machines, having the name of the machine in the prompt can help identify which session is which, and reduce the possibility of running a command on the wrong machine.It may also be useful to display the current working directory in the prompt, so that it is clear where one is in the file hierarchy at any time.The prompt variable is a commonly set in your personal shell initialisation file.

Aliases [alias]

Aliases [alias]

$ alias [name=value] with no options it lists which commands have aliases substitutes the name for value when executing

commands so that the command set can be richer NB there are no spaces around the “=”

trinity$ alias dir=”ls –l”trinity$ dir-rw-r--r-- 1 morpheus users 257 Apr 18 10:51 dantedrwxr-xr-x 2 morpheus users 117 Apr 18 10:43 newdir-rw-r--r-- 1 morpheus users 0 Apr 18 10:43 newfile

A popular use of aliases is to provide a way of using familiar commands on new systems, however, these need to be recreated on each new machine that is used and can cause problems if the vendor changes the way the command works between versions.Aliases can also be dangerous, if they change the way a command works or which command is run without showing this to the user.In general, aliases should be avoided.

Removing Aliases [unalias]

Removing Aliases [unalias]

$ unalias name

removes name from the alias tables

trinity$ unalias dirtrinity$ dirksh: dir: not found

Command recall [history]

Command recall [history]

$ history Recalls the last commands $HISTSIZE limits the size of the history $HISTFILE determines where the history is stored

trinity$ history28 man set29 man which30 man -k which31 dir32 historytrinity$

When typing long commands, or pipelines, it can be useful to be able to recall them to run again.The history command lists the most recently run commands.

Command line editing: the vi way

Command line editing: the vi way

$ set –o vi Allows you to recall command, re-execute

them and edit them using vi commands- j move cursor down one line- k move the cursor up one line- h move the cursor to the left on character- l move the cursor to the right one character- space move the cursor to the right one character

trinity$ set –o vi press <esc> key and then k until the desired command is displayedpress <CR> to execute that command

ksh offers two methods of command line editing, using keys borrowed from the two most popular Unix editors.

Command line editing: the emacs way

Command line editing: the emacs way

$set –o emacs Allows you to recall commands, re-execute them and

edit them using emacs commands- Ctrl-n move down one line- Ctrl-p move up one line- Ctrl-f move right one character- Ctrl-b move left one character- Ctrl-d delete a character- Ctrl-a move to start of line- Ctrl-e move to end of line

trinity$ set –o emacspress <ctrl-p> until command is displayedpress <CR> to execute command

Emacs command editing mode may be preferred by people who are more used to the emacs text editor.

Shell Scripting

Shell Scripting

“I write scripts to serve as skeletons awaiting the flesh and sinew of images.”

Ingmar Bergman

What is a command ?

What is a command ?

A command is a file which is marked executable May be a pre-compiled binary or a script

If you have execute permissions on the file, then you may run it Remember the fork()/exec() model

Commands may also be a functions built into the shell e.g. cd. alias, history

Shell Scripts

Shell Scripts

What is a script ? A file which is marked executable whose first line is of the form

The #! header tells the operating system what sort of commands are in the script

Why script? A means of automating tasks

Shell scripts can be … Simple Complex

#!/path/to/command

#! - can be pronounced as hash-bang, sh-bang, or hash-shriek

Flow control

Flow control

Commands within a script are usually executed one after the other in the order they appear in the file

Flow control commands can use input data to decide which command to execute next test condition or [ condition ]

Examples of flow control commands include if / else / elif until / while for

Flow control allows scripts to be more than just a list of commands, by allowing branches in logic based on conditions and data.

Testing conditions [test]

Testing conditions [test]

The test command is used to test conditions

It can be used in two ways

Commonly used conditions include file tests

- -d path tests if the path supplied is a directory- -x path tests if path is an executable file

integer comparisons- -eq and -ne equal to and not equal to

string comparisons- = and != is the same as and is not the same

as

test [condition] [ [condition] ]

The test utility evaluates the condition and indicates the result of the evaluation by its exit status. An exit status of zero indicates that the condition evaluated as true and an exit status of 1 indicates that the condition evaluated as false.

When comparing strings and variables in scripts, it is a good idea to place them in quotes.

Flow control: if

Flow control: if

Used to test the status of a condition and proceed with an action of the status of the condition is true

fi marks end of if block. Square brackets [ ] are used as shorthand for the test command- So, must have a space after the “[“

if [ condition ] then

commandfi

Flow control: else

Flow control: else

Creates a two-way branch perform one action if the status of the condition is true perform another action if the status of the condition is false

trinity$ cat elsetest.sh#!/usr/bin/bashif [ “$LOGNAME” != “neo” ]then

echo “Go away, $LOGNAME. You are not The One.”else

echo “Hurrah. You are $LOGNAME. I have found The One!”fitrinity$ who am Imorpheus pts/2 Apr 19 20:20 (trinity.matrix.com)trinity$ ./elsetest.shGo away, morpheus. You are not The One.trinity$ su – neoPassword: ********trinity$ ./elsetest.shHurrah. You are neo. I have found The One!

Flow control: elif (i.e.“else if”)

Flow control: elif (i.e.“else if”)

Offers a second condition if the previous is false

if [ condition1 ]then

command1elif [ condition2 ]then

command2else

command3fi

Flow control: nested if statements

Flow control: nested if statements

if statements can be nested

This allows multiple conditions to be tested Offers a second test, if the previous one is true



echo “Both conditions are true.”fi

fi

Flow control: while / until

Flow control: while / until

while and until statements are useful for “looping” until a particular condition occurs. This method allows “flexibility” if the number of tasks

repeated (such as a for loop) is uncertain.

The while command tests and executes a series of commands as long as a condition is true.

The until command continues the loop until a true condition exists.

The until construct is not available in csh.

Flow control: while / until examples

Flow control: while / until examples

num=0while [ $num -lt 7 ]do

echo “number is $num”num=`expr $num + 1`

done

password=todayname=nobodyuntil [ “$name” = “pass” ]do

echo “Enter Pass”read name

done

trinity$ sh while.shnumber is 0number is 1number is 2number is 3number is 4number is 5number is 6number is 7trinity$

trinity$ sh until.shEnter Pass fredEnter Pass billEnter Pass passwordEnter Pass passtrinity$

In the first example, the loop continues while the condition is true, i.e. until the condition is false.In the second example, the loop continues until the condition is true, i.e. while the condition is false.

The two contructs can be used interchangeably, depending on which is the easier form of logic to express.

Flow control: for

Flow control: for

The for statement is used to process a list operations are repeated for a list of known or “fixed”

values loop continues until each value has been processed arguments are either sourced in script, or stated

directly after for statement.

for arg in [list]do

command(s)...done

Use the for command to iterate through a know list of arguments.

Positional Parameters

Positional Parameters

Data can be passed to the script from the command line

These parameters are referenced with the following variables $0 name of command used to call script $1 to $9 first nine positional parameters $# contains number of parameters $* all positional parameters as one long

string “$*” put single pair of double quotes

around whole string $@ same as $* “$@” put double quotes around each

parameter

Data can be passed to scripts from the command line, as with any command.These parameters are accessed using the positional parameters variable.$0 is the first item from the command line, i.e. the command used to invoke the script$1-$9 are the first 9 parameters passed to the command.$* is a single string containing all parameters, quoting it, will put quotes around the whole string.$@ is a single string containing all parameters, however, quoting it will put quotes around each parameter in the string$# contains the total number of paremeters

Positional Parameters: continued

Positional Parameters: continued

Can access beyond $9 by using set braces e.g. ${10} will access 10th argument

${var} can be used instead of $var useful if followed by alpha-numerics

$? is the exit status of the last command 0 means successful Script can be ended with exit n, where n is the

exit status or condition code of the script

Set braces can be used to access beyond the ninth parameter and also to make variable names clear.

Networking

Networking

“The Network is the Computer”

S. McNeally

Networking and Distributed Systems

Networking and Distributed Systems

Most computer installations today are based on networks allows information and resources

to be shared more easily applications can be distributed

over multiple systems

Nowadays, typical computer installations consist of multiple systems connected together using some form of networking technology. It is now possible to construct highly complex “heterogeneous” networks, where the computers may be of different types, running different operating systems and even using different connection technologies. For example, it is not uncommon to have networks consisting of a mixture of PCs running MS-DOS or Windows connected with UNIX based workstation or server machines and even mainframes. Connection technologies range from simple serial connections through LAN based methods (Ethernet, Token Ring) to complex proprietary mechanisms tailored to a particular type of computer.Networked systems allow individual users on individual systems to share information more easily - files can be copied between systems rather than transported on some backup medium. Resources such as printers can also be shared, allowing them to be used more effectively.There is also a trend towards building distributed applications, where functionality is split across multiple systems. For example, a powerful central computer could store a large database that is accessed from a number of smaller systems that have less processing power but more sophisticated display and input facilities. This allows more user-friendly interfaces to be constructed for data input, and for displaying the results of calculations.

Basic Network Services

Basic Network ServicesFile Transfer

copying files from one system to another

Remote Terminal

interactive access to another system

Remote Execution

execution of a program on another system

Remote Printing

sharing printers amongst several systems

Electronic Mail

sending and receiving messages to/from users on other systems

Basic network facilities:File Transfer, allowing individual files to be copied from one system to another. Facilities for this range from very simple mechanisms such as Kermit, through the UNIX Basic Networking Utilities (uucp) to the sophisticated File Transfer Protocol (FTP) as provided by TCP/IPRemote Terminal access, allowing a user at one system to undertake an interactive session on another. The mechanism is to have the user “log in” to the remote system, then the behaviour is exactly as if they had logged in to a local system.Remote Execution, allowing a user at one system to execute a single program on another system, without having to actually log in to the remote system.Remote printing, allowing a user at one system to print a file on a printer that is attached to another system elsewhere on the network. This usually happens transparently (without any special actions required on behalf of the user) but the user may be required to specify the name of the system containing the printer.Electronic Mail, allowing users on different systems to send messages to each other. Electronic mail systems are becoming increasingly sophisticated, it is not uncommon to be able to send voice messages and other multimedia “documents” in this way.

Distributed File Systems

Distributed File Systems

Systems can access files on remote machines as if they were local

Applications and users are unaware of the location of the files

Files appearlocal here

Files areactuallystored here

The networking facilities described above generally require the user to have some knowledge of the network, such as which machine they wish to communicate with. Modern network environments may also provide facilities where the actual networking takes place at a lower level, and does not require specific knowledge by the user that the network is being used.Distributed file systems were the first systems to provide this. In the UNIX environment, a user has access to files through a single hierarchical file system which hides the details of which physical (or logical) disk actually contains the files (unlike DOS, where each logical drive must be explicitly be named and has its own hierarchy of files and directories). With distributed file systems, this concept is extended so that the hierarchy on a system may include directories and files which exist on a different system in the network.The most common UNIX distributed file system is the Network File System, or NFS, covered in some detail later in the course. Others are the Andrew File System (AFS), and the DCE DIstributed File System (DFS).The concept of “transparent” access to files on a remote system also forms the basis of Novell’s NetWare network operating system.

Distributed Window Systems

Distributed Window Systems

• High power system -calculations can be done here

• No hi-res screens

Graphical output can be displayed on screen

Input taken from keyboard and mouse

The main Graphical User Interface systems for UNIX offer more than the ability to create and manage windows and graphical output on the screen.The architecture of the X-11 system allows it to operate in a distributed manner in a networked environment. The application requiring the graphical output, and keyboard and mouse input need not be running on the same system as the graphics screen, keyboard and mouse being used. This allows a situation where a compute-intensive application can be run on a more powerful machine on the network, which does not have graphics facilities. The output is transmitted across the network to a less powerful system which has a graphics display and can display the output.Applications may be configured so that this division of computation from I/O is invisible to the user, it will appear as though the application is running on the local, graphics-based system.

Protocol Layers

Protocol Layers

To manage complexity, divide into layers

Application Application

message message

1

2

3

message

message

stack

message

1

21

321 message

message

message1

2

3

stack

1

21

321

Protocols to perform communication between computer systems are generally very complex. To help manage this complexity, the protocols are divided into a series of layers, each of which is responsible for handling some aspect of the communication. For example, one protocol layer will be responsible for translating the bits of data from the computer’s memory into electrical pulses on the connection medium, and then converting the pulses back to bits at the destination computer. Another layer could be responsible for ensuring that each block of data is sent to the correct destination (viewed from a higher level).The protocols at each layer each communicate conceptually with a peer protocol (ie a protocol at the same level) on the remote system. In reality, data is passed down through the layers on the sending system and up through the layers at the receiving system. At each layer on the sending system, control information is added to the original message. At the receiving system, the control information is interpreted by the peer protocol and removed from the message, so that the original message arrives intact.A collection of protocols structured in this way is known as a protocol stack.

The OSI 7 Layer Model

The OSI 7 Layer Model

Generic model for describing inter-system communications

7 Application

6 Presentation

5 Session

4 Transport

3 Network

2 Datalink

1 Physical

7 Application

6 Presentation

5 Session

4 Transport

3 Network

2 Datalink

1 Physical

Provides applications w ith access to the OSI environment

Ensures consistent representation of data among all systems

Management of a dialogue session between two applications

Enhances quality of service given by lower three layers

Ensures blocks of data arrive at their correct destinations

Blocks data and ensures correct transmission and reception

Defines physical and electrical characteristics

Provides applications with access to the OSI environment

Ensures consistent representation of data among all systems

Management of a dialogue session between two applications

Enhances quality of service given by lower three layers

Ensures blocks of data arrive at their correct destinations

Blocks data and ensures correct transmission and reception

Defines physical and electrical characteristics

The OSI 7-Layer ModelISO has defined a standard model to describe communications between computer systems, known as the 7 layer model for Open Systems Interconnection (OSI). This model is useful as a framework against which to analyse and compare different protocols. Each of the 7 layers deals with a different aspect of communication between systems:

1. Physical - describes the physical and electrical standards of the communications medium, such as voltages, resistances and cabling standards.2. Data Link - describes how the communications medium is accessed, and how groups of bits known as frames are sent and received.3. Network - describes how data is sent from one system to another, including any routing details.4. Transport - describes end-to-end communication, independent of any routing.5. Session - deals with the dialogue between systems.6. Presentation - ensures that data consistency is maintained between systems, irrespective of any differences in machine data representation.7. Application - presents a service to the user.

The TCP/IP Protocol Stack

The TCP/IP Protocol Stack

ApplicationApplication

TransportTransport

NetworkNetwork

Data LinkData Link

OSI Layer 5-7

OSI Layer 4

OSI Layer 3

OSI Layer 1-2

TELNET, FTP

TCP, UDP

IP (ICMP, ARP, RARP)

Ethernet, Token RingFDDI, serial, etc.

The TCP/IP StackThe Internet protocol suite can be described using a stack model similar to, though simpler than the OSI 7 layer model.At the heart of TCP/IP is the Internet Protocol IP. This is a connectionless, unreliable datagram service whose main job is to forward packets from one system to another. It can be placed at level 3 (Network) of the ISO model.Above IP come the two transport protocols, whose job is to move information from one endpoint to another.TCP, Transmission Control Protocol, is a reliable, connection oriented byte stream service between two hosts on an internetwork. Considerations of routing, reliability, sequencing and flow control are dealt with using the facilities of IP and another Network level protocol known as ICMP (Internet Control Message Protocol).UDP, User Datagram Protocol, is a datagram based service between two internetwork hosts. UDP is not reliable.Higher level network services are known as Applications, and encompass protocols at layers 5-7 of the ISO model. A number of application services are defined now. These include file transfer with FTP and TFTP, and remote terminal emulation with TELNET. IP is the “lowest” protocol defined in the Internet suite. No attempt is made to define data link protocols. This makes the Internet Protocols very flexible. All that is needed is a connection medium between two systems, and an implementation of IP that supports that medium. Currently, IP is available over Token Ring, Ethernet, FDDI, serial links (although these generally have to be high speed to provide reasonable service) and wide area links such as X.25.

Concepts

Concepts

The Network Layer handles the transmission of IP packets across networks

The Transport Layer provides a flow of data between two hosts for each application

• This can take 2 forms• Reliable flow

• Unreliable flow

TCP UDP

IP ICMP

APPS

TCP UDP

IP ICMP

APPS

All that the Network layer provides is transmission of IP packets from host to host across networks. It does not have any knowledge of which applications own which data, but simply funnels all data for a particular host from one end to another.The Transport Layer is where we start to break the data down and direct it to and from applications across networks.

Connectionless protocols

Connectionless protocols

• There is no guarantee of delivery

• There is no guarantee on the order in which data arrives at the other end

• Such protocols are ‘unreliable’ - like IP

DATAER

Imagine frames of data as postcards. You want to send 10 frames to a friend in another city. Sending them via an unreliable protocol means that you have no guarantee that all 10 postcards will arrive at the other end (though, like the postal service, in general they will), plus you have no guarantee in which order the postcards will be delivered.IP works in this way. IP packets can be lost or damaged. There is no mechanism at the network layer to deal with the notification of losses to the sender, or to request that the packets are re-sent. That is left up to a higher level protocol.

Connection-oriented protocols

Connection-oriented protocols

DATA DATA

TWO-WAY COMMUNICATION

• No data loss - ‘reliable’

• Data arrives at the other end in the same order that it was sent

Consider data exchange as a telephone conversation. First, a connection is establish - i.e. the number is dialed and connects. Then two-way data exchange takes place. At all time the data arrives at the other end in the same order that it leaves - your words don’t arrive in a different order, and vice versa. When the transfer (conversation) is complete, the connection is closed - we hang up.This is an entirely reliable data transfer, as we can’t lose anything that is transferred from the other end.

Ports + Sockets

Ports + Sockets

• A socket is a pair of data queues for an application process

• A port is the address of a socket

INPUT OUTPUTSOCKET

APPLICATION PROCESS

PORT

Sockets are usually implemented as an area of memory within the kernel of an operating system. Incoming data is ‘bound’ via its port address to a socket, and thus directly into an application process.Port addresses are numbers in the range 0-65535

Important Port Numbers

Important Port Numbers

Some port numbers have been reserved for common services across the internet

Port Service7 Echo13 Date + Time19 Character Generation20 & 21 FTP23 Telnet25 Mail (SMTP)80 WWW110 Mail (POP)119 News

In general, port numbers below 1024 are reserved for what are called Well Known Services (WKS). Numbers above 1024 are free for user applications to do with as they wish.

UDP - Fundamentals

User Datagram Protocol (UDP) Connectionless Unreliable

Used for Domain Name Service (DNS) Routing Information Protocol (RIP) Real Audio Network File System (NFS) Trivial File Transfer Protocol (TFTP) Network Time Protocol (NTP) Plus many others

UDP - Fundamentals

UDP can be used wherever reliability is unimportant and where a stream of data is not needed - i.e. no frame numbering is required.

UDP - Frame format

UDP - Frame format

Source Port Number Destination Port Number

UDP Length UDP Checksum

DATA(0-65508)

2 2OCTETS

Within IP

Frame Format

IP HEADER UDP HEADER UDP DATA

Note that the UDP checksum covers both the UDP header and the UDP data fields. This is distinct from the IP checksum which is a header checksum only, and does not cover the integrity of the data.This is not normally a problem with IP, as the data-link layer protocols often provide a checksum for the network layer data. However, in some cases (e.g. SLIP) such a check is not offered, and so it is required at this level.However, if the receiving host detects an error in the UDP data frame, it will not return an error to the sending host, but simply discard the frame - the same behaviour that IP demonstrates.

UDP - Issues

Unreliability No good for ‘streamed’ services where reliability is

important, where a connection is needed

Examples FTP Mail Telnet WWW

UDP - Issues

TCP - Fundamentals

Transmission Control Protocol (TCP) Connection-oriented Reliable

Used for FTP Mail News Web And many more

TCP - Fundamentals

TCP - Frame format

TCP - Frame format

SOURCE PORT NUMBER DESTIN ATION PORT NUMBER

SEQUENCE NUMBER

ACKNOWLEDGEMEN T NUMBER

LENGTH RESERVED WINDOW SIZE

TCP CHECKSUM URGENT POINTER

OPTIONS (if any)

DATA(0-????)

SEEEXPANDED

URG

ACK

PSH

RST

SYN

FIN

EXPANDEDSECTION

OCTETS2 2

(6 bits)

4 Bits

Frame Format

As for UDP there is still a source & destination port number.The sequence number field identifies the byte in the data session stream that the first byte of the frame represents. This is so that the frames can be reassembled in the correct order at the other end. TCP (a connection-oriented) protocol is run over IP (connectionless), and so this mechanism is needed for the destination host to make sense of the order of the incoming frames.The acknowledgement number field allows the receiving host to respond with an acknowledge that it has correctly recieved the frame. This is the basis of TCP’s reliability. If a frame is not acknowledged, then it is send out again until it is.A header length field is necessary as there are a number of TCP options (see overleaf) that can be included.The six flags in the expanded section involve various control messages that are sent in order to set up the connection or to close it. They have the following meanings:SYN - Synchronise sequence numbers to initiate a connectionACK - Acknowledgement frameRST - Reset the connectionFIN - Sender has finished sending dataURG - Urgent incoming dataPSH - Pass this data onto the application as soon as possible

TCP - A Sample Session

TCP - A Sample Session

HostA

HostB

3 Phases:

HostA

HostB

HostA

HostB

12

3

DATA TRANSFER

TERMINATION

SYN <seq1> <mss>

ACK <seq2>+1 <mss>, SYN <seq2>

ACK <seq2>+1

ESTABLISHMENT

D ATA

D ATA

FIN <seq1>+n, ACK <seq2>+m

ACK <seq1>+n+1

x n

x m

FIN <seq2>+m, ACK <seq1>+n+1

ACK <seq2>+m+1

‘mss’ stands for Maximum Segment Size, and is a limit on the max size of data frames that can be sent to the host. This is primarily useful if the host in on a network with a small MTU, and so wants to avoid too much fragmentation of the data.The reason we have 4 termination signals, but only 3 to set up the connection is that each side of the connection must be shut down individually. TCP is full-duplex communication, and half-duplex can be achieved by performing a ‘half-close’ - i.e. shutting only one side of the connection down.

TCP - Options

TCP - Options

End of option

list

No operation

Max segment

size

Window scale factor

Time-stamp

1

1

1

1

1

1

1

1

1

2

4 4

kind=0

kind=1

kind=2

kind=3

kind=8

len=4

len=3

len=10

max segment size

shiftcount

Timestamp Value Timestamp echo reply

Name Services

Name Services

Several Name Services exist Common examples include

- DNS- LDAP

System databases and Name Services are configured in /etc/nsswitch.conf

Domain Name Service (DNS)

Domain Name Service (DNS)

DNS maps IP addresses to computer names and vice versa Mapping achieved with a name server

Provides a more user friendly address format

Hierarchical structure

DNS does other things too Holds service records on where to find things

SRV records Hold mail router information

MX records

Domain Name Service: Examples

Domain Name Service: Examples

Form of DNS Address company.com

- A company’s domain name ahost.mycompany.com

- A computer in the domain www.company.com

- A web server company’s domain www.othercompany.co.uk

- A web server for a UK company

Structure of DNS name space

Structure of DNS name space

org ukcom

.

orgco gov

The DNS system knows a hierarchical structure: • The root node(RED) is the "dot" domain. This dot is the origin of all domains. It is comparable with the root of a UNIX filesystem. • Below the root node you will find a number of Top Level Domains (YELLOW). These can further be distinguished in Generic Top Level Domains (gTLD), such as com, org and net, and Country Code Top Level Domains (ccTLDs), such as nl (for the Netherlands), au (for Australia) and uk (for the United Kingdom).

• Below a Top Level Domain an organization can apply for a subdomain. The application criteria and procedure for this varies from TLD to TLD. When an application has been granted, then that organization becomes the "owner" of a domain, and can use it to store information about its own hosts and (possibly) other subdomains. Furthermore, the DNS system is decentralized. This means that there is no central database which holds all the information, but organizations all keep their own databases on their own servers. Through special so-called "glue records", these databases all point to each other, making global lookups possible

LDAP

LDAP

Lightweight Directory Access Protocol Protocol for enabling anyone to locate organizations,

individuals, and other resources such as files and devices in a network

LDAP has a universal format, which supports display of all names.

A typical LDAP server is a simple network-accessible database where an organization stores information about its authorized users and what privileges each user has.

Provides centralised management

/etc/nsswitch.conf

/etc/nsswitch.conf

Configuration file for name services and system databases

Tells system where to find user data and computer name/IP address mapping

Data locations include files

- e.g. /etc/passwd, /etc/shadow, /etc/hosts

dns ldap

Connect to another host [telnet]

Connect to another host [telnet]

$ telnet hostname [port] Connect to a remote host whose name is hostname

for the purposes of transferring running some interactive programs

[port] by default this is 23 but could be otherse.g. 80 (http), 25 (SMTP), 143 (IMAP)

neo$ telnet trinityTrying trinity.matrix.com...Connected to 10.1.5.1.Escape character is '^]'SunOS 5.9login:

Fetching remote files [ftp]

Fetching remote files [ftp]

$ ftp hostname Connect to a remote host whose name is hostnamefor the purposes of transferring files

[port] by default this is 23 but could be others e.g. 80

neo$ ftp trinityConnected to 10.1.5.1.220 trinity FTP server (Unix® System V) readyName (10.1.1.1:unknown): morpheus331 Password RequiredPassword: *****230 User morpheus logged in.ftp>

Inside ftp

Inside ftp

The following commands are available at the ftp>prompt nlist|dir|ls list the files in the current remote directory cd changes the current remote directory pwd displays the current remote directory !command execute the command command locally ascii|bin specifies the time of file to be transferred get filename gets the file filename from remote directory

and makes a local copy put filename puts the file filename from local directory

and makes a local copy in the remote directory mput|mget allow multiple puts or gets with wildcards to be

done quit ends the ftp session

Secure remote connections [ssh]

Secure remote connections [ssh]

$ ssh [-l login_name] [hostname | user@hostname] [command] Makes secure connection to hostname using public key

encryption By default, connects as the current user Can specify a command to run on the remote machine [-l login_name] username to connect to host as [user@hostname] alternative way to specify user [hostname] host to connect to [command] command to run on remote

machine

neo$ ssh trinityroot@trinity’s password: *****trinity$

Remote login without passwords [ssh]

Remote login without passwords [ssh]

SSH secures connections using public/private keys

Public keys can be shared between machines to allow non-interactive login Useful to allow scripts to run things on remote

machines without needing to hardcode a password Can be dangerous as if one machine is compromised,

many can be

Keys are created with ssh-keygen

Startup and shutdown

Startup and shutdown

“There is only one satisfying way to boot a computer”

J. H. Goldfuss

Boot Stages

Boot Stages

There are three distinct stages in the boot flow. Firmware Kernel Initialisation & hardware Software run levels

Briefly, it can be summarised as follows

POST loads kernel

kernel initialisesruns initruns rc scripts

PROM loads bootstrap

The power on self tests are used when a system is powered on or reset from the PROM level. They check out the CPU and the MLB and the tests can vary from system to system. By default the POST output is not displayed to the monitor, however it can be viewed by connecting an ASCII terminal to the serial port.User commands are used to control the operation of the PROM. They can change PROM variables such as the boot device and security modes.The PROM contains an entire programming language and operating system known as forth. Developed in 1978, this high-level language can fit into an 8KByte chip.

Firmware

Firmware

The output from the POST can be seen on an ASCII terminal attached to Serial Port A of the machine

Power On Self TestsPower on banner

Test memoryPROM loads boot block

POST output

POST output

ok setenv diag-switch? trueok setenv auto-boot? falseok reset

PROM Checksum testContext Reg TestSetting Segment MapSizing MemoryMapping ROMSetting up RAM for monitor.Setting up memory used in decompress.Decompressing code to RAM ...Remapping monitor's virtual addresses to RAM.Probing /sbus@1,f8000000 at 0,0 dma esp sd st leProbing /sbus@1,f8000000 at 1,0 Nothing thereProbing /sbus@1,f8000000 at 2,0 Nothing thereProbing /sbus@1,f8000000 at 3,0 bwtwoCan't open input device.SPARCstation IPC, No KeyboardROM Rev. 2.4, 12 MB memory installed, Serial £27831.Ethernet address 8:0:20:a:e9:da, Host ID: 52006cb7.

The tests carried out by POST are simple yes/no type tests. It checks the integrity of the CPU, hardware and any devices it can probe.

POST output

TestingTesting audio chip; listen for a beepSynchronous Error Reg TestSynchronous Virtual Address Reg TestAsynchronous Error Reg TestAsynchronous Virtual Address Reg TestSystem Enable Reg TestTesting 12 megs of memory. Still to go 0Cache Tag Memory TestCache Data Memory Test

SBus slot 0 le esp dmaSBus slot 1SBus slot 2SBus slot 3 bwtwo

Type b (boot), c (continue), or n (new command mode)>

Open Boot PROM (OBP)

Open Boot PROM (OBP)

The boot PROM consists of Power On Self Tests (POST) Device drivers to know what to boot from User commands (Forth Toolkit) User diagnostics (Forth Toolkit) User definable parameters

It is a single, upgradeable chip on the Main Logic Board.

The current revision is OBP 4.x

The power on self tests are used when a system is powered on or reset from the PROM level. They check out the CPU and the MLB and the tests can vary from system to system. By default the POST output is not displayed to the monitor, however it can be viewed by connecting an ASCII terminal to the serial port.User commands are used to control the operation of the PROM. They can change PROM variables such as the boot device and security modes.The PROM contains an entire programming language and operating system known as forth. Developed in 1978, this high-level language can fit into an 8KByte chip.

OBP: boot command

OBP: boot command

To start the OS from the OBP ok prompt use

Boot options include no options boot from default device [cdrom] boot from cdrom [net] boot from network using ARP/RARP

- [:dhcp] use DHCP to fetch network address- [ - install] perform a Jumpstart installation

[-r] reconfigure boot [-s] boot to run level 1 (Single user) [-v] verbose output [-a] interactive boot

boot [device] [options]

Kernel Initialisation

Kernel Initialisation

The operating system is loaded from the boot PROM

The kernel is the core of the operating system

bootstrap file loads kernel

reports memory,ethernet address,CPU type.

checks/rebuilds devices

kernel identification

identifies root, swap & dump executes init

The /etc/system file

The /etc/system file

/etc/system contains system parameters

Read by kernel on boot

The following types of customization are available in the /etc/system file: moddir Changes path of kernel modules. forceload Forces loading of a kernel module. exclude Excludes a particular kernel module. rootfs Specify the type for the root file system.

(ufs is the default.) rootdev Specify the physical device path for root. set Set the value of a tuneable system parameter.

Incorrect editing of /etc/system can render a machine unbootable!

If the /etc/system file is edited, it is strongly recommended that a copy of the working file be made to a well-known location. In the event that the new /etc/system file renders the system unbootable, it might be possible to bring the system up with a boot -a command that specifies the old file. If this has not been done, the system may need to be booted from CD or network so that the file can be mounted and edited.

Boot output

Boot output

{ok} bootBoot device: disk File and args:Loading ufs-file-system package 1.4 04 Aug 1995 13:02:54.FCode UFS Reader 1.12 00/07/17 15:48:16.Loading: /platform/SUNW,Ultra-Enterprise/ufsbootLoading: /platform/sun4u/ufsbootSunOS Release 5.9 Version Generic_112233-01 64-bitCopyright 1983-2002 Sun Microsystems, Inc. All rights reserved.Use is subject to license terms.configuring IPv4 interfaces: hme0.starting DHCP on primary interface hme0Hostname: trinityThe system is coming up. Please wait.starting rpc services: rpcbind done.Setting default IPv4 interface for multicast: add net 224.0/4: gateway 10.42.0.1syslog service starting.syslogd: line 24: WARNING: loghost could not be resolvedvolume management starting.The system is ready.

trinity console login:

init and run levels

init and run levels

Solaris works in one of a number of operational states called ‘run levels’

A ‘run level’ is a mode of operation in which certain resources are made available

- a known set of processes are running- certain files are mounted and (possibly) shared- Solaris run levels are different to other UNIXes

init reads /etc/inittab

/sbin/rcS/sbin/rc2

executes /sbin/autopush

login prompt

/sbin/rc3

Predefined run levels for Solaris

Predefined run levels for Solaris

Solaris run levels differ from other UNIX-like systems

O ptional and configurable run lev els.a,b,c

Force the sy stem to re-read /etc/inittab.Q, q

Bring the sy stem to S ingle user lev el where some file sy stems are mounted and user log ins are disabled.

S, s

Halt and reboot to run lev el 3.6

Halt the sy stem and boot interactiv ely (boot -a). orHalt the sy stem and poweroff (sun4m architecture)

5

Full multiuser lev el w ith (NFS) resources shared.3

Multiuser lev el w ith no (NFS) resources sha red.2

Bring the sy stem to S ingle user lev el where some file sy stems are mounted and user log ins are disabled.

1

Bring the sy stem to PROM monitor lev el0

ActionRun level

Other UNIX-like OSs have different definitionse.g. LinuxRunlevel System State0 Halt System1 Single user mode2 Basic multi user mode without NFS (The same as 3, if you don't have networking)3 Full multi user mode (text based)4 unused5 Multi user mode with GUI6 Reboot System

The /etc/inittab file


This file tells the init process what processes to create for each run level.

# cat /etc/inittabap::sysinit:/sbin/autopush -f /etc/iu.apfs::sysinit:/sbin/rcS>/dev/console 2>&1 </dev/consoleis:3:initdefaultp3:s1234:powerfail:/usr/sbin/shutdown -y -i5 -g0 >/dev/console 2>&1. . . . . . . . . . . . . . . . . . . . . .

The /etc/inittab file defines three main items for the /sbin/init process.The system’s default run level.What processes to start, monitor, and restart if they die.What actions to be taken when the system changes run levels.


Each entry contains four fields separated by colons. id s3

rstate 3

action wait

command :/sbin/rc3>/dev/console 2>&1 </dev/console

s3:3:wait:/sbin/rc3>/dev/console 2>&1 </dev/console

id This is a field of up to four characters which is used to uniquely identify the entry.rstate This field defines at which run level to execute.action This key word field tells init how to treat the process.

Valid keywords are as follows:initdefault The default run level.respawn Restart the process when it dies.powerfail Start the process when init receives a power

fail. (Useful for UPS systems.)sysinit Start the process and wait for completion.wait Start process and wait for it to finish.

cmd The command or script to execute.

Run Control Scripts

Run Control Scripts

Run Control Script for each level calls further script files for that level.

/etc

rc0 rc3rc1 rc2 rc5 rc6 rcS

rc0.d rc1.d rc2.d rc3.d rcS.d

S15nfs.server

init.d

K20lp

For each run level there will be an entry in /etc/inittab which calls the rc scripts. These controlling scripts are located in /sbin but are linked to files in /etc. Each run level script in turns executes the files found under the /etc/rc#.d directory in alphanumeric order.Each run control scripts found in these sub directories is either in the format K## or S##. The K## files are ‘kill files’ and are used to stop processes when the system enters this run level. The S## files are ‘start files’ and are used to start processes.There is also a directory /etc/init.d which is used for stopping and starting individual services or processes without having to change run levels. Each of these scripts is hard linked to the corresponding files in the /etc/rc*.d directories. These files are in simple to remember format are used with either the parameter stop or start.

/etc/init.d/lp stop/etc/init.d/sendmail stop ; /etc/init.d/sendmail start

A Typical Run Control Script

A Typical Run Control Script

Run Control files are simple Bourne shell scripts.trinity$ cat /etc/init.d/lp#!/sbin/sh## Copyright (c) 1997, 2001 by Sun Microsystems, Inc.# All rights reserved.##ident "@(#)lp 1.10 01/11/04 SMI"

case "$1" in'start')

if [ -z "$_INIT_PREV_LEVEL" ]; thenset -- `/usr/bin/who -r`_INIT_PREV_LEVEL="$9"

fi

[ $_INIT_PREV_LEVEL = 2 -o $_INIT_PREV_LEVEL = 3 ] && exit 0

PRINTERS=/etc/lp/printersCONFIG=configuration

...

The above example is used to start and stop the print process. The file has two other names: /etc/rc1.d/K20lp and /etc/rc2.d/S80lp. The K name ensure that the process is killed when entering run level 1, and the S name ensures that it is started when entering run level 2.

Note : To disable a start (or kill) file do not simply add on an extension. The conventional way is to change the K or S prefix to a lower case k or s. This works because the /sbin/rc scripts only search for K* and S*.

The /etc/init.d directory

The /etc/init.d directory

The /etc/init.d directory contains hard links to the files in /etc/rc*.d

This allows processes to be stopped or started without having to change run levels.

trinity$ /etc/init.d/lp stoptrinity$ /etc/init.d/volmgt stoptrinity$ /etc/init.d/lp start

The /etc/init.d directory is used by the system administrator to start and stop system daemons (processes) by hand. This is sometimes necessary when processes go wrong, but is also useful to test updated rc scripts. Note that system processes should always be stopped in this way, rather than simply using the kill command, because some system processes may also require other processes to be stopped and/or files (especially lock files) to be cleaned up.The scripts in this directory are hard links to their appropriate S scripts and K scripts found in the other directories. They do not have the S or K prefix because the administrator explicitly passes a start or stop parameter depending on whether the script should start or stop something.Administrators usually create new run control scripts in this directory, and then link them into the appropriate /etc/rc?.d directory with a K or S name and corresponding number. Thus the administrator may define the run-level in which the processes are started and stopped, and the order in which this occurs.

Run level commands

Run level commands

Four commands exist to change run levels: shutdown

halt

reboot

telinit

Run level commands: shutdown

Run level commands: shutdown

The /usr/sbin/shutdown command is normally used to change to run level S or run level 0

shutdown [-y] [-gseconds] [-irun_level]

trinity$ shutdown -g300 -i0Shutdown started. Sat Nov 25 11:03:00 BST 1995Broadcast Message from root (console) on viper Sat Nov 25 11:03:00..THE SYSTEM IS BEING SHUT DOWN NOW ! ! !Log off now or risk your files being damaged.

The system is coming down. Please wait.System services are now being stopped.Print services stopped.Stopping the syslog service

By default shutdown will take the system to run level S (single user mode). The shutdown command will broadcast a warning message to all users who are currently logged onto the system that it is about to go down. It will not inform any users who may be NFS mounting directories. Use the rwall command to inform them.The options available with this command are as follows :-y shutdown the system with no operator intervention.g A grace period before shutdown occurs (in seconds).i Specify the run level to change to. Choices are 0, 1, 2, 5 & 6.

When in single user mode typing ^D (control-D) will move the system back to the default run level (run level 3).

Note : It is good practice to exit any GUI environment (eg OpenWindows or CDE) and change directory to root before running shutdown.

Run level commands: halt

Run level commands: halt

The /usr/sbin/halt command brings the system to run level 0 (PROM monitor level).

trinity$ haltsync file systems [2] [2] [3] [3] [4]ok

Using either of these commands will not broadcast any warning messages to the users. The commands begin the shutdown process immediately.

Run level commands: reboot

Run level commands: reboot

The /usr/sbin/reboot command performs a clean shutdown and brings the system back to run level 3 (default).

Options can be passed to the boot command with reboot -- -r will perform a reconfiguration boot on reboot

reboot [bootoptions]

The reboot utility restarts the kernel. The kernel is loaded into memory by the PROM monitor, which transfers control to the loaded kernel.Although reboot can be run by the super-user at any time, shutdown(1M) is normally used first to warn all users logged in of the impending loss of service.The reboot utility performs a sync(1M) operation on the disks, and then a multi-user reboot is initiated.

Run level commands: telinit

Run level commands: telinit

$ telinit [0123456abcQqSs]

The /usr/sbin/telinit command can be used instead of the shutdown, halt and reboot commands to change system run levels.# init 6INIT: New run level 6The system is coming down. Please wait.System services are now being stopped.Print services stopped.Stopping the syslog serviceNov 23 15:12:21 viper syslogd: going down on signal 15The system is down.rebooting....SPARCstation 10 (1 X 390Z50), Keyboard PresentROM Rev 2.10..............

The init command can move the system to any of the run levels. It is somewhat more direct that shutdown, however, giving no opportunity for users to logout.

Devices

Devices

There are a number of mechanical devices which increase sexual arousal, particularly in women. Chief among these is the Mercedes-Benz 380SL convertible.

P. J. O'Rourke

Local Devices

Local Devices

Local devices are those which are directly connected to the machine

These can include Hard Disk Drives CDROM Drives Tape Drives and Jukeboxes

Device Naming Conventions

Device Naming Conventions

Device names in the OS have four parts Controller number Target ID or SCSI ID Drive number or LUN Slice or partition

Examples include c0t0d0s0 the first disk drive in a machine c0t6d0s0 the internal CDROM drive

Links to these devices can be found in /dev/dsk and /dev/rdsk

These conventions are used within the operating system

Device aliases [devalias]

Device aliases [devalias]

Devices are also named in the boot PROM

The command devaliascan be used to show the available devices

ok devaliasscreen /sbus@1,f8000000/bwtwo@3,0ttyb /zs@1,f1000000:bttya /zs@1,f1000000:akeyboard! /zs@1,f0000000:forcemodekeyboard /zs@1,f0000000disk /sbus/esp@0,800000/sd@3,0net /sbus/le@0,c00000cdroma /sbus/esp@0,800000/sd@6,0:acdrom /sbus/esp@0,800000/sd@6,0:ctape /sbus/esp@0,800000/st@4,0floppy /fdtape0 /sbus/esp@0,800000/st@4,0tape1 /sbus/esp@0,800000/st@5,0disk3 /sbus/esp@0,800000/sd@0,0disk2 /sbus/esp@0,800000/sd@2,0disk1 /sbus/esp@0,800000/sd@1,0disk0 /sbus/esp@0,800000/sd@3,0scsi /sbus/esp@0,800000ok

To reference devices in the OBP, use the strings shown in devalias

Partitions

Partitions

Logical divisions within each physical disk.- A single range of contiguous blocks- A physical disk may have up to eight partitions

Partitions are called slices in Solaris, and are indicated by the letter ‘s’ when naming each partition device.

Partitions may be formatted in a variety of ways- Unix UFS- Veritas’ VxFS- A paritition may also be left raw

- e.g for kernel swap space

Why Partition?

Why Partition?

Simplifies Administration Backup individual partitions OS updates may only affect specific partitions Partitions can exist on external disks

Improves Data Security Localises damage thus protecting other partitions

Disconnect sensitive partitions

Performance Enhancement Reduces seek time Localises fragmentation Partitions can be configured to suit file sizes

Partition Tables

Partition Tables

The Partition table of a disk is held in a Volume Table of Contents (VTOC)

Partitions must be contiguous and must start and end on cylinder boundaries

A Partition is defined as an offset and a size

0 55

Size55

Partition 0

129

Size74

Partition 1

Size167

296

Size105

Partition 6 Partition 7

O ffset

Initial partitioning is carried out at installation time and ensures there are no overlapping or non-contiguous partitions. So, the first partition starts on cylinder 0 and continues through to the end of cylinder 54. The next partition then starts on cylinder 55 and so on.Should partitions need to be changed after installation time then the format utility is available in Unix for this.The output from prtvtoc reads the disk label and returns the partitioning information.

Listing partitions [prtvtoc]

Listing partitions [prtvtoc]

The prtvtoc command is used to report disk geometry and list partitions [-f] report disk free space [-h] omit headers from report [-s] only show column headers device name of the device to list

/usr/sbin/prtvtoc [-fhs] device

NAME prtvtoc - report information about a disk geometry and partitioning

SYNOPSIS prtvtoc [-fhs] [-t vfstab] [-m mnttab] device

DESCRIPTION The prtvtoc command allows the contents of the VTOC (volume table of contents) to be viewed. The command can be used only by the super-user.

The device name can be the file name of a raw device in the form of /dev/rdsk/c?t?d?s2 or can be the file name of a block device in the form of /dev/dsk/c?t?d?s2.

prtvtoc output

prtvtoc output

trinity$ prtvtoc /dev/rdsk/c0t1d0s2* /dev/rdsk/c0t1d0s2 partition map** Dimensions:* 512 bytes/sector* 80 sectors/track* 19 tracks/cylinder* 1520 sectors/cylinder* 3500 cylinders* 2733 accessible cylinders** Flags:* 1: unmountable* 10: read-only* First Sector Last* Partition Tag Flags Sector Count Sector Mount Directory

0 2 00 0 62320 623191 3 01 62320 197600 2599192 5 01 0 4154160 41541596 4 00 259920 3894240 4154159

trinity$ prtvtoc /dev/rdsk/c0t1d0s2* /dev/rdsk/c0t1d0s2 partition map** Dimensions:* 512 bytes/sector* 80 sectors/track* 19 tracks/cylinder* 1520 sectors/cylinder* 3500 cylinders* 2733 accessible cylinders** Flags:* 1: unmountable* 10: read-only* First Sector Last* Partition Tag Flags Sector Count Sector Mount Directory

0 2 00 0 62320 623191 3 01 62320 197600 2599192 5 01 0 4154160 41541596 4 00 259920 3894240 4154159

Creating partitions [format]

Creating partitions [format]

The format utility is used to define disk partitions.

trinity$ formatSearching for disks...doneAVAILABLE DISK SELECTIONS:

0. c0t1d0 <CDC Wren IV 94171-344 cyl 1545 alt 2 hd9 sec 46>/sbus@1,f8000000/esp@0,8000000/sd@1,01. c0t3d0 <SUN0207 cyl 1254 alt 2 hd 9 sec 36>/sbus@1,f8000000/esp@0,8000000/sd@3,0

Specify disk (enter its number): 1

The format utility can only be run with root privileges and is a suite of programs which allow the administrator to carry out a range of functions on the disk. When format is run it will identify all disks known to the system. If the new disk is not listed by format then there is no entry in /dev and /devices. A reconfiguration boot will be required.

The format Main Menu

The format Main Menu

The format main menu offers a range of functions.

FORMAT MENU:disk - select a disktype - select (define) a disk typepartition - select (define) a partition tablecurrent - describe the current diskformat - format and analyze the diskrepair - repair a defective sectorlabel - write label to the diskanalyze - surface analysisdefect - defect list managementbackup - search for backup labelsverify - read and display labelssave - save new disk/partition definitionsinquiry - show vendor, product and revisionvolname - set 8-character volume namequit

format> partition

Menu choices can be abbreviated to two letters. The partition option is used to partition the disk selected in in the previous menu.

The partition Menu

The partition Menu

This menu allows the modification of individual partitions or the entire disk

PARTITION MENU:0 - change '0' partition1 - change '1' partition2 - change '2' partition3 - change '3' partition4 - change '4' partition5 - change '5' partition6 - change '6' partition7 - change '7' partitionselect - select a predefined tablemodify - modify a predefined tablename - name the current tableprint - display the current tablelabel - write partition map and label to the diskquit

format>

The partition menu offers the following functions :-0-7 Specify the Tag, Flag, Offset and Size of the partition.select Select a pre-defined partition table.modify Modify a pre-defined partition table.name Name the current partition.print Display the current partition table.label Write the current table to the disk VTOC.

Partitioning a disk using modify


Preparing a new partition table.partition> modifySelect partitioning base:

0. Current partition table (original)1. All Free Hog

Choose base (enter number) [0]? 1Part Tag Flag Cylinders Size Blocks

0 root wm 0 0 (0/0/0) 01 swap wu 0 0 (0/0/0) 02 backup wu 0 - 1253 198.39MB (1254/0/0) 4062963 unassigned wm 0 0 (0/0/0) 04 unassigned wm 0 0 (0/0/0) 05 unassigned wm 0 0 (0/0/0) 06 usr wm 0 0 (0/0/0) 07 unassigned wm 0 0 (0/0/0) 0

Do you wish to continue creating a new partitiontable based on the above table[yes]? yesFree Hog partition[6]? 7

The modify option can only be used on an unmounted disk. It allows you to specify the size of each partition on the disk (except partitions 2 and the Free Hog partition) but it will not allow you to change either the Tag or the Flag.The Free Hog partition is used to store all the disk space and this contracts as space is allocated to each partition.In the above example we will be creating three partitions as follows :-

Partition 0 50 MbytesPartition 1 125 MbytesPartition 7 The rest (Free Hog)


Dividing the disk into three partitions.Enter size of partition '0' [0b, 0c, 0.00mb]: 50mbEnter size of partition '1' [0b, 0c, 0.00mb]: 125mbEnter size of partition '3' [0b, 0c, 0.00mb]: <return>Enter size of partition '4' [0b, 0c, 0.00mb]: <return>Enter size of partition '5' [0b, 0c, 0.00mb]: <return>Enter size of partition '6' [0b, 0c, 0.00mb]: <return>

Part Tag Flag Cylinders Size Blocks0 root wm 0 - 316 50.15MB (317/0/0) 1027081 swap wu 317 - 1107 125.14MB (791/0/0) 2562842 backup wu 0 - 1253 198.39MB (1254/0/0) 4062963 unassigned wm 0 0 (0/0/0) 04 unassigned wm 0 0 (0/0/0) 05 unassigned wm 0 0 (0/0/0) 06 usr wm 0 0 (0/0/0) 07 unassigned wm 1108 - 1253 23.10MB (146/0/0) 47304

Okay to make this the current partition table[yes]? <return>Enter table name (remember quotes): "Big Al"Ready to label disk, continue? yespartition>

The modify option takes care of sizes and offsets and ensures there are no overlapping or unused space. Sizes can be specified in either Blocks, Cylinders or Mbytes and modify will always round up or down to the nearest cylinder boundary.The table name is optional and can be up to eight characters in length. If you want to include spaces then the name must be in quotes.The label option at the end of modify writes the new partition table back to the disk.

File Systems

File Systems

File systems are ways of organising the way data is stored on disks

The default file system in Solaris is ufs

Another popular file system is Veritas’ journaling VxFS Journaling file systems often provide better resilience

and faster recovery than non-journaling file systems Journals should be stored somewhere sensible

- Journal in swap may be fast, but does not survive a reboot, thus useless.

Network File Systems

Network File Systems

Network file systems allow access to storage not physically attached to the machine

They use the network to access the data

Types of network file systems include NFS CIFS SMB

While useful, network file systems can cause problems NFS timeouts on remotely mounted drives can lead to

machines hanging

Media errors

Media errors

Errors occur on all media Disk, tape, CDROM Disk errors are reported to the console window Error messages are also logged in the file /var/adm/messages

Error for command 'read(10)' Error Level: RetryableRequested Block 2422272, Error Block: 2422283Sense Key: Media ErrorVendor 'SEAGATE':ASC = 0x11 (unrecovered read error), ASCQ = 0x0, FRU = 0x0Error for command 'read(10)' Error Level: RetryableRequested Block 2422272, Error Block: 2422283Sense Key: Media ErrorVendor 'SEAGATE':ASC = 0x11 (unrecovered read error), ASCQ = 0x0, FRU = 0x0

Checking and fixing file systems [fsck]

Checking and fixing file systems [fsck]

The fsck command is used to check the integrity of file systems

It can also be used to interactively repair inconsistencies

fsck [-F FSType] [-m] [-V] [special...]

fsck [-F FSType] [-n | N | y | Y] [-V] [-o FSType-specific-options] [special...]

NAME fsck - check and repair file systems

SYNOPSIS fsck [-F FSType] [-m] [-V] [special...]

fsck [-F FSType] [-n | N | y | Y] [-V] [-o FSType- specific-options] [special...]

DESCRIPTION fsck audits and interactively repairs inconsistent file system conditions. If the file system is inconsistent the default action for each correction is to wait for the user to respond yes or no. If the user does not have write permission fsck defaults to a no action. Some corrective actions will result in loss of data. The amount and severity of data loss can be determined from the diagnostic output.

Backup: Principles

Backup: Principles

Backups must be rigorously performed Follow a well defined strategy

Restore backups regularly Check that backups are being written to the media

correctly

Implement a disaster recovery plan Test out a disaster recovery (if possible) Well documented, preferably with an off-site copy

Backup: Principles

Store backup media properly Keep away from any electro-magnetic fields Store off-site if possible

Prevent accidental erasure of a backup Enable write-protect on backups Label backups clearly Well documented rotation strategy

Clean backup systems Keep tape heads clean

Backup Commands: tar

Backup Commands: tar

tar is a universal Unix command, usually used to back up files and directories

[-c] create a tar archive [-r] append files to archive [-t] list files in archive [-u] add files not in archive [-x] extract files from archive [-f filename] send to device at filename or

create tar file called filename

tar [crtux] [bflLmovw] filename

The tar command is commonly used to backup files and directories. It can be used by any user and is relatively simple in it’s syntax. In general tar is very portable between different platforms and Operating Systems but has the drawback of not being able to handle multiple-volumes.When using tar , files and directories are all compacted into one single file (a tar file) and this is specified with the -f filename option. Normally this is a device, such as /dev/rmt/0, but can also be a file. This is very useful if you wish to email a number of files and directories to another person. By creating a single tar file it is much easier to send.

Backup Commands: tar examples

Backup Commands: tar examples

To create a tar archive.

To extract a single file or the whole backup.

To list the contents of the archive.

trinity$ cd /export/home/morpheustrinity$ tar -cvf /dev/rmt/0 .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

trinity$ tar -xvf /dev/rmt/0 ./games. . . . . . . . . . . . . . . . . trinity$ tar -xvf examples.tar. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . .

trinity$ tar -tvf /dev/rmt/0. . . . . . . . . . . . . .

Take care when using tar to always specify the relative pathname of the files and directories to be archived. This will ensure that the archive is portable and may be restored. By using an absolute pathname when writing the archive, the files can only be extracted into exactly the same directory path. This tends to limit the portability of the files being archived.

Backup Commands: cpio

Backup Commands: cpio

cpio is commonly used in command pipelines.

cpio supports multi-volume backups.

[-i] input (read) [-o] output (create) [-t] list

cpio -[iot] [pvdmBH]

Although not as syntactically straightforward as tar, cpio is a powerful command for storage and retrieval. There are many additional options for cpio which are listed in the man pages.cpio’s major distinction from tar is that it allows the files to be archived to be specified on the standard input. Consequently, cpio especially useful when used in conjunction with commands that generate file names. For example, it is common for cpio to be used in conjunction with find to selectively backup files. In the following example, all files owned by fred are archived.

find . -user fred -print | cpio -o > /dev/rmt/0

Backup Commands: cpio examples

Backup Commands: cpio examples

To create a cpio backup.

To restore from a cpio backup.

To list the contents of the backup

trinity$ find . -print | cpio -ovB > /dev/rmt/0. . . . . . . . . . . . . . . . . . .

trinity$ cpio -ivdmB < /dev/rmt/0. . . . . . . . . . . . . . . . . .trinity$ cpio -ivdmB “fil*” < /dev/rmt/0. . . . . . . . . . . . . . . . . .

trinity$ cpio -itvB < /dev/rmt/0. . . . . . . . . . . . . . . . . .

Backup Commands: dd

Backup Commands: dd

dd provides access to raw devices

It may be used to tar to and from a remote machine ...

... or to convert file formats

dd [option=value]

trinity$ tar cvf - . | rsh smith dd of=/dev/rmt/0. . . . . . . . . . . . . . . . . . . . . . . .trinity$ rsh smith dd if=/dev/rmt/0 | tar xvBpf -. . . . . . . . . . . . . . . . . . . . . . . .

trinity$ dd if=/dev/rmt/0 conv=swab | tar xvf -

The dd command stands for copy and convert and is literally a bit-copier. For this reason it is useful when converting data between raw physical devices, and for copying the entire contents of one disk to another disk or for duplicating tapes.It is not usually used for backup purposes as it is slow and provides little or no verbose output. However, when combined with utilities such as tar, it supports remote backups.In the final example above dd is used to convert the byte ordering of the data on tape, before passing this data to tar for extraction. Such a command would be useful when reading tapes on a Sun system which had been written on a byte swapped machine such as those produced by Silicon Graphics.The ‘copy-and-convert’ utility is called ‘dd’ rather than ‘cc’, because when it was written for Unix ‘cc’ had already been claimed by the C compiler. ‘dd’ was the next best thing!dd is particularly useful for making backups of CDs to ISO9660 image files.

Backup Commands: mt

Backup Commands: mt

mt provides controls for magnetic tape drives

fsf forward space over count EOF marks bsf backward space over count EOF marks rewind rewind the tape retention rewind and wind the tape forward erase erase the entire tape

mt [-f tape_name] command [count]

# mt -f /dev/rmt/0 rewind# mt -f /dev/rmt/0n fsf 2 ; ufsrestore -rvf /dev/rmt/0n#

The /usr/bin/mt command is used to position and control a magnetic tape. The mt command is useful when several archives (for example, ufsdump, tar, cpio) must share the same tape. mt allows early archives to be scanned past, so that subsequent backups appear later on the tape.Note that in order for several backups to share a tape it is important that the device is identified by its ‘no - rewind’ name. /dev/rmt/0n tells the device driver controlling access to the tape drive not to rewind when the device file is closed.

solaris fundamentals

Documents

unix lore

unix philosophy

unix flavours

resulting work

thoughtful solutions

licence terms

licence identical

files processes