network programming intro to distributed systems fall 2013

NETWORK PROGRAMMINGINTRO TO DISTRIBUTED SYSTEMSFALL 2013

L3-socketDongsu Han

Some material taken from publicly available lec-ture slides including Srini Seshan’s and David Anderson’s

Today’s Lecture Lecture

Transport protocols: UDP and TCP Version control using SVN

TA-led programming session Editor, socket programming Debugging with gdb

Announcement Programming Assignment 1 is out. (Due in two weeks-- 추석 )

Review of Previous Lectures What is a distributed system? (L1)

Definition Example

What are some of the important decisions made in the Inter-net architecture? (L2) Narrow waist: Network provides minimal functionality (best-effort)

Why?: to connect existing networks that are heterogeneous Stateless architecture: no per-flow state inside network

Why? For survivability

4

Transport Protocols

Lowest level end-to-end proto-col. Header generated by sender is in-

terpreted only by the destination Routers view transport header as

part of the payload; they are not supposed to see the transport header.

7

6

5

7

6

5

Transport

IP

Datalink

Physical

Transport

IP

Datalink

Physical

IP

router

2 2

1 1

What does the transport layer do? No per-flow (connection) state in the network

Transport layer maintains the connection state. Transport layer identifies an end-to-end connection.

How does the transport layer identify an end-point (ap-plication)?

De-multiplexing: Port numbers Network layer gets you to the host. A port identifies the sending or receiving socket (appli-

cation). A socket is one endpoint of a two-way communication

between two programs running on the network. A socket is bound to a port number so that the transport layer can identify the application that data is destined to be sent.

Identifying two end-points: An endpoint = <IP address, Port #> Source endpoint = <SrcIPAddr, SrcPort> destination endponit = <DestIPAddr, DestPort>

7

Protocol and Port Demultiplexing Multiple choices at each layer Applications open a socket (identified by the port

number)

Port 80 Port 22 Port 22Port 80

TCP UDP

IP

NET1 NET2 NETn…

TCP/UDPIP

Port Number= 80

Network

Protocol Field= TCP

Type Field=IP

8

Server and Client

TCP/UDP

IP

Ethernet Adapter

Server

TCP/UDP

IP

Ethernet Adapter

Clients

Server and Client exchange messages over the net-work through a common Socket API

Socket API

hardware

kernel space

user spaceports

Transport protocols UDP

Provides only integrity and de-multiplexing TCP adds…

10

UDP: User Datagram Protocol [RFC 768]

“No frills,” “bare bones” Internet transport protocol, “Best effort” service

UDP segments may be: Lost, duplicated (bug, spurious retrans-

mission at L2) Delivered out of order to app

Connectionless: No handshaking between UDP sender,

receiver Each UDP segment handled indepen-

dently

Why is there a UDP?• No connection establishment

(which can add delay)• Simple: no connection state

at sender, receiver• Small header• No congestion control: UDP

can blast away as fast as desired

11

UDP, cont. Often used for stream-

ing multimedia apps (e.g., Skype) Loss tolerant Rate sensitive

Other UDP uses (why?): DNS

Reliable transfer over UDP Must be at application

layer Application-specific er-

ror recovery

Source port # Dest port #

32 bits

Applicationdata

(message)

UDP segment format

Length ChecksumLength, in

bytes of UDPsegment,includingheader

Transport protocols UDP

Provides only integrity and de-multiplexing TCP adds…

Connection-oriented Reliable (error control) Ordered byte stream (in-order delivery) Bi-directional byte-stream Flow and congestion controlled

TCP Protocol implemented entirely at the ends

Fate sharing Abstraction provided

Connection oriented protocol (hand-shake or connection establish-ment)

In-order byte-stream Implemented functionality (next lecture)

Flow control Reliability (error control) Reassembly (in-order delivery) Congestion control

Difference in abstraction UDP

Recv(len) call gives you a packet (the payload part)

TCP Connection establishment (hand-shake) Will see in the TA

led session. Recv(len) call give you a sequence of bytes

UDP segment (packet)

Byte stream

UDP segment (packet)

len len

Incoming stream

ApplicationRecv()

len len

Byte stream

Applicationsend()

TCP– What goes on underneath

Byte stream Packet (1~7) Packet (8~10)

Receive buffer

Packet (9~11)

Packet (8~10)

I need seq# 8.

16

User Datagram Protocol(UDP): An Analogy

Example UDP applicationsMultimedia, voice over IP

UDP Single socket to receive mes-

sages No guarantee of delivery Not necessarily in-order delivery Datagram – independent packets Must address each packet

Postal Mail Single mailbox to receive letters Unreliable Not necessarily in-order delivery Letters sent independently Must address each reply

Transmission Control Pro-tocol (TCP): An Analogy

TCP Reliable – guarantee delivery Byte stream – in-order delivery Connection-oriented – single

socket per connection Setup connection followed by

data transfer

Telephone Call Guaranteed delivery In-order delivery Connection-oriented Setup connection followed by

conversation

Example TCP applicationsWeb, Email, Telnet

REVISION CONTROL & MAKE

Overview Revision control systems

Motivation Features Subversion primer

Make Simple gcc Make basics and variables Testing

Useful Unix commands

Dealing with large codebases Complications

Code synchronization Backups Concurrent modifications

Solution: Revision Control System (RCS) Store all of your code in a repository on a server Metadata to track changes, revisions, etc… Also useful for writing books, papers, etc…

Basic RCS features (1/2) Infinite undo

go back to previous revisions Automatic backups

all your code ever, forever saved Changes tracking

see differences between revisions

Basic RCS features (2/2) Concurrency

Multiple people working at the same time Snapshots

Save stable versions on the side (i.e. handin) Branching

Create diverging code paths for experimentation

Typical RCS workflow

1. Create repository on RCS server2. Checkout the repository to your local machine3. Work locally: create, delete and modify files4. Update the repository to check for changes other

people might have made5. Resolve any existing conflicts6. Commit your changes to the repository

RCS implementations Revision Control System (RCS)

Early system, no concurrency or conflict resolution Concurrent Versions System (CVS)

Concurrency, versioning of single files Subversion (SVN)

File and directory moving and renaming, atomic commits

Concurrent edits (1/2)

Carnegie MellonFile v1

Both check out file

Edits (lines 1-20)

File v2

Edits (lines 30-40)

commit

Update

Merged automatically!

Concurrent edits (2/2)

Carnegie MellonFile v1

Both check out file

Edits (lines 1-20)

File v2

Edits (lines 10-20)

commit

Update

Conflict needing manual in-tervention!

Resolving conflicts When changes can’t be merged automatically

SVN gives you 3 files: file.mine : your file file.rx : the remote file file : original file with marked conflicts

You can Discard your changes Discard others’ changes Go over each conflict and arbitrate as appropriate

Interacting with SVN Command line

Readily available in andrew machines Graphical tools

Easier to use Subclipse for Eclipse gives IDE/SVN integration tortoiseSVN

실습 server IP: 143.248.141.52 ~ 143.248.141.59 (8 대 ) ID: your student id (e.g., 20110111) Password: ee324EE#@$

Svn password 20060584 28482 20071122 6534 20081275 51085 20090058 50346 20090276 54586 20100120 23860 20100137 29297 20100441 17889 20100827 30690 20100967 7532 20110343 4967 20110456 9139 20110773 8574 20111047 36671

Command line SVN example (1/2)

$ svn co –-username=“<<student id>>” https://ina.kaist.ac.kr/svn/test

$ cd test/trunk

$ vim <<student id>>.txt$ vim <<student id>>.$ svn add <<student id>>.txt

$ svn commit -m 'adding a text file with my student id‘$ svn update

$ cd ../$ svn cp trunk tags/<<student id>>_submission$ svn commit -m 'making a tag of the trunk for submission!‘

$ echo -e ”<<student id>>" >> student.txt$ svn commit –m “added my student id”

Command line SVN example (2/2)

Revision control lets you note (and then see) what you changed:

> svn log student.txtr986 | ntolia | 2006-08-01 17:13:38 -0400 (Tue, 01 Aug 2006) | 6 lines

This allows the sp to get rid of chunks early before a transfer is complete.Useful when a file is requested in-order and the file size > mem cache size

> svn diff -r 1:2 student.txtIndex: file===================================================================--- file (revision 1)+++ file (revision 2)@@-1,2+1,3@@ This isatestfile -It startedwithtwolines +It nolongerhastwolines +it hasthree

General SVN tips Update, make, test, only then commit Merge often (svn up) Comment commits Avoid commit races Modular design avoids conflicts

Know more

subversion.tigris.org for SVN software & info svnbook.red-bean.com for SVN book

Makefile You are on your own.

Make Utility for executable building automation Saves you time and frustration Helps you test more and better

Simple gcc

If we have files:• prog.c: The main program file• lib.c: Library .c file• lib.h: Library header file

% gcc -c prog.c -o prog.o% gcc -c lib.c -o lib.o% gcc lib.o prog.o -o binary

gcc flags

• Useful flags1. -g: debugging hook2. -Wall: show all warnings3. -Werror: treat warning as errors4. -O0, -O1, -O2, -O3: optimization level5. -DDEBUG: macro for DEBUG (#define DEBUG)

• Avoid using dangerous optimizations that could af-fect correctness

More gcc% gcc -g -Wall -Werror -c prog.c -o prog.o% gcc -g -Wall -Werror -c lib.c -o lib.o% gcc -g -Wall -Werror lib.o prog.o -o binary

This gets boring, fast!

Makefile basics • Build targetstarget: dependency1 dependency2 ...

unix command (start line with TAB)unix command

binary: lib.o prog.ogcc -g -Wall lib.o prog.o -o binary

lib.o: lib.cgcc -g -Wall -c lib.c -o lib.o

prog.o: prog.cgcc -g -Wall -c prog.c -o prog.o

clean:rm *.o binary

Makefile example

Makefile variables (1/7)

• VariablesCC = gcc

CFLAGS = -g -Wall -WerrorOUTPUT = binary

binary: lib.o prog.ogcc -g -Wall lib.o prog.o -o binary

lib.o: lib.cgcc -g -Wall -c lib.c -o lib.o

prog.o: prog.cgcc -g -Wall -c prog.c -o prog.o

clean:rm *.o binary


CC = gccCFLAGS = -g -WallOUTPUT = binary

$(OUTPUT): lib.o prog.o$(CC) $(CFLAGS) lib.o prog.o -o binary

lib.o: lib.c$(CC) $(CFLAGS) -c lib.c -o lib.o

prog.o: prog.c$(CC) $(CFLAGS) -c prog.c -o prog.o

clean:rm *.o $(OUTPUT)


CC = gccCFLAGS = -g -WallOUTPUT = binary

$(OUTPUT): lib.o prog.o$(CC) $(CFLAGS) lib.o prog.o -o binary





CC = gccCFLAGS = -g -WallOUTPUT = binaryOBJFILES = lib.o prog.o

$(OUTPUT): $(OBJFILES)$(CC) $(CFLAGS) $(OBJFILES) -o binary







%.o: %.c# $<: dependency (%.c)# $@: target (%.o)$(CC) $(CFLAGS) -c $< -o $@



Simple Test Script (1/2)% ./server 6667 &% cat testfile.01 | ./testscript.py% cat testfile.02 | ./testscript.py% killall -9 server

Simple Test Script (2/2)#/bin/sh

echo “Starting server on port 6667.”./server 6667 &SERVERPID = $!

echo “Running test files.”cat testfile.01 | ./testscript.pycat testfile.02 | ./testscript.py

echo “Killing server process.”kill $(SERVERPID)


all: $(OUTPUT)


%.o: %.c# $<: dependencies (%.c)# $@: target (%.o)$(CC) $(CFLAGS) -c $< -o $@


Augmenting the Makefile for testing (1/2)


all: $(OUTPUT) test


%.o: %.c# $<: dependencies (%.c)# $@: target (%.o)$(CC) $(CFLAGS) -c $< -o $@

test: $(OUTPUT)sh ./testscript.sh


Augmenting the Makefile for testing (2/2)

Using the Makefile% make% make test% make clean

• Know moreGoogle– “makefile example”– “makefile template”– “make tutorial”– Chapter 3 of Dave Andersen’s notes

Extra: useful Unix commands (1/2)

• find “func_name” in files% grep -r func_name .

• replace “bad_func_name” to “good_func_name”% sed -e “s/bad_func_name/good_func_name/g”\prog.c > prog.c.new

Extra: useful Unix commands (2/2)

• find a file named “prog.c”% find -name prog.c

• download files from the Internet% wget http://address/to/file.tar.gz

• untar and unzip a file% tar xzvf file.tar.gz

Assignment You have access to svn repository at

https://ina.kaist.ac.kr/svn/<student id> However, we will change your password (to some un-

known integer). Luckily, we made a server that gives you a hint. When your client asks whether the username and the

password match, the server can tell you whether the password provided is less than, equal to, or greater than your real password.

https://ina.kaist.ac.kr/svn/%3Cstudent

Assignment I Protocol is defined in the document. (See course web

page).Query

Response

Server (provided by the in-structor)Client

Part I: Assume the network is reliable. (due 9/23) Part II: The server may drop a packet. (due 9/27)

network programming intro to distributed systems fall 2013

Documents