net_py

Chapter II

net_py

Reliable or Unreliable

Do you need service that keeps packets in order and guarantees delivery?

Are your packets “individual” and so can arrive in any order or possibly even not at all?

In the above example, you would only need to worry about non-delivery (and hence non-reply).

Example: Suppose you plan to send a single packet and get a single reply?

net_py

UDP

Short, self-contained requests and responses.

Real-time traffic like voice.

Author says it is not used often but that is not to say it is not useful.

A single server port can receive packets from millions of distinct clients over time with no additional memory allocation beyond original setup.

congested network routers tend to be more sympathetic to UDP traffic compared to TCP traffic since they know that the latter will be retransmitted if dropped by the router.

You can add your own “99% reliability” more cheaply (performance) than using TCP's 100% reliability.

easier to use

net_py

Addresses and Ports

UDP

IP

Ethernet

running program

socket memory allocation

port number

file descriptor

net_py

Addresses and Ports

Ports allow multiple programs to use the same TCP/IP stack

Packets come in and go up the same stack. They are “demultiplexed” at the top of the stack by port number to different programs.

The pair (IP address:port number) is called a socket and identifies a program connected to the internet, running on some machine.

Every packet sent on the internet contains a quadruple

that identifies the connection.

Example: www.mysite.com:8080

(sourceIP: source port, destIP: dest port)

net_py

Port Ranges

Well-known: 0-1023

Registered: 1024-4915: used by large companies

The Rest: above, used by us

How to find a well-known or registered port number

Check out the IANA (www.iana.org)

>>> import socket>>> socket.getservbyname('domain')

net_py

Python Virtual Environment

● It is a good idea to create your own virtual environment for the various python programming projects found in this course.

● This gives you an easy way to install special python packages just for your own use and leave the main python install on your computer alone.

● This is a useful skill for future python development you might do.

● Here is an on-line tutorial; read it carefully and follow the instructions.

http://dabapps.com/blog/introduction-to-pip-and-virtualenv-python/http://virtualenvwrapper.readthedocs.org/en/latest/

net_py

Using virtualenvwrapper

● The virtualenv tutorial suggests that inside each project directory (say Project1) there should be a project-specific virtual environment (Project1/env).

● There is an alternative approach suggested by virtualenvwrapper – an extension of virtualenv.

● When using using virtualenvwrapper your file hierarchy looks like:

● All the executable code in .virtualenvs is from the wrapper package.

Projects:ArcGIS BenchMark GeoIP Memcached QGIS Rabbit

.virtualenvs:ArcGIS hook.log.1 postmkvirtualenv premkvirtualenvBenchmark initialize postrmproject prermprojectBenchMark Memcached postrmvirtualenv prermvirtualenvGeoIP postactivate preactivate QGISget_env_details postdeactivate predeactivate Rabbithook.log postmkproject premkproject README

net_py

Sockets

Python makes calls to the lower level operating system-level calls that implement the networking functionality

The python interface is slightly OO

Python's contact to these OS calls is through a program construct called a socket

Sockets are “files” just like everything else and are accessed via a file descriptor, which in python you never see but in C it is all you see.

Sockets are the file descriptor

We read and write to sockets the same way we do files

net_py

import socket, syss = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # AF_INET is a “family” type # SOCK_DGRAM means UDP and not TCP

MAX = 65535 # packet sizePORT = 1060 # port used by our server

if sys.argv[1:] == ['server']: # are these two lists equal s.bind(('127.0.0.1', PORT)) # specify your local socket address # since you are a server print 'Listening at', s.getsockname() # returns a list while True: data, address = s.recvfrom(MAX) # wait for traffic print 'The client at', address, 'says', repr(data) # data.toString() s.sendto('Your data was %d bytes' % len(data), address) # send reply to original sender

net_py

elif sys.argv[1:] == ['client']: print 'Address before sending:', s.getsockname() # 0.0.0.0:0 means no port on any interface s.sendto('This is my message', ('127.0.0.1', PORT)) print 'Address after sending', s.getsockname() # bind is automatic, but only to a port number # 0.0.0.0:34567 means port number 34567 on any interface data, address = s.recvfrom(MAX) # overly promiscuous - see Chapter 2 # will accept a datagramfrom anyone and not just the server print 'The server', address, 'says', repr(data)

else: print >>sys.stderr, 'usage: udp_local.py server|client'

net_py

elif sys.argv[1:] == ['client']: print 'Address before bind:', s.getsockname() s.bind('',55000) print 'Address after bind:', s.getsockname() s.sendto('This is my message', ('127.0.0.1', PORT)) print 'Address after sending', s.getsockname() # bind is automatic, but only to a port number # 0.0.0.0:55000 means port number 55000 on any interface data, address = s.recvfrom(MAX) # overly promiscuous - see Chapter 2 # will accept a datagramfrom anyone and not just the server print 'The server', address, 'says', repr(data)

else: print >>sys.stderr, 'usage: udp_local.py server|client'

net_py

Ports

Remote access to a socket is through its port number.

Access from the program that opens the socket is through the file descriptor.

net_py

Socket

Author says “both IP address and port number start as all zeros – a new socket is a blank slate”. This is not entirely true.

recv() vs recvfrom():

recv(): does not return sender address and typical of client code because clients typically go to a single server and so “know” where any data they receive comes from.

recvfrom() returns sender address and is typical of server code because servers receive data from many clients and typically need to reply to a client request with a packet sent right back to the same client. The socket.sendto() function takes the address returned by recvfrom().

net_py

Congestion:

If you timeout and resend and the problem is that the server is down, you will add useless traffic to the network and cause congestion that will slow everyone down.

Best answer is each time you timeout extend the timeout interval so that eventually you are resending a packet only once per hour or more.

exponential backoff: delay *= 2

What about giving up?

What about trying forever?

best to give up only if you need a timely answer or not at all

weather icon example

net_py

More Complications:

Suppose a successful request/reply exchange takes 200 ms on average.

What does the server do with duplicate requests?

In both cases the server must keep track of request identifiers so it knows what requests have already been received.

If client also keeps track of request IDs and whether or not they have been replied to then the client can quietly drop duplicate replies.

Client can use this information to set a minimum timeout delay to be at least 200 ms?

1) Reply again2) Don't bother to repeat reply (How does the server know a previous reply got to the original sender?).

net_py

reply lost

no need to pass offto higher level but must keep data ID to know it has already been forwarded

optional

net_py

Reliable Code

#!/usr/bin/env python# Foundations of Python Network Programming - Chapter 2 - udp_remote.py# UDP client and server for talking over the networkimport random, socket, syss = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)MAX = 65535PORT = 1060# usage: udp_remote.py server [ <interface> ]# usage: udp_remote.py server [ <interface> ]if 2 <= len(sys.argv) <= 3 and sys.argv[1] == 'server': # server side interface = sys.argv[2] if len(sys.argv) > 2 else '' # two single quotes s.bind((interface, PORT)) # '' is the same as '0.0.0.0' == all interfaces print 'Listening at', s.getsockname() while True: data, address = s.recvfrom(MAX) # server always needs client address if random.randint(0, 1): # flip a coin and reply only if heads print 'The client at', address, 'says:', repr(data) s.sendto('Your data was %d bytes' % len(data), address) else: # tails print 'Pretending to drop packet from', address

net_py

Reliable Code

# Usage: udp_remote.py client <host>elif len(sys.argv) == 3 and sys.argv[1] == 'client': hostname = sys.argv[2] s.connect((hostname, PORT)) # no packets exchanged; print 'Client socket name is', s.getsockname() delay = 0.1 # initial retry delay while True: # keep resending if no reply s.send('This is another message') print 'Waiting up to', delay, 'seconds for a reply' s.settimeout(delay) # so recv() will block only so long try: data = s.recv(MAX) # blocking <delay> seconds except socket.timeout: # timeout expired delay *= 2 # double timeout delay time if delay > 2.0: # time to stop all this nonsense raise RuntimeError('I think the server is down') else: break # we are done after one success, # and can stop looping print 'The server says', repr(data)

net_py

Code Alternatives:

• See homework

net_py

Connecting vs Implicit Connecting

• In Listing 2-1 we used sendto(msg,address) on the client and an implicit binding happened when the first datagram was sent.

• In Listing 2-2 we used send(msg) so the binding had to be done explicitly, and at the same time we indicate where the message is to be sent

• In the first situation we could send to various different servers by modifying the address argument. In the latter we can only send to the address we originally connected to.

s.connect(remote_host,remote_PORT)...s.send(data) check netstat -an | grep udp

at this point

net_py

How things look after s.sendto(msg,address):

[pletcha@archimedes ~]$ python>>> import socket>>> s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)>>> s.sendto('my message',('wyvern.cs.newpaltz.edu',50000))10>>> s.getsockname()('0.0.0.0', 52011)

[pletcha@archimedes ~]$ netstat -an | grep udpudp 0 0 0.0.0.0:55188 0.0.0.0:* udp 0 0 0.0.0.0:47503 0.0.0.0:* udp 0 0 0.0.0.0:52011 0.0.0.0:* udp 0 0 0.0.0.0:60386 0.0.0.0:* udp 0 0 192.168.122.1:53 0.0.0.0:*

anyone can send data to my port 52011.

net_py

Sneaking into the conversation

[pletcha@archimedes ~]$ python>>> import socket>>> s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)>>> s.sendto('my message',('wyvern.cs.newpaltz.edu',50000))10>>> s.getsockname()('0.0.0.0', 33897)>>> data,address = s.recvfrom(4000)>>> print dataFake reply

[pletcha@archimedes ~]$ python

>>> import socket>>> s = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)>>> s.sendto('Fake reply',('127.0.0.1',33697))10

use the bound port numberas your destination

net_py

How things look after s.connect(address):

> python>>> import socket>>> s =socket.socket(socket.AF_NET,socket.SOCK_DGRAM)>>> s.connect((wyvern.cs.newpaltz.edu,50000))

[pletcha@archimedes ~]$ netstat -an | grep udpudp 0 0 0.0.0.0:55188 0.0.0.0:* udp 0 0 137.140.8.104:51400 137.140.4.187:50000 ESTABLISHEDudp 0 0 0.0.0.0:47503 0.0.0.0:* udp 0 0 0.0.0.0:60386 0.0.0.0:* udp 0 0 192.168.122.1:53 0.0.0.0:* no one can send data

to my port 51400 exceptwhat I connected to(I really didn't connectsince no data was sent)

net_py

Exercise

•

• Lesson to learn: If you want to use sendto() instead of connect() followed by send(), then you can use recvfrom() and look at each sender address to be sure it is an address you recognize.

• Exercise: Think of a UDP application that would have you sending data to several destinations so you could expect answers back from them all.

Repeat the “sneak into the conversation” example running the client softwareof Listing 2-2 and see that your message never gets read by the application.It is rejected by UDP

What happens is that Ethernet and IP recognize the sneaky packet as intendedfor this machine. IP forwards the packet to UDP. UDP looks up the destinationport and sees that it will only accept data from (wyvern, 50000)

net_py

Request IDs

• Sending a packet ID with every packet makes it easier to identify replies and know what request they are replying to.

• (See homework)

• Packet IDs are some protection against spoofing. Client that doesn't call connect() can be exposed to unexpected (spoofing) traffic. If the unwelcome traffic doesn't know the packet ID it is not possible to fake a response.

net_py

Binding to Interfaces

• We have used bind() to listen on 127.0.0.1 or on ' ', which means all network interfaces.

• We can specify an interface if we know its IP address (remember /sbin/ifconfig -a)

• Connecting to the College using VPN I have 4 network interfaces on my laptop

net_py

4 Network Interfaces at once:

[pletcha@archimedes 02]$ ifconfig -acscotun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST> mtu 1399 inet 137.140.108.133 netmask 255.255.255.224 destination 137.140.108.1 ...

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 ...

virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255 ...

wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.124 netmask 255.255.255.0 broadcast 192.168.1.255 ...

net_py

Ports are bound to Interfaces

[pletcha@archimedes 02]$ cat all_interfaces.sh #!/usr/bin/bash

./udp_remote.py server 127.0.0.1&sleep 1; ./udp_remote.py server 137.140.108.133&sleep 1; ./udp_remote.py server 192.168.122.1&sleep 1; ./udp_remote.py server &sleep 1; ./udp_remote.py server 192.168.1.124&

[pletcha@archimedes 02]$ ./all_interfaces.sh Listening at ('127.0.0.1', 1060)Listening at ('137.140.108.133', 1060)Listening at ('192.168.122.1', 1060)Traceback (most recent call last): File "./udp_remote.py", line 13, in <module> s.bind((interface, PORT)) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args)socket.error: [Errno 98] Address already in use[pletcha@archimedes 02]$ Listening at ('192.168.1.124', 1060)

net_py

local host vs remote host:

• Local packets can arrive destined for 127.0.0.1 but they can also arrive locally by using the machine IP address, say 192.168.122.1.

• By binding to 127.0.0.1, external clients can not talk to you.

• By binding to 192.168.122.1 both internal and external clients can talk to you.

• Lesson to Learn: Binding means specifying both a port number and a network interface (or all interfaces), so the basic data structure is not just a port but an (IP address, port) pair, in other words, a socket.

net_py

Two Interface problem:

• Suppose a machine has two external interfaces – 192.168.122.1 and 192.168.1.124.

• If we open a socket at (192.168.1.124,1060) but try to send data to (192.168.122.1, 1060) what will happen? Apparently it depends on the OS.

• On my laptop I get:

[pletcha@archimedes 02]$ ps -ef | grep remotepletcha 28577 3450 0 18:07 pts/1 python ./udp_remote.py server 192.168.122.1

[pletcha@archimedes 02]$ ./udp_remote.py client 192.168.1.124Client socket name is ('192.168.1.124', 48271)Waiting up to 0.1 seconds for a replyTraceback (most recent call last): File "./udp_remote.py", line 33, in <module> data = s.recv(MAX)socket.error: [Errno 111] Connection refused

net_py

Fragmentation, ETC:

Read the last 4 pages of Chapter 2 yourselves.