june 21-25, 2004lecture 1: basic skills1 lecture 1 basic skills presenter name presenter institution...
TRANSCRIPT
June 21-25, 2004 Lecture 1: Basic Skills 1
Lecture 1Basic Skills
Presenter Name
Presenter InstitutionPresenter email address
Grid Summer Workshop June 21-25, 2004
June 21-25, 2004 Lecture 1: Basic Skills 2
Welcome! You are at the Grid Summer School
If you are looking for the underwater basket weaving classes, they are two block to the west.
Sponsored by: Center for Gravitational Wave Astronomy GriPhyN iVDGL NMI GRIDS Center
June 21-25, 2004 Lecture 1: Basic Skills 3
Goals for this week Learn about Grids
What is a grid? How do you…
Submit jobs? Transfer data? Cope with failure?
Where are Grids going? Use grids
Hands-on exercises Local and remote grids
Format of the week Morning
Lecture Exercises
Lunch! Mmm… Afternoon
Lecture Exercises
June 21-25, 2004 Lecture 1: Basic Skills 4
Day 1 Basic Skills What is a Grid? Basics of Using a Grid
June 21-25, 2004 Lecture 1: Basic Skills 5
Day 2 Running a real application on a grid
What requirements do real applications have? How should you build a real application?
Taking care of an application Condor-G DAGMan Staging your application
June 21-25, 2004 Lecture 1: Basic Skills 6
Day 3
June 21-25, 2004 Lecture 1: Basic Skills 7
Day 4
June 21-25, 2004 Lecture 1: Basic Skills 8
Day 5
June 21-25, 2004 Lecture 1: Basic Skills 9
Without further adieu...
June 21-25, 2004 Lecture 1: Basic Skills 10
Step 1: Basic Networking Skills We apologize in advance if you already know
these concepts. We want to make sure that there is a certain basic
level of understanding for all students.
June 21-25, 2004 Lecture 1: Basic Skills 11
IP Addresses All computers on the Internet use TCP/IP. All computers have at least one IP address.
32-bit number Written as four numbers, like: 128.105.3.61
An IP Address identifies a network interface, not a computer. A computer can have multiple IP addresses.
June 21-25, 2004 Lecture 1: Basic Skills 12
DNS DNS maps IP addresses to names, and vice-versa
www.amazon.com 207.171.163.30 Discover this with “host” or “nslookup” or “dig” Try all three—how do they differ?
June 21-25, 2004 Lecture 1: Basic Skills 13
Whois Who owns or runs a domain? Amazon.com: Amazon.com, Inc. (HOS276-ORG) [email protected]
PO BOX 81226
SEATTLE, WA 98108-1300
US
+1 206 266 4064 fax: +1 206 266 7010
Discover this with “whois”.
June 21-25, 2004 Lecture 1: Basic Skills 14
Ping! Are you awake? Is a computer on the network? Use ping to find out% ping www.cs.wisc.edu
PING marzipan.cs.wisc.edu (128.105.7.11) 56(84) bytes of data.
64 bytes from marzipan.cs.wisc.edu (128.105.7.11): icmp_seq=1 ttl=63 time=0.327 ms
64 bytes from marzipan.cs.wisc.edu (128.105.7.11): icmp_seq=2 ttl=63 time=0.241 ms
--- marzipan.cs.wisc.edu ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 2611ms
rtt min/avg/max/mdev = 0.241/0.276/0.327/0.036 ms
Note that a computercan have multiple names
June 21-25, 2004 Lecture 1: Basic Skills 15
Internet routes Between you and a computer on the network, there is
an often complex route. % traceroute www.cs.uwm.edu
traceroute to miller.cs.uwm.edu (129.89.143.24), 30 hops max, 40 byte packets
1 svi-121.cisco1.cs.wisc.edu.105.128.in-addr.arpa (128.105.121.248) 0.423 ms 0.242 ms 0.227 ms
2 rh-cssc-b280c-2-core-vlan-492.net.wisc.edu (144.92.128.186) 0.404 ms 4.985 ms 0.489 ms… snip…
6 r-uwmilwaukee-isp-atm1-0-1.wiscnet.net (140.189.8.2) 2.730 ms 2.603 ms 2.689 ms
7 space-needle-mke.csd.uwm.edu (216.56.1.194) 2.836 ms 2.718 ms 2.748 ms8 miller.cs.uwm.edu (129.89.38.24) 2.754 ms * 2.796 ms
June 21-25, 2004 Lecture 1: Basic Skills 16
Port numbers A port number indicates which program to talk to
on a computer. Some port numbers are standard:
HTTP (web): port 80 SMTP (mail): port 25 Ping: port 7
Some port numbers are assigned dynamically when you run a server.
June 21-25, 2004 Lecture 1: Basic Skills 17
Netstat Netstat can answer the question: is a program running on
a port on the local computer. netstat --protocol=inet –l
tcp 0 0 *:finger *:* LISTEN -l meant “listening for connections”. Look for active
connections:netstat --protocol=inet | grep ssh% netstat --protocol=inet | grep sshtcp 0 0 chopin.cs.wisc.edu:ssh ppp-67-38-160-108:20715 ESTABLISHED tcp 0 0 chopin.cs.wisc.edu:ssh 68.185.181.47:1176 ESTABLISHED …
June 21-25, 2004 Lecture 1: Basic Skills 18
Telnet Telnet isn’t just for remote access to a computer Telnet can tell you if some remote services are
running correctly. Is ssh running?
Find ssh port number in /etc/services. It’s 22. telnet <host> 22. Example:telnet beak.cs.wisc.edu 22
Trying 128.105.146.14...
Connected to beak.cs.wisc.edu (128.105.146.14).
Escape character is '^]'.
SSH-1.99-OpenSSH_3.6.1p2
^] (That is control-right bracket)
telnet> quit
June 21-25, 2004 Lecture 1: Basic Skills 19
Telnet & HTTP (Just for fun)% telnet www.cs.wisc.edu 80Trying 128.105.7.31...Connected to www.cs.wisc.edu (128.105.7.31).Escape character is '^]'.GET http://www.cs.wisc.edu/~roy/index.html<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"><html>
<head><title> Alain Roy</title>… snip …
I typed the lines in red, and got back a web page.
June 21-25, 2004 Lecture 1: Basic Skills 20
Why are we telling you this? We have a shameful secret:
Maintaining a grid is hard work. Things break. You will get involved in debugging them.
You can use all of these commands to figure out where a problem lies.
June 21-25, 2004 Lecture 1: Basic Skills 21
Example Bad complaint: Waaah!!! My grid server-thingy is
down Good complaint: The Globus gatekeeper isn’t
running, even though the computer is up. Is the computer up? Use ping Is the computer accessible on the network? Use traceroute Can you telnet to the gatekeeper?
June 21-25, 2004 Lecture 1: Basic Skills 22
Grid Without a Grid (GWAG) Now we’re going to learn how to do grid work
without using grid technology. What is a grid? We’ll tell you this afternoon. For
now: Run jobs on other computers Transfer data Discover information Do it all securely
Could we have thought of a worse acronym? GWAG?
June 21-25, 2004 Lecture 1: Basic Skills 23
GWAG: Security Explain OpenSSL Explain OpenSSH Explain HTTPS Note password authentication Describe ssh keygen stuff, to eliminate passwords.
Note how this has to be set up on a per-computer basis.
This needs to be fleshed out.
June 21-25, 2004 Lecture 1: Basic Skills 24
GWAG: Job submission ssh can run jobs remotely:
ssh [email protected] ls This expects a few things:
Your program already exists on the remote computer A shell is invoked to set up your environment, so you
can find ls.
June 21-25, 2004 Lecture 1: Basic Skills 25
GWAG: File Transfer How do you transfer data to be executed? scp executable [email protected]:.
Note the final period at the end of that line How do you transfer the output back? scp [email protected]:output . Submitting your job takes a minimum of three
commands. If you don’t set up your keys, you’ll type your password three times.
June 21-25, 2004 Lecture 1: Basic Skills 26
GWAG: Issues Issues you will find:
How do you run three jobs sequentially that require data transfer? You just run each command in a row.
How do you run three jobs in parallel that require data transfer? Make each transfer/job/transfer a script. Run the scripts in
parallel from the command line How do you deal with failures?
What if scp or ssh returns an error? What if scp or ssh never returns? (It hangs)
How do you keep track of everything? In an ad-hoc way?
June 21-25, 2004 Lecture 1: Basic Skills 27
GWAG: Dealing With the Issues We will talk about grid solutions to these
problems. Keep an eye out for:
GSI for security GRAM or Condor-G for transfer of input/output files Condor-G for job reliability DAGMan for running sets of jobs