1 python training for hp oso guido van rossum cnri 7/23/1999 9am - 1pm
TRANSCRIPT
![Page 1: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/1.jpg)
1
Python Trainingfor HP OSO
Guido van RossumCNRI
7/23/19999am - 1pm
![Page 2: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/2.jpg)
2
Plug
• The Practice ofProgramming
• Brian W. Kernighanand Rob Pike
• Addison-Wesley, 1999
Mostly about C, but very useful!
http://cm.bell-labs.com/cm/cs/tpop/
![Page 3: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/3.jpg)
3
CODE STRUCTURE
![Page 4: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/4.jpg)
4
The importance of readability
• Most time is spent on maintenance
• Think about the human reader
• Can you still read your own code...– next month?– next year?
![Page 5: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/5.jpg)
5
Writing readable code
• Be consistent(but not too consistent!)
• Use whitespace judicously• Write appropriate comments• Write helpful doc strings
– not novels
• Indicate unfinished business
![Page 6: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/6.jpg)
6
Modifying existing code
• Conform to the existing style– even if it’s not your favorite style!– local consistency overrides global
• Update the comments!!– and the doc strings!!!
![Page 7: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/7.jpg)
7
Organizing code clearly
• Top-down or bottom-up?• Pick one style, stick to it• Alternative: group by functionality
– eg:• constructor, destructor• housekeeping• low level methods• high level methods
![Page 8: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/8.jpg)
8
When to use classes(...and when not!)
• Use a class:– when multiple copies of state needed
• e.g.: client connections; drawing objects
• Use a module:– when on copy of state always suffices
• e.g.: logger; cache
• Use functions:– when no state needed; e.g. sin()
![Page 9: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/9.jpg)
9
Class hierarchies
• Avoid deep class hierarchies– inefficient
• multi-level lookup
– hard to read• find method definitions
– easy to make mistakes• name clashes between attribute
![Page 10: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/10.jpg)
10
Modules and packages
• Modules collect classes, functions
• Packages collect modules
• For group of related modules:– consider using a package
• minimizes chance of namespace clashes
![Page 11: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/11.jpg)
11
Naming conventions(my preferred style)
• Modules, packages: lowercase• except when 1 module ~ 1 class
• Classes: CapitalizedWords• also for exceptions
• Methods, attrs: lowercase_words• Local variables: i, j, sum, x0, etc.• Globals: long_descriptive_names
![Page 12: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/12.jpg)
12
The main program
• In script or program:def main(): ...if __name__ == “__main__”: main()
• In module:def _test(): ...if __name__ == “__main__”: _test()
• Always define a function!
![Page 13: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/13.jpg)
13
DOCUMENTATION
![Page 14: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/14.jpg)
14
Writing comments
• Explain salient points (only)n = n+1 # include end point
• Note dependencies, refs, bugs# Assume reader() handles I/O errors# See Knuth, vol.3, page 410# XXX doesn’t handle x<0 yet
![Page 15: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/15.jpg)
15
Writing doc strings
"""Brief one-line description.
Longer description, documentingargument values, defaults,return values, and exceptions."""
![Page 16: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/16.jpg)
16
When NOT to use comments
• Don’t comment what’s obviousn = n+1 # increment n
• Don’t put a comment on every line• Don’t draw boxes, lines, etc.
#------------------------def remove_bias(self):#------------------------ self.bias = 0
![Page 17: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/17.jpg)
17
One more thing...
UPDATE THE COMMENTS WHEN UPDATING THE CODE!
(dammit!)
![Page 18: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/18.jpg)
18
THE LIBRARY
![Page 19: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/19.jpg)
19
The library is your friend!
• Know what's there• Study the library manual
– especially the early chapters:• Python, string, misc, os services
• Notice platform dependencies• Avoid obsolete modules
![Page 20: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/20.jpg)
20
Stupid os.path tricks
• os.path.exists(p), isdir(p), islink(p)• os.path.isabs(p)• os.path.join(p, q, ...), split(p)• os.path.basename(p), dirname(p)• os.path.splitdrive(p), splitext(p)• os.path.normcase(p), normpath(p)• os.path.expanduser(p)
![Page 21: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/21.jpg)
21
PORTING YOUR BRAIN
(from Java to Python)
![Page 22: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/22.jpg)
22
Class or module?
• Stateless operatons, factory funcs• Java: static methods• Python: functions in module
• Singleton state• Java: static members, methods• Python module globals, functions
![Page 23: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/23.jpg)
23
Private, protected, public?
• Java:• private, protected, public
– enforced by compiler (and JVM?)
• Python:• __private
– enforced by compiler– loophole: _Class__private
• _private, _protected, public– used by convention
![Page 24: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/24.jpg)
24
Method/constr. overloading
• Java:
class C { int f() { ... } int f(int i) { ... } int f(int i, int arg) { ... }}
• Python:
class C: def f(i=0, arg=None): ...
![Page 25: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/25.jpg)
25
Java interfaces
• In Python, interfaces often implied
class File: def read(self, n): ...
class CompressedFile: def read(self, n): ...
![Page 26: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/26.jpg)
26
Abstract classes
• Not used much in Python• Possible:
class GraphicalObject: def draw(self, display): raise NotImplementedError def move(self, dx, dy): raise NotImplementedError ....
![Page 27: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/27.jpg)
27
ERROR HANDLING
![Page 28: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/28.jpg)
28
When to catch exceptions
• When there's an alternative optiontry: f = open(".startup")except IOError: f = None # No startup file; use defaults
• To exit with nice error messagetry: f = open("data")except IOError, msg: print "I/O Error:", msg; sys.exit(1)
![Page 29: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/29.jpg)
29
When NOT to catch them
• When the cause is likely a bug• need the traceback to find the cause!
• When the caller can catch it• keep exception handling in outer layers
• When you don't know what to dotry: receive_message()except: print "An error occurred!"
![Page 30: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/30.jpg)
30
Exception handling style
• Bad:try: parse_args() f = open(file) read_input() make_report()except IOError: print file, "not found"
# (what if read_input()# raises IOError?)
• Good:parse_args()try: f = open(file)except IOError, msg: print file, msg sys.exit(1)read_input()make_report()
![Page 31: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/31.jpg)
31
Error reporting/logging
• Decide where errors should go:– sys.stdout - okay for small scripts– sys.stderr - for larger programs– raise exception - in library modules
• let caller decide how to report!
– log function - not recommended• better redirect sys.stderr to log object!
![Page 32: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/32.jpg)
32
The danger of “except:”
• What's wrong with this code:try: return self.children[O] # first childexcept: return None # no children
• Solution:except IndexError:
![Page 33: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/33.jpg)
33
PYTHON PITFALLS
![Page 34: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/34.jpg)
34
Sharing mutable objects
• through variablesa = [1,2]; b = a; a.append(3); print b
• as default argumentsdef add(a, list=[]): list.append(a); return list
• as class attributesclass TreeNode: children = [] ...
![Page 35: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/35.jpg)
35
Lurking bugs
• bugs in exception handlerstry: f = open(file)except IOError, err: print "I/O Error:", file, msg
• misspelled names in assignmentsself.done = 0while not done: if self.did_it(): self.Done = 1
![Page 36: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/36.jpg)
36
Global variables
# logging module
log = []
def addlog(x): log.append(x)
def resetlog(): log = [] # doesn’t work!
# logging module# corrected version
log = []
def addlog(x): log.append(x)
def resetlog(): global log log = []
![Page 37: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/37.jpg)
37
kjpylint
• Detects many lurking bugs– http://www.chordate.com/kwParsing/
![Page 38: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/38.jpg)
38
PERFORMANCE
![Page 39: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/39.jpg)
39
When to worry about speed
• Only worry about speed when...
– your code works (!)– and its overall speed is too slow– and it must run many times– and you can't buy faster hardware
![Page 40: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/40.jpg)
40
Using the profile module>>> import profile>>> import xmlini>>> data = open("test.xml").read()>>> profile.run("xmlini.fromxml(data)")>>> profile.run("for i in range(100): xmlini.fromxml(data)") 10702 function calls in 1.155 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function) 1 0.013 0.013 1.154 1.154 <string>:1(?) 1 0.001 0.001 1.155 1.155 profile:0(for i in range(100): xmlini.fromxml(data)) 0 0.000 0.000 profile:0(profiler) 500 0.018 0.000 0.018 0.000 xmlini.py:105(end_group) 700 0.032 0.000 0.032 0.000 xmlini.py:109(start_item) 700 0.050 0.000 0.050 0.000 xmlini.py:115(end_item) 200 0.007 0.000 0.007 0.000 xmlini.py:125(start_val) 200 0.014 0.000 0.014 0.000 xmlini.py:129(end_val) 1600 0.190 0.000 0.270 0.000 xmlini.py:134(finish_starttag) 1600 0.163 0.000 0.258 0.000 xmlini.py:143(finish_endtag) 100 0.004 0.000 0.004 0.000 xmlini.py:152(handle_proc) 100 0.007 0.000 0.007 0.000 xmlini.py:162(handle_charref) 100 0.007 0.000 0.007 0.000 xmlini.py:167(handle_entityref) 3600 0.161 0.000 0.161 0.000 xmlini.py:172(handle_data) 100 0.003 0.000 0.003 0.000 xmlini.py:182(handle_comment) 100 0.420 0.004 1.141 0.011 xmlini.py:60(fromxml) 100 0.007 0.000 0.007 0.000 xmlini.py:70(__init__) 100 0.004 0.000 0.004 0.000 xmlini.py:80(getdict) 200 0.012 0.000 0.012 0.000 xmlini.py:86(start_top) 200 0.014 0.000 0.014 0.000 xmlini.py:92(end_top) 500 0.029 0.000 0.029 0.000 xmlini.py:99(start_group)
![Page 41: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/41.jpg)
41
Measuring raw speed
# Here's one wayimport timedef timing(func, arg, ncalls=100): r = range(ncalls) t0 = time.clock() for i in r: func(arg) t1 = time.clock() dt = t1-t0 print "%s: %.3f ms/call (%.3f seconds / %d calls)" % ( func.__name__, 1000*dt/ncalls, dt, ncalls)
![Page 42: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/42.jpg)
42
How to hand-optimize code
import string, types
def dictser(dict, ListType=types.ListType, isinstance=isinstance): L = [] group = dict.get("main") if group: for key in group.keys(): value = group[key] if isinstance(value, ListType): for item in value: L.extend([" ", key, " = ", item, "\n"]) else: L.extend([" ", key, " = ", value, "\n"]) ... return string.join(L, "")
![Page 43: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/43.jpg)
43
When NOT to optimize code
• Usually• When it's not yet working• If you care about maintainability!
• Premature optimization is the root of all evil (well, almost :)
![Page 44: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/44.jpg)
44
THREAD PROGRAMMING
![Page 45: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/45.jpg)
45
Which API?
• thread - traditional Python APIimport threadthread.start_new(doit, (5,))# (can't easily wait for its completion)
• threading - resembles Java APIfrom threading import Thread # and much more...t = Thread(target=doit, args=(5,))t.start()t.join()
![Page 46: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/46.jpg)
46
Atomic operations
• Atomic:i = Nonea.extend([x, y, z])x = a.pop()v = dict[k]
• Not atomic:i = i+1if not dict.has_key(k): dict[k] = 0
![Page 47: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/47.jpg)
47
Python lock objects
• Not reentrant:lock.acquire(); lock.acquire() # i.e. twice!
– blocks another thread callslock.release()
• No "lock owner"• Solution:
– threading.RLock class• (more expensive)
![Page 48: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/48.jpg)
48
Critical sections
lock.acquire()try: "this is the critical section" "it may raise an exception..."finally: lock.release()
![Page 49: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/49.jpg)
49
"Synchronized" methods
class MyObject:
def __init__(self): self._lock = threading.RLock() # or threading.Lock(), if no reentrancy needed
def some_method(self): self._lock.acquire() try: "go about your business" finally: self._lock.release()
![Page 50: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/50.jpg)
50
Worker threads
• Setup:def consumer(): ...def producer(): ...for i in range(NCONSUMERS): thread.start_new(consumer, ())for i in range(NPRODUCERS): thread.start_new(producer, ())"now wait until all threads done"
![Page 51: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/51.jpg)
51
Shared work queue
• Producers:while 1: job = make_job() Q.put(job)
• Consumers:while 1: job = Q.get() finish_job(job)
• Shared:import QueueQ = Queue.Queue(0) # or maxQsize
![Page 52: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/52.jpg)
52
Using a list as a queue
• Shared:Q = []
• Producers:while 1: job = make_job() Q.append(job)
• Consumers:while 1: try: job = Q.pop() except IndexError: time.sleep(...) continue finish_job(job)
![Page 53: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/53.jpg)
53
Using a condition variable
• Shared:Q = []cv = Condition()
• Producers:while 1: job = make_job() cv.acquire() Q.append(job) cv.notify() cv.release()
• Consumers:while 1:
cv.acquire()
while not Q:
cv.wait()
job = Q.pop()
cv.release()
finish_job(job)
![Page 54: 1 Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9am - 1pm](https://reader035.vdocuments.us/reader035/viewer/2022062404/551507bf550346c77d8b4692/html5/thumbnails/54.jpg)
54
TIME FOR DISCUSSION