10/13/20072007 sosp women's workshop1 beyond the os textbook - getting started in os research...
TRANSCRIPT
10/13/2007 2007 SOSP Women's Workshop 1
Beyond the OS textbook - getting started in OS research
Kim KeetonHP [email protected]
Dilma Da SilvaIBM [email protected]
http://www.hpl.hp.com/personal/Kimberly_Keeton http://www.research.ibm.com/people/d/dilma/
10/13/2007 2007 SOSP Women's Workshop 2
Getting started - a steep uphill climb?
TextbookLessons
CompletedResearch
Project
10/13/2007 2007 SOSP Women's Workshop 3
A little bit about Kim Education
PhD, Computer Science, UC Berkeley (1999) MS, Computer Science, UC Berkeley (1994) BS, Computer Engineering and Engineering and
Public Policy, Carnegie Mellon (1991) Career
HP Labs, 1999 - present Research interests
Storage systems, storage management, dependability, information management
10/13/2007 2007 SOSP Women's Workshop 4
A little more about Kim
10/13/2007 2007 SOSP Women's Workshop 5
A little bit about Dilma Education
PhD, Computer Science, Georgia Tech (1997) MS, Computer Science, Univ Sao Paulo, Brazil (1990) BS, Computer Science, Univ of Sao Paulo, Brazil (1986)
Career IBM Research, 2000-present Assistant Professor, University of Sao Paulo, 1996-2000
Research interests Operating systems, distributed systems
Parallel computing, software engineering, mobile computing
10/13/2007 2007 SOSP Women's Workshop 6
A little more about Dilma
Musical activities: embracing lack of “natural talent” Passions: books , knitting
Navigating politics: large extended family
10/13/2007 2007 SOSP Women's Workshop 7
Outline
Getting prepared Picking the “right” problem Finding a creative solution Quantitatively evaluating your solution Communicating your results The lessons along the way
10/13/2007 2007 SOSP Women's Workshop 8
Getting prepared: system building Sharpening your system-building skills
Find teams and partners that are right for you Learning by hand-on work, learning by example
(Knuth’s literary programming) Develop aesthetic for elegant, simple solutions;
fight temptation (yours, others) to build Taj Mahals! Your arsenal:
Tool chains, scripting, debugging, performance evaluation, tracing
Model checking Simulation
10/13/2007 2007 SOSP Women's Workshop 9
Getting prepared: Map the territory Read papers to learn … and to reflect on
the approach taken Identify the good, the bad, and the ugly Reading groups can be useful
Be curious … with focus Challenge your pride & prejudice
tendencies Common ineffective behavior:
Blindness to whole bodies of work Drowning on papers
10/13/2007 2007 SOSP Women's Workshop 10
Picking the “right” problem There are lots of right problems, not just one! Many ways to find a great problem to work
on: Understand technology trends and changing
usage patterns Run experiments to understand real problems Attend talks, including interesting ones in other
areas Look for open questions in future work section Carve out niche in larger group project Do an internship to understand “real world”
problems Solve a problem that impacts you personally Stumble across it while solving another problem
10/13/2007 2007 SOSP Women's Workshop 11
Picking the “right” problem
Do “real stuff”: solve problems that someone cares about Exposure to industrial-strength real-world problems
Pick a small problem and let it grow Course projects are a great starting point
Easier to make contributions if you arrive early to a field Part of the contribution is framing the research question “Good research is done with a shovel, not with tweezers.” --
Roger Needham Things to remember:
It doesn’t need to be ACM-Dissertation-Award perfect You’re not going to work on it for your entire career Make sure you are excited enough to work on it for several years
10/13/2007 2007 SOSP Women's Workshop 12
Finding a creative solution Keep things simple unless there’s a
good reason not to Pick innovation points carefully; be
compatible everywhere else Best results are obvious in retrospect
Borrow techniques from other fields Machine learning Data mining Optimization Control theory User interfaces
10/13/2007 2007 SOSP Women's Workshop 13
Quantitatively evaluating your solution Computer systems ideas must
be quantitatively evaluated Apply the scientific method Use experiments to answer
questions Use standard benchmarks Use measurements of real
systems Understand your results
10/13/2007 2007 SOSP Women's Workshop 14
Apply the scientific method Goal: (dis)prove hypothesis
Not to measure what’s easy to instrument Think of it as minimizing the effort to prove your point
Useful exercise #1: back-of-the-envelope calculations to see potential of an idea
Useful exercise #2: draw graph you expect to see If experimental results differ:
Flaw in intuition? Flaw in system? Flaw in measurement? Uncovered interesting behavior?
Use intuition to ask questions, not to answer them
10/13/2007 2007 SOSP Women's Workshop 15
Use experiments to answer questions Microbenchmarks
Focus on a particular aspect of system Easy to reason about
Macrobenchmarks Do not focus on particular aspect of system More realistic evaluations of idea
10/13/2007 2007 SOSP Women's Workshop 16
Use standard benchmarks “For better or worse, benchmarks shape a field.” – Dave
Patterson Benefits of standard workloads:
Workload is well-defined, agreed upon by community Provides comparability with other systems May provide load generation tools
Transaction Processing Performance Council (TPC) (http://www.tpc.org)
Standard Performance Evaluation Corporation (SPEC) (http://www.spec.org)
10/13/2007 2007 SOSP Women's Workshop 17
Use measurements of real systems Traces of actual system behavior will ground your work in reality
Storage Networking Industry Association (SNIA) IOTTA Repository (http://iotta.snia.org/repository/resources/) System call traces: Coda, Drew Roselli's traces and others Network file system traces: Berkeley Auspex, Harvard NFS Block-level storage traces: HP Labs cello
Computer Failure Data Repository (CFDR) (http://cfdr.usenix.org/) Node outages in more than 20 large HPC clusters
Other options: Work with industrial partners Measure your local systems
10/13/2007 2007 SOSP Women's Workshop 18
Understand your results Use knowledge of statistics to evaluate results
Example: perform multiple iterations of experiment and calculate confidence intervals
Give yourself enough time to collect and understand the data Things always take longer than you think You won’t always get the measurements right the first time You may decide you need measure something new to make
the point
10/13/2007 2007 SOSP Women's Workshop 19
Communicating your results
Report in sufficient detail for others to reproduce results
Deadlines: friend or foe?Ask colleagues for feedback
Early drafts of papers Practice talks
Benefits You’ll get useful feedback People will be more aware of what you’re doing
10/13/2007 2007 SOSP Women's Workshop 20
Learning from mistakes Innovation involves risk
“Would you like the formula for success?" TJ Watson asked. "Double your rate of failure."
Being smart: early failure detection Inventory of “failures” and “successes”
We learn from both
10/13/2007 2007 SOSP Women's Workshop 21
It’s a learning process We can make the journey smoother:
Improve system building skills Learn from past and current efforts in the research community Learn about trends and problems in the field Adhere to scientific methods Practice careful evaluation Communicate
Summary
TextbookLessons
CompletedResearch
Project
10/13/2007 2007 SOSP Women's Workshop 22
More importantly …
HAVE FUN!
10/13/2007 2007 SOSP Women's Workshop 23
Acknowledgments
Eric Anderson Carla Ellis Manish Gupta Jim Gray Arif Merchant Jeff Mogul Brad Morrey Dave Patterson
Sharon Perl Anna Povzner Craig Soules Alistair Veitch Janet Wiener John Wilkes Jay Wylie