experiments in computer science - don't loathe them, but love them
DESCRIPTION
Some subjective recommendations of how to proceed with experimentsTRANSCRIPT
Steffen [email protected]
1WeST
Web Science & Technologies
University of Koblenz ▪ Landau, Germany
Experiments
Don‘t loathe them, but love them!
Steffen Staab
Steffen [email protected]
2WeST
...a month in the lab can save an hour in the library...
Unknown source
Steffen [email protected]
3WeST
...avoid the experiment
and
you may avoid finishing your thesis...
Steffen 2014
Steffen [email protected]
5WeST
Why do we loathe experiments?
Experiments require effort require time require money may be complicated may involve human subjects (BAD!) may even involve real people, i.e. not students (WORSE!!!) have unpredictable outcomes (THE WORST!!!!!) May have to be redone (THE NIGHTMARE!!!!!)
Steffen [email protected]
7WeST
1 What do you talk about?
Define your key terms clearly!
Don‘t redefine existing terms!Don‘t use figurative speech.Negative example: Beer is a kind of bread made from barley.
Define them independently of how you compute them:Negative example: We are doing this experiment to show that Lakhsa is enjoyed by more people than Surströmming
Use the terms exactly as you defined them:Negative example: The strömming must be enjoyed cold.
Steffen [email protected]
8WeST
2 Formulate a hypothesis
What is your experiment about?
Only possible to define after key concepts are clear!
As precise as possible:
...enjoy more...?
vs
...would do immediately again, if given the choice....
Briefly say what you abstract from and why.
A clear hypothesis saves time!
Steffen [email protected]
9WeST
3 Fail-safe hypothesis
Hypothesis that is always interesting, no matter whether the experiment succeeds or fails comparing different methods is often interesting Be impartial / not biased in favor of your method
AND write in this way not: „we want to show that our method is better“
BUT„we want to find out whether fermentation makes the fish more enjoyable for Swedish and non-swedish“
Tackling unpredictability!
Steffen [email protected]
10WeST
4 Write before performing experiment
Write The experiment The description of the outcome Receive input/criticism on this writing
BEFORE
you perform the experiment
Avoid having to redo experiment!
Steffen [email protected]
11WeST
5 Reverse experiment planning
Lead question: What can make your experiment go wrong? May not be understood / handled by test subjects Do assume that someone can misunderstand something Keep your original data, video recordings if possible ...
Real people are somewhat unpredictable!
Steffen [email protected]
12WeST
6 More is better...
Not just one data set
Do not vary arbitrarily, but vary on what may help to explore your hypotheses
Not just one baseline Especially not one that is very weak,
only because it is available• Tackling the „hard baseline“ is not impossible• Tackling the „hard baseline“ right away may be easier
than doing 5 rounds until the paper is accepted
More....is less time!
Steffen [email protected]
13WeST
7 When more is worse
Grounding in theory
Counterexample:
„....we investigated all the 100680 parameter configurations....“
Experiments produce outliers! Statistical effects Normalization exists, but is very strict
Save effort!
Steffen [email protected]
14WeST
8 When reporting: Focus the reader‘s eye
Issue: Experiments may be complex Hard to do Hard to understand for the reader
Measures: Do not overwhelm the reader Focus the reader‘s eyes
Do not just show graphs/charts/etc. Each graph/chart should give an answer to a question Reader should know why he sees the chart
Manage complexity!
Steffen [email protected]
15WeST
9 Sound experiment structure
Evaluation procedure must be independent from the method you used-7should also be applicable for a large set of competitors
1. Set up; i.e. data, conditions
2. Evaluation measure, baselines/competitors
3. Results
4. Discussion
Keep this order and do not discuss results before you have described setup (maybe briefly in intro/abstract)
Steffen [email protected]
16WeST
10 Be honest and ethical
If there are issues, report about them Publish the methods Publish the software Publish the data
Steffen [email protected]
17WeST
Disclaimer
I have never eaten Surströmming I might have eaten Lakhsa in a Tibetan restaurant, but
don‘t remember it