www.ci.anl.govwww.ci.uchicago.edu

Scientific research: What Anna Karenina teaches us about useful negative results

Daniel S. KatzSenior Fellow, Computation Institute (University of Chicago & Argonne National Laboratory)Affiliate Faculty, CCT (LSU)

www.ci.anl.govwww.ci.uchicago.edu

2 Parallel and Distributed Application Paradigms

• "Happy families are all alike; every unhappy family is unhappy in its own way.”

-- Leo Tolstoy. Anna Karenina

www.ci.anl.govwww.ci.uchicago.edu

3 Parallel and Distributed Application Paradigms

Anna Karenina principle• Describes an endeavor in which a deficiency in any one of a number

of factors dooms it to failure• Consequently, a successful endeavor (subject to this principle) is

one where every possible deficiency has been avoided• In statistics, the term is used to describe significance tests: there

are any number of ways in which a dataset may violate the null hypothesis* and only one in which all the assumptions are satisfied -- https://en.wikipedia.org/wiki/Anna_Karenina_principle

*The null hypothesis usually refers to a general statement or default position that there is no relationship between two measured phenomena, or no difference among groups -- Everitt, Brian (1998). The Cambridge Dictionary of Statistics

www.ci.anl.govwww.ci.uchicago.edu

4 Parallel and Distributed Application Paradigms

Scientific research process

• The goal of research is provable (reproducible) knowledge– Expressed descriptively

o Something an observer can read, i.e., in a paper, book, algorithm, etc.

– Expressed physicallyo Something an observer can touch, measure, etc., i.e., an experiment

– Expressed digitallyo Something an observer can run on a computer, i.e., simulation or

analysis software and required data

• Both physical and digital results usually also require descriptive analysis

www.ci.anl.govwww.ci.uchicago.edu

5 Parallel and Distributed Application Paradigms

The value of research results

• How do we decide if there is value in a result?– Results can be positive or negative– Must be new (and arguably provable)– Positive results that show something novel– Positive results that show something already in a

novel way

www.ci.anl.govwww.ci.uchicago.edu

6 Parallel and Distributed Application Paradigms

The value of negative research results

• Many results can show that something doesn't work

• First demonstration that something doesn’t work is novel– May be sufficient for publication by itself– May require understanding of why

• Otherwise, want to provide new understanding of why it doesn't work– Potentially leading to possible paths to make it work

www.ci.anl.govwww.ci.uchicago.edu

7 Parallel and Distributed Application Paradigms

Back to Anna Karenina

• "Happy families are all alike; every unhappy family is unhappy in its own way.” -- Leo Tolstoy. Anna Karenina

• Lesson– To sell a story, it needs to be novel, perhaps by

explaining why the family is unhappy