the scientific method and the design of experiments

VIL 2010 All Rights Reserved 1

The Scientific Method and the Design of

ExperimentsDr Leslie T Falkingham

www.vil.org.uk

Xi’an Jiaotong University March 2010

VIL 2010 All Rights Reserved Xi’an 2010. 2

This Lecture covers the following topics:

1. What is The Scientific Method and where did it come from?

2. Different types of experiments

3. How to design experiments – The Rule of Three

4. The iterative nature of experiments

5. How to interpret the results

6. How to present the results

Introduction


Sir Francis Bacon and the Baconian Method

“Novum Organum“ 1620

Born 1561

Trinty College Cambridge University 1573- 76

Lord Chancellor of England 1618-1621

April 1626 Died - inventing the Frozen Chicken!

What is The Scientific Method and where did it come from?

Sources: Adapted from Wikipedia article on Sir Francis Bacon


What is The Scientific Method and where did it come from?

The “Baconian Method“

Bacon is credited with inventing the modern Scientific Method. Originally called the “Baconian Method“. This was based on Inductive Reasoning and a number of principles:

The researcher should follow the Logic sequence Observation (Facts) – Axiom (Hypothesis) – Law.

The Researcher should remove bias from the research to arrive at true Facts and a Theory based solely on the facts.

If proven the Theory then becomes a Law.


Different types of experiments

The Logic Sequence

Experiments are carried out as an application of logic. There are three main types of scientific experiment;

To provide data on which a hypothesis may be based.

To confirm or deny a hypothesis by testing its positive predictions.

To confirm or deny a hypothesis by testing its negative predictions



To provide data on which a hypothesis may be based.

This where data on a subject is collected before inducing a hypothesis of the relationship between variables.

By definition it predates the creation of a hypothesis and so it is important to collect all data which may be relevant. The researcher must be careful not to bias the experiment by forming a working hypothesis and selecting data accordingly.



To confirm or deny a hypothesis by testing its positive predictions.

This where the experiment is designed to test a hypothesis of the relationship between variables.

Predictions are made using the hypothesis to be tested Which states that if a certain action is taken then it will cause a certain outcome. Care must be taken to ensure that a true relationship is proven between the action and the effect, and that this could not occur due to unrelated reasons.



To confirm or deny a hypothesis by testing its negative predictions.

This where the experiment is designed to test a hypothesis of the relationship between variables.

Predictions are made using the hypothesis to be tested Which states that if a certain action is taken then it will not cause a certain outcome. Care must be taken to ensure that a true relationship is proven between the action and the effect, and that this could not occur due to unrelated reasons.


Control experiments – testing the capability of the experiment.

When setting up the experiment try to perform tests where the result is already known. Both for positive and negative results.

For example, if performing bil or HV ac testing. Operate the equipment with the device on test removed and the connections short circuited and then repeat with an open connection. This should give a result of no withstand in the first case, and total withstand in the second case. “Sanity“ tests such as these are simple, but are vital to remove doubt and errors in an experimental setup.

How to design experiments


The Rutherford Experiment

In 1904 it was hypothesised by J J Thomson that the atom consisted of electrons embedded in a sphere of positive matter, known as the “Plum Pudding“ model. Which became the accepted atomic theory at the time.

In 1909 an experiment was carried out by Geiger and Marsden on behalf of ernest Rutherford. This had unexpected results and resulted in the present “Planetary“ model of the atom, proposed in Rutherford‘s paper of 1911.



The Rutherford Experiment

The “Plum Pudding“ Model “The Planetary“ Model


Source Wikipedia, The Rutherford Experiment


The Rutherford Experiment The experiment was intended to probe

the structure of the atom, it did this by firing alpha particles at a thin sheet of gold. The Plum Pudding theory predicted that the particles would suffer small deflections in their path which could be used to determine the distribution of charge within the atom. This was actually seen, but also occasionally a particle was deflected back towards the source. This result was completely unexpected and was not predicted by the theory.


Source Wikipedia, The Rutherford Experiment


The Rutherford Experiment Because of the strange results Rutherford hypothesised

that the atom had a solid centre of positive charge which contained most of the mass of the atom. This is now essentially thought to be correct.

The reason that his results were not published until 1911, two years after the experiments were carried out, was that he and his co-workers spent over a year trying to understand what was wrong with the experiment! Only after he was convinced that the results were correct did he come up with his new theory.

He did not reject the results which did not fit the theory. But because they did not fit, he very carefully checked to ensure that they were both correct and repeatable. Then he changed the theory – and the world!



Important factors in designing experiments.

1. The experiment must be designed to meet the objectives of the experiment. Sounds obvious, but if after perfoming the experiment there is still doubt about the meaning of the results then it was not well designed.

2. The experimental equipment must be able to unambiguously meet the objectives of the experiment. Many experiments fail due to inadequate or inappropriate equipment or instrumentation.

3. When designing any experiment it is important to try not to extrapolate results, and to make sure if possible that the experimental data set covers more than just the region of interest.

4. It is important to understand that when performing any experiment the results will be affected by uncontrolled variations. No matter how well designed the experiment is it is impossible to remove all external factors. This will show itself as “Rogue“ results.



The Rule of Three

If something happens once - it is chance

If something happens twice - it is coincidence

If something happens three times - it is significant

This is a crude but effective statistical approach to the probability of a result being real. The essential feature of an experiment should be that when it is repeated you find the same result. By applying this rule you significantly reduce the probability of experimental error, and you demonstrate repeatability.

How to design experiments – the Rule of Three


The Rule of Three

Depending on the experiment this may mean;

Repeating the experiment three times (Three sets of data points) – shows that the experiment for the sample under test gives consistent results, and gives an indication of the level of experimental error

Performing the experiment on three identical samples (Three sets of data points), - shows that the results are consistent for that type of sample

Repeating the experiment three times on each of three samples (Nine sets of data points). – Shows that the experiment gives consistent results with a level of experimental error, and that the results are generally true for the type of sample under test



Rogue Results

When performing any experiment it is important to understand that the results will be affected by uncontrolled variations.

No matter how well designed the experiment is it is impossible to remove all external factors. This will show itself as “Rogue“ results.

These may be clearly observed as deviations in a sequence for example. As shown in the following example;



“Rogue“ Results


Y

0

2

4

6

8

10

12

14

0 1 2 3 4 5 6 7 8 9

Y



“Rogue“ ResultsChart Title

0

2

4

6

8

10

12

14

0 1 2 3 4 5 6 7 8 9

Y

Poly. (Y)



“Rogue“ ResultsY

0

2

4

6

8

10

12

14

0 2 4 6 8 10 12

Y



“Rogue“ ResultsChart Title

-2

0

2

4

6

8

10

12

14

0 2 4 6 8 10 12

Y

Linear (Y)



“Rogue“ Results

0

2

4

6

8

10

12

14

16

18

0 2 4 6 8 10 12

X

Y

Series1

Series2

Series3


The Experimental Sequence Generally experiments are iterative to some extent.

As the first results are known then areas of interest will appear, and weaknesses in the experimental set up and procedure will be exposed.

Once the experiment is designed, it is normal to perform the experiment in three phases.

Phase 1: Initial set up and experimental verification

Phase 2: Improved experiment – be careful, depending on how you change the experiment the original results may no lnoger be compatible with the later results.

Phase 3: Focus of investigation on areas of interest

The Iterative Nature of Experiments


The Experimental Sequence Phase 1: Initial set up and experimental verification

The experiment must be carefully designed to meet the objectives of the experiment. This means taking time to understand the relationship between the vaiables being measured and to identify any other factors which may affect the realtionship. Verfiy that the experiment can give you the desired results and establish the level of likely experimental error.

If the experiment is not well thought out, then it is worthless. Take time to carefully consider the experiment, also consider how the experimental results may be criticised once the experiment is completed. If you do not do this before the experiment, others will do so after you publish!



The Experimental Sequence Phase 2: Improved experiment – be careful,

depending on how you change the experiment the original results may no longer be compatible with the later results.

In this phase you collect most of the data. This is the main part of the experiment and the collection and recording of data must be rigorous. Record everything!



The Experimental Sequence Phase 3: Focus of investigation on areas of interest

The data in phase 2 normally will have identified points of interest, “rogue“ data points, extrapolations, areas of change. In this phase you look at the data and investigate these points of interest more thoroughly, normally by performing additional tests to investigate specific identified points.



Points of interest


0

2

4

6

8

10

12

14

16

18

0 2 4 6 8 10 12

X

Y

Series1

Series2

Series3


The Results Normally there is a body of knowledge already existing

which can be applied to the results. Similar work by others, Theoretical background, etc. Interpret and explain the results in relation to this previous knowledge.

Where appropriate use basic statistical tests to validate your results, such as Chi-squared ( χ 2 test), or Student’s T-test, etc. In addition if the problem is complex, then the data may be susceptible to analysis using statistical tyechniques such as Orthoganol Arrays

The important point is not to ignore any data even if it does not fit your existing theory or understanding

How to Interpret the Results

See: World Class Quality, K R Bote, and Statistics for Technology, C Chatfield


How to Interpret the Results

The Results

0

2

4

6

8

10

12

14

16

18

0 2 4 6 8 10 12

X

Y

Series1

Series2

Series3


The Key Principles There are six main principles:

1. Choose the method which presents the results in the most clear and easy to understand way

2. Explain what you think your results mean and how they relate to theory

3. Explain why you did the experiment and what you were trying to achieve

4. Describe the experiment in such a way as to allow others to verify the results

5. Explain the limitations of the experiment and the experimental error

6. If you do not understand a data point, or a result, indicate it and say so. You do not have to explain everything!

How to Present the Results



Graphical or tabular presentations


How to Present the Results A graph without explanation can be misleading or

meaningless



“Rogue“ Results

0

2

4

6

8

10

12

14

16

18

0 2 4 6 8 10 12

X

Y

Series1

Series2

Series3


Plan your experiment carefully to ensure that it can meet the objectives of the experiment.

Understand the limitations of the experiment, and experimental error and ensure that you have allowed for these.

Realise that “Rogue“ results are normal and treat them seriously – these can actually be the most important part of the experiment.

Interpret your results in the light of existing knowledge and previous work of yourself of others

Remember the six key principles of presenting the results. If you follow them, publication and explaining your work to others is quite simple.

Conclusions


Questions?

the scientific method and the design of experiments

Documents