© a. taylor do not duplicate without author’s permission 1 reliability and validity

© A. Taylor

Do not duplicate without author’s permission 1

Reliability and Validity

© A. Taylor


Objectives

• There are five (5) main objectives for this module, to:

1. Be able to recognize various tools of measurement

2. Understand what is meant by reliability

3. Be able to evaluate the reliability of a measure

4. Understand what is meant by validity

5. Be able to evaluate the validity of a measure

© A. Taylor


We have already learned about internal and external validity

of a study

Now we are going to learn about reliability and validity

of a measure

© A. Taylor


We have already learned about internal and external validity

of a study

Now we are going to learn about reliability and validity

of a measure

Don’t get confused. Even though they both use the terms validity, they use them in a slightly different way.

© A. Taylor


In Other Words

• When evaluating a Study discuss the– Internal Validity– External Validity

• When evaluating a Measure discuss the– Reliability– Validity

You already learned about this in the Integrating Research Concepts tutorial

We are learning about this now

© A. Taylor


What is Reliability?

• Reliability is:– the consistency of your measurement

instrument

– the degree to which an instrument measures the same way each time it is used under the same condition with the same subjects

© A. Taylor


What is Reliability?


instrument


What do I mean by…?

© A. Taylor


What in the world is a measurement instrument?

• Any tool that you use to measure with…• What “instrument” might you use to measure the

following items?

1. How heavy the apples are

2. How hot the meat is

3. How much orange juice there is

4. How tall the wall is

a scale

a meat thermometer

a measuring cup

a yardstick

© A. Taylor


More on Instruments

What would the following instruments measure?

The size of someone’s foot

EyesightThe perfect amount of spaghetti!

© A. Taylor


More on InstrumentsOften there are many different instruments that could be

used to measure the same thing.

Take distance for example…

Or an odometer

You could use a measuring tapeMeasure it out on a map

Or walk it with a pedometer

You can count paces

Or get really creative…Use a distance wheel

© A. Taylor


What instrument you choose depends on several factors

• Ease of use – do you know how to use it?I still can’t figure out my pedometer.

• Access – do you have access to the instrument?Where do you get a distance wheel from? What if you don’t have a car – and therefore no odometer?

• Appropriateness – is it the best tool for what you are measuring? You wouldn’t use a ruler to measure the distance from here to Ohio; you wouldn’t use an odometer to measure the length of a room.

• Accuracy – how precise do your measurements have to be?Counting paces is easy and cheap but only an estimate.

• Cost – how expensive is it to use the instrument or interpret the results?You can also use a laser but that can be really expensive.

© A. Taylor


Back to Reliability…


instrument


© A. Taylor


Reliability

• Imagine that you are using a ruler

to measure a book

And it measures about 7 inches across

What do you think would happen if you waited 10 minutes and measured the book again, how long would it be then?

…Probably still 7 inches

What if you spun the ruler around!

And shook it up really good?!

Now what would it say?

…Probably still 7 inches

© A. Taylor


Reliability

• Your ruler… – was consistent– measured the same way each time it

was used under the same condition with the same object

The book did not change and therefore the ruler reported back the same measurement

Your ruler is RELIABLE

© A. Taylor


Reliability• Every time the water begins to

boil the kettle whistles

• Every time it is 6:30 my alarm clock goes off

• Every time I use the little plastic up I have 2 tablespoons of medicine

All of these tools of measurement are reliablereliable given the same temperature, time and volume, they measure the same consistentlyconsistently.

© A. Taylor


Reliability• Reliability alone does not mean that you

have a good instrument however.

• Imagine the following reliable instrument:

0165Every morning Margie gets on the scale and every morning it reads 165 lbs.

It seems pretty reliable since Margie hasn’t gained or lost any weight.

One day an elephant got on Margie’s scale and it still read 165 lbs.!

How is this scale reliable?

It does measure the same way under the same conditions (Margie’s) – but a lot of other conditions too (the elephant and who knows what else).

© A. Taylor


Reliability

• What about this reliable instrument…

This clock reads 6:15

If nothing changes – if time stands still, will the clock still say the same thing?

YES! It’s very reliable. You always know exactly what it is going to say.

The problem is, even if time doesn’t stand still, the clock will not move…but it IS still reliable.

© A. Taylor


This brings us to Validity

© A. Taylor


What is Validity?

• Validity asks – if an instrument measures what it is

supposed to

– how “true” or accurate the measurement is

© A. Taylor


Reliable but not Valid165

These instruments are very RELIABLE

They both report consistently – too consistently

But, neither measures what it is supposed to:

• The scale is not really measuring weight

• The clock is not measuring time

They are NOT VALID

© A. Taylor


Reliable but not Valid

• Remember our reliable ruler?

Can it measure how loud the radio is? how full the

glass is?

how smart the girl is?

The ruler may be reliable (and perhaps even valid) but not in these situations!

It is only valid for measuring length.

© A. Taylor


Putting Reliability and Validity Together

• Every instrument can be evaluated on two dimensions:– Reliability

• How consistent it is given the same conditions

– Validity• If it measures what it is supposed to and how

accurate it is

© A. Taylor


Putting Reliability and Validity Together

• Imagine that I have 3 fish tank thermometers, a blue one, a red one, and a green one.

• The blue one always reads the same temperature no matter how hot or cold the water is.

• The red one shows a different temperature every time even if I just measured it 5 seconds earlier.

• The green one seems to read accurately, warm when the water is warm and cold when the water is cool.

© A. Taylor


Complete the chart below

Is it consistent?

Is it measuring what it is supposed

to?

Is it reliable?

Is it valid?

Blue always reads the same temperature no matter what

Red different temperature every time even if nothing has changed

Green warm when the water is warm and cold when the water is cool

Yes

No

Yes, the thermometer only changes if the temperature changes

No

No

Yes

Reliable but Not valid

Not reliable Not valid

Reliable and Valid

© A. Taylor


What can be said of the reliability and validity of the following?

• A spelling test with the following item: 2 + 5 = ____ – Probably reliable, if you get it wrong once you will probably get it wrong again

(assuming no new learning) – same with getting it right.

– Lacks validity, this is more appropriate for a math test, not a spelling test

• An elastic ruler (every time you use it is stretches to a different length)– Lack reliability

– You can’t have validity without reliability

• A thermometer used to measure volume – Probably reliable

– Lacks validity for this task

• A scale that reads 40 pounds at baseline – Reliable, will consistently be 40 lbs. off

– Not valid

© A. Taylor


Consider the Peabody Picture Inventory

Is it reliable? – if you don’t know the vocabulary today, are you going to suddenly know it tomorrow or next week? Probably not.

We’ll say it’s reliable

Imagine someone wanted to use it as a measure of cognitive development, is it valid?

Well, there are a lot of ways to operationalize cognitive development. Some might say that this is a better measure of English language competence rather than general cognition.

We’ll say that it lacks some validity for general cognition.

© A. Taylor


Consider a Maze Task to measure fine motor skills

Is it reliable? – if you do the maze once, do you think you need the same amount of time to finish it a second time?

You’ll probably be faster the second time just from practicing – even though your motor skills didn’t change any.

We’ll say it lacks reliability.Is it valid?

No, a measure must be reliable in order to be valid.

It is probably a more valid measure of memory.

© A. Taylor


Is it reliable? – if you don’t know the answers, are you going to suddenly know them tomorrow or next week? Probably not.

We’ll say it’s reliable

Is it valid?

This doesn’t look like a measure of World Geography to me, it focuses on America. It doesn’t even seem like American geography – maybe American Geographical Trivia.

It’s not a valid measure of World Geography.

1. The nickname for the state of New Hampshire is: a. Vacationland b. The Pine Tree State c. The Granite State

2. The most populated U.S. city is: a. New York b. Los Angeles c. Chicago

3. The "Coyote" State is: a. New Mexico b. Nebraska c. South Dakota

4. The largest state is: a. Texas b. Alaska c. Hawaii

5. The highest mountain is the U.S. is: a. Mt. McKinley b. Mt. Shasta c. Mt. Logan

Consider the World Geography quiz below:

© A. Taylor


Thinking More about Validity

Validity is whether or not the instrument measures what it is designed to measure.

Below are three constructs that you can use to evaluate the validity of a measure:

• Face Validity

• Predictive Validity

• Concurrent Validity

© A. Taylor


Three Validities

• Face Validity -- Do the questions look like measure what they are supposed to?

• Predictive Validity -- Do you imagine that this measure would predict something that it logically should?

• Concurrent Validity – Do you think this measure correlates strongly with something that it logically should?

© A. Taylor


Face Validity

Do the questions look like they measure what they are supposed to?

• What does the question below look like it would measure?

Choose the item that best describes you:(0) I do not feel sad. (1) I feel sad. (2) I am sad all the time and I can't snap out of it. (3) I am so sad or unhappy that I can't stand it.

If you said depression you would be correct. It looks like an item from a depression scale – and it is!

It has face validity.

If someone said this was from a parental attachment scale, then you could say that it lacks face validity.

© A. Taylor


Predictive Validity

Does the measure predict something that it logically should?

The SATAccording to their website, “the SAT is the one test that shows colleges what you know”. College admissions officers use this test to determine who will do well in college.

What future measures do you expect that the SAT would be correlated with?

If you said grades in college you would be right, and the SAT IS correlated with that! – somewhat…

It has predictive validity

© A. Taylor


Concurrent Validity

Does this measure correlate strongly with something that it logically should right now?Unlike predictive validity, concurrent validity compares measures taken at the same time.

What current measures do you imagine that GPA should correlate with?

If you said current test scores and teacher ratings you would be correct.

GPA has some concurrent validity

© A. Taylor


Resources• The College Board (2008). SAT: Know more. Retrieved September

8, 2008 from http://www.collegeboard.com/sat-achieve-more/?s_kwcid=sat%20test|2592657944

• Creasey, G. L. (2005). Research methods in lifespan development. Upper Saddle River, NJ: Pearson Education.

• Family Education Network (2008). American Geography. In Back to school with family education American geography. Retrieved September 8, 2008 from http://quizzes.familyeducation.com/maps/united-states/55376.html

• Korbin, J. L., Patterson, B. F., Shaw, E. J., Mattern, K. D., & Barbuti, S. M. (2008). Validity of the SAT for Predicting First Year College Grade Point Average. In College Board research report #2008-5. Retrieved September 8, 2008 from http://professionals.collegeboard.com/profdownload/Validity_of_the_SAT_for_Predicting_First_Year_College_Grade_Point_Average.pdf

http://quizzes.familyeducation.com/maps/united-states/55376.html

© a. taylor do not duplicate without author’s permission 1 reliability and validity

Documents

authors permission

validity slide

reliability validity

measure slide

terms validity

yardstick slide

measure dont

measurement instrument