MEASUREMENT AND DATA QUALITY

Areej Faeq

OBJECTIVES After this lecture ,you will able to:- Define new terms in measurement . Identify the important to use measurement in research. Describe the major characteristics of measurement and identify major sources of

measurement error Identify and calculate MEAN,MEDIAN,MODE,&STANDER DEVIATION .

Describe aspects of reliability and validity ,and specify how each aspect can be assessed. Interpret the meaning of reliability and validity . Describe the function and meaning of sensitivity ,specify ,specify ,and likelihood ratios. Evaluate the overall quality of a measuring tool used in study

Measurement

The assignment of numbers to represent the amount of an attribute present in an object or person, using specific rulesAdvantages:Removes guessworkProvides precise informationNumbers are less vague than words and therefore can

communicate information more correctly. e.g Obese than 80Kg

Background

The "levels of measurement" is an expression which typically refers to the theory of scale types developed by the psychologist Stanley Smith Stevens.

Stevens proposed his theory in a 1946 article titled "On the theory of scales of measurement”.

The Theory Of Scale Types

Stevens (1946, 1951) proposed that measurements can be classified into four different types of scales. These were: Nominal Ordinal Interval Ratio

Principles of Measurement:

Classical measurement theory e.g Psychosocial constructs such as depression or social support.

Alternative measurement theory or Item Response theory e.g Cognitive constructs, Achievement or ability.

Levels of Measurement:

A variable’s level of measurement determines what mathematic operations can be performed in a statistical analysis.

Nominal Scale A categorical variable, also called a nominal variable, is

for mutual exclusive, but not ordered categories. Nominal scales are mere codes assigned to objects as

labels, they are not measurements. Not a measure of quantity. Measures identity and

difference. People either belong to a group or they do not. Sometimes numbers are used to designate category

membership.

Examples Eye colour: blue, brown, green, etc. Biological sex (male or female) Public sector , Private sector etc. Married, single, divorced, widowed Country of Origin

1 = United States 3 = Canada2 = Mexico 4 = Other

(Here, the numbers do not have numeric implications; they are simply convenient labels)

What Statistic Can I Apply?

Ordinal Scale

This scale has the ability to rank the individual attributes of two items in the same group but their unit of measurement is not available in this scale, like student A is taller than student B but their actual heights are not available.

Distinguish an ordering: greater than, less than.Does not assume that the intervals between

numbers are equal.

Examples

Rank your food preference where 1 = favourite food and 4 = least favourite: ____ sushi ____ chocolate ____ hamburger

____ papaya Final rankings in a horse race is an ordinal variable. The

horses finish first, second, third, fourth, and so on. The difference between first and second is not necessarily equivalent to the difference between second and third, or between third and fourth.

Examples

Likert scale: How do you feel about Stats?1 = I’m totally dreading this class!2 = I’d rather not take this class.3 = I feel neutral about this class.4 = I’m interested in this class.5 = I’m SO excited to take this class!

Interval Scale

Classifies data into groups or categories Determines the preferences between items Zero point on the internal scale is arbitrary zero, it is not

the true zero point Distinguish an equal-interval ordering. The difference in temperature between 38 degrees C and

40 degrees C is the same as the difference between 36 degrees C and 38degrees C.

Examples

Celsius temperature is an interval variable. It is meaningful to say that 25 degrees Celsius is 3 degrees hotter than 22 degrees Celsius, and that 17 degrees Celsius is the same amount hotter (3 degrees) than 14 degrees Celsius. Notice, however, that 0 degrees Celsius does not have a natural meaning. That is, 0 degrees Celsius does not mean the absence of heat!

What Statistic Can I Apply?

Ratio Scale

This is the highest level of measurement and has the properties of an interval scale; coupled with fixed origin or zero point.

It clearly defines the magnitude or value of difference between two individual items or intervals in same group.

Examples

Measurements of heights of students in this class (zero means complete lack of height).

Someone 6 ft tall is twice as tall as someone 3 feet tall.

Heart beats per minute has a very natural zero point. Zero means no heart beats.

Errors of measurementInstruments that are not perfectly accurate yield measurements containing some error. In classical measurement theory, any observed (Obtained) score can be decomposed conceptually in to two parts : a) An error component b) A true component Obtained score = true score ± error Obtained score An actual data value for a participant (e.g., anxiety scale

score) True score: The score that would be obtained with an

infallible measure Error: The error of measurement, caused by factors that

distort measurement

Many factors contribute to errors of measurement:

There are two types of errors some are random or variable, others are systematic, which represent bias.1. Situational contaminants:-Scores can be affected

by the conditions under which they are produced e.g. A participant’s awareness of an observer’s presence (reactivity).

Other environmental factors are: Temperature , lighting etc.

Many factors contribute to errors of measurement Cont.

2. Response set biases:- Relatively enduring characteristics of respondents can interfere with accurate measures. e.g. social desirability, acquiescence.

3. Transitory personal factors:- A person’s score can be influenced by such temporary personal states as fatigue, hunger, anxiety or mood.

4. Administration Variations:- Alterations in the method of collecting data from one person to the next.

Many factors contribute to errors of measurement Cont.:

5. Item Sampling:- Errors can be introduced as a result of the sampling of items used in the measure.

6. Instrument Clarity:- If the directions for obtaining measures are poorly understood, then scores may be affected by misunderstanding. E.g. Self - report instrument may be interpreted differently by different respondents.

7. Instrument format: Technical characteristics of an instrument. E.g open ended questions yield different information than closed ones.

Question Is the following statement True or False?The true score is data obtained from the actual research study.

FalseThe true score is the score that would be obtained with an infallible measure. The obtained score is an actual value (datum) for a participant.

Summary of Levels of Measurement

Determine if one data value is a

multiple of another

Subtract data values

Arrange data in order

Put data in categories

Level of measurement

NoYesNominalNoNoYesYesOrdinalNoYesYesYesIntervalYesYesYesRatio

NoNo

Yes

Criterion to assess the quality of instrument:-Reliability The consistency and accuracy with which an instrument measures the target attribute. Reliability assessments involve computing a reliability coefficient.1. Reliability coefficients can range from .00 to 1.00. 2. Coefficients below .70 are considered

unsatisfactory.3. Coefficients of .80 or higher are desirable.

The three key aspects of Reliability:1. Stability 2. Internal consistency3. equivalence

Stability

The stability of an instrument is the extent to which similar results are obtained on two separate occasions.■ Assessments of an instrument’s stability involve

procedures that evaluate test – retest reliability. e.g. Administer the same measure to a sample twice and then compare the scores by computing a reliability coefficient, which is an index of the magnitude of the test’s reliabilityStatistical analysis is correlation –coefficient.

How to read a correlation coefficient:Two relationships The possible values for a correlation coefficient ranges

from – 1.00 through .00 to + 1.00. Positive relationship: Positive relationship value should be 1 e.g. Anxiety scale - Administer the scale twice with 2 weeks

duration Negative Relationship: When two variables are inversely related, increases in one

variable are associated with decreases in the second variable. The value of negative relationship is -1.e.g.: IQ is more in tall person

Note:

■The higher the coefficient, the more stable the measure.

■The reliability coefficient is higher for short term retests than long term retests.

Internal consistencyInternal consistency reliability : is a measure of reliability used to evaluate the degree to which different test items that probe the same construct produce similar results.  The extent to which all the items on an instrument are

measuring the same unitary attribute . Evaluated by administering instrument on one occasion. Appropriate for most multi-item instruments. The most widely used approach to assessing reliability. Assessed by computing coefficient alpha (Cronbach’s

alpha) Alphas ≥.80 are highly desirable.

When determining the reliability of a measurement tool, which value would indicate that the tool is most reliable?A. 0.50B. 0.70C. 0.90D. 1.10

EquivalenceThe degree of similarity between alternative forms of

an instrument or between multiple rates/observers using an instrument

Most relevant for structured observations Assessed by comparing agreement between

observations or ratings of two or more observers (inter observer/interrater reliability)

The degree to which two or more independent observers or coders agree about the scoring on an instrument.

Inter ratter reliability can be assessed. When ratings are dichotomous.

To calculate the proportion of agreements Number of agreement ___________________________Number of agreement + disagreements

The statistics used is Cohen’s Kappa which adjust for chance agreements. Multi rater Kappa when more than two raters

Reliability Principles

Low reliability can undermine adequate testing of hypotheses. Reliability estimates vary depending on procedure used to

obtain them. Reliability is lower in homogeneous than heterogeneous

samples. Reliability is lower in shorter than longer multi-item scales. More items tapping the same concept should be added. Items that have no discriminating power should be removed

Validity

The degree to which an instrument measures what it is supposed to measure

Four aspects of validity ( Face validity ,Content validity, Criterion -related validity ,Construct Validity)

It is the degree to which an instrument measures what it is supposed to measure.

A measuring device that is unreliable cannot possibly be valid. Validation efforts should be viewed as evidence gathering enterprises. The more evidence gathered, using various methods to assess

validity, the stronger the inference.

Types of validity:Face validity: Refers to whether the instrument looks as though it is measuring the

appropriate. Based on judgment; no objective criteria for assessmentContent Validity: Concerns the degree to which an instrument has an appropriate

sample of items for the construct being measured and adequately covers the construct domain.

Content validity is relevant for both affective and cognitive measures

Content Valid cont’d

An content validity is necessarily based on judgement. No objective methods to ensure content validity. Use a panel of substantive experts to evaluate and document the

content validity of new instruments. Evaluated by expert evaluation, often via a quantitative measure—

the content validity index (CVI) Validation by minimum of three.

Calculate the Content Validity index,(CVI) Experts rate items on a 4 – point scale of relevance, the item(I) CVI is computed as the number of ratters giving a rating

of either 3 or 4 , divided by the number of experts. I-CVI of .80 is considered an acceptable value. Scale CVI (S) CVI can be also done

Question

Is the following statement True or False?Face validity of an instrument is based on judgment. TrueFace validity refers to whether the instrument looks like it is an appropriate measure of the construct. There are no objective criteria for assessment; it is based on judgment.

Criterion-Related Validity

The degree to which the instrument is related to an external criterionValidity coefficient is calculated by analysing the

relationship between scores on the instrument and the criterion. Two types:1. Predictive validity: the instrument’s ability to distinguish people

whose performance differs on a future criterion2. Concurrent validity: the instrument’s ability to distinguish individuals

who differ on a present criterion

Construct Validity

Concerned with these questions: 1. What is this instrument really measuring? 2. Does it adequately measure the construct of interest?

Methods of Assessing Construct ValidityKnown-groups techniqueTesting relationships based on theoretical predictions or Known groups Technique: The instrument is administered to groups hypothesized to differ on the critical attribute because of some known characteristics. E.g Anxiety among primi & Multi in labour.

Factor Analysis: It is a method for identifying clusters of related variables – that is ,dimensions underlying a central construct . It is a statistical procedure for identifying unitary clusters of items. e,g Assess nursing students confidence in caring mentally ill patients

Methods of Assessing Construct Validity Cont.:Convergent and Discriminant Validity:• An important construct validation tool is a procedure known as the

Multitrait – multimethod matrix method which involves convergence and Discriminability.

• Convergence is evidence that different methods of measuring a construct yield similar results .e.g. Self report, Observation etc.

• Discriminability is the ability to differentiate the construct from other similar constructs. e.g. Psychological & Physical problems (HIV)

Methods of Assessing Construct Validity Cont.:Hypothesized Relationship: Testing hypothesized relationships, often on the basis of theory. e.g. Smoking ---Cancer

Concurrent Validity:

Concurrent Validity refers to a measurement device’s ability to vary directly with a measure of the same construct or indirectly with a measure of an opposite construct. It allows us to show that our test is valid by comparing it with an

already valid test.

Criteria for Assessing Screening/Diagnostic Instruments1. Sensitivity: the instruments’ ability to correctly identify a “case”—i.e.,

to diagnose a condition2. Specificity: the instrument’s ability to correctly identify non cases,

that is, to screen out those without the condition 3. Likelihood ratio: Summarizes the relationship between sensitivity and

specificity in a single numbera. LR+: the ratio of true positives to false positivesb. LR-: the ratio of false negatives to true negatives

Thank you

THE END