RESEARCH DESIGN & SURVEYS

TOPICS

Sampling issues The Instrument (surveys) Bias Non experimental research design Experimental vs. quasi-experimental research

designs Research designs with limited power to assess

cause and effect Research design with more power to assess

cause and effect

Sampling Issues

The populationThe Universe •Census •True parameters

The SampleThe sampling FramePROBABILISTIC •Random •StratifiedNON PROBABILISTIC•Convenient •Snowballing, etc.

Does the sample represent the entire population?Can we make any inference?

Making Inferences

Taking only a sample of data from a population becomes necessary as a feasibility/cost issue.

However, even when we seem to have a full population, treating as such means we also assume there is no error in the data coding.

Hence, some believe we should always use inferential statistical methods when working with surveys or other quantitative studies.

Avoiding Biased samples

A sample is valid when it reflects the population from which it is drawn.

The central limit theorem has been proven to show that the larger the sample relative to the population from which it is drawn, the more it will reflect the parameter of the population.

Hence, by definition larger samples are more likely to be accurate than smaller samples.

Surveying Methods

The sample The instrument(aggregate opinion )

The data gathering method

Bias

Systematic bias occurs, and is a huge problem, when bias is incurred directly from the manner in which a survey is collected.

For example, would a survey evaluating perceptions of Democrats be unbiased if conducted exclusively outside a joint meeting of Christian Evangelicals and the NRA?

Bias Sampling error may also occur, and if

minimal not a problem, to the degree that we can not be sure whether we fully captured the parameter of a population.

Nonsampling error results from many other problems of research design, such as poorly worded questions or non-responses of certain subjects. There is always potential error in every empirical project.

Data

The raw data Data ProcessingCodifying data

Data Analysis and Findings Communication

Data Analysis

Findings Communication

Sample Distributions Though covered more thoroughly in a

few weeks, a sample distribution would be the way a statistic would be distributed if drawn from many samples of a population.

Essentially, this is hypothetical and a best guess of what the population parameters may be if we cannot fully know that in reality.

Interviews and Questioning Environment, demeanor, neutrality, are all

necessary to gain samples of data with the least bias.

Focus groups in particular require structure through questions in the hope of excellent qualitative data without leading those who respond to certain answers, although structure may vary: Group interview or Interactive focus group.

The more the researcher needs specific answers to questions, the more they need to structure the session.

Questions Questionnaires require much skill in

order to avoid error. Avoiding biases of those collecting the

information in types of questions. Avoid value-laden or biased words that act

as queues or triggers for certain political/social groups.

Use clear standard language but also make sure your audience understands the words. Example, asking high schools students whether they are taught civics.

Structure of Questions

Open v. Closed. As with focus groups, the more you structure answer options the more you can have answers to specific questions. Open-ended questions are good for a full range of respondent views.

Instrumentation: How the instrument (e.g. survey) is conducted. For example, a survey through internet will exclude those without access to such technology.

Selection: Selection effects based on who is and who isn’t included in the experiment or survey. Sampling and instrumentation problems.

Additional Problems to be aware of

Additional problems of survey instrumentation or experiments

A rival hypothesis is an alternative explanation that may invalidate what we think as cause and effect. There are a few examples:

History: Events that happen in the outside world during the experiment. Ask opinions about civil liberties after 9/11 compared to before, when 9/11 is not the focus of the study.

Maturation: Acquisition and processing of information by subjects is not constant through time. Could be a real problem for some panel studies.

Additional Problems to be aware of

Rivalry between treatment and control group: How aware are the subjects of the experiment. In a survey, this is called the Effect, where respondents try to guess at or please the person asking the questions.

A combination (interaction) between the above threats to validity. (e.g. instrumentation-history)

Research Design

I want to study the relationship between level of development and income distribution in developing countries.

I need a hypothesis I need a strategy (research design) to

undertake hypothesis testing I need to be aware that there is not a

perfect strategy (research design) but I need to do the best possible research.

The hypothesis

The economic development theory suggests that as countries get richer income distribution worsens but eventually income distribution improves.

LEVEL OF DEVELOPMENT

GAPDIST.

LOW HIGH

LOW

HIGH

Non experimental research

CROSS SECTIONAL

LONGITUDINAL

TREND

COHORT

PANEL

Source: Sampieri et al. 1998:192

NON EXPERIMENTAL RESEARCH

CROSS SECTIONAL: The researcher would take a sample of countries with different levels of development (low, medium, high) and analyze the match between the theory and facts.

This approach is the weakest to determine causality, but strong in generalizing.

GINI CONCENTRATION RATIOS LOW INCOME GINIBangladesh .375Sri Lanka .485MIDDLE INCOMEMexico .523Brazil .569Costa Rica .485HIGH INCOMEUSA .369Sweden .288Canada .338Japan .285

GINI=0 TOTAL EQUALITYGINI =1TOTAL INEQUALITY

NON EXPERIMENTAL RESEARCH LONGITUDINAL: The researcher would select

a country or countries that have move from lower to higher levels of development (eg. USA, Canada, etc.) and determine whether income distribution over time followed the trend suggested by the theory

Causality can be infer but lacks detail about the true causes as well as weak on generalization.

LONGITUDINAL TENDENCY: EVOLUTION OF INCOME

DISTRIBUTION OVER TIME IN MEXICO. COHORT: EVOLUTION OF INCOME

DISTRIBUTION OVER TIME OF MEXICAN WOMEN BORN AFTER 1960.

PANEL: EVOLUTION OF INCOME DISTRIBUTION OF THE SAME GROUP OF WOMEN OVER TIME (MARIA, SUSAN, BRENDA, JOAN, GLORIA & ISABEL)

CASE (S) STUDY

The researcher will take a case (s) study and analyze in depth what factors (policies, institutions, political systems, culture, etc.) are associated with improving income distribution in a given country.

Why is income distribution better in Chile than in Mexico and Brazil despite having the same level of development?

Improves details and understanding of causes and effect but lacks generalization.

Experimental vs. Quasi-experimental Research Designs

Experimental research design: The researcher has control over the experiment in terms of sample selection, treatment, environment, etc.

Experimental designs are typical in psychology, medicine, education, etc.

Quasi-experiments: The researcher does not have control over the experiment, rather the experiment occurs in a “natural” setting.

Quasi-experimental design are typical in economics, sociology, public administration, urban planning, political sciences, etc.

RESEARCH DESIGN

0t = Observation in time t of experimental group

X = Treatment

0c = Control group

Research design with limited power

POST-TEST ONLY

X O 1

POST-TEST WITH CONTROL GROUP

X O1

Oc

PRE-TEST POST-TEST

O1X O2

PRE-TEST POST-TEST WITH CONTROL GROUP

O1X O2

Oc Oc

Research designs with more causal power

CONTROL WITH MORE OBSERVATION IN THE PRETEST

O1 O2 O3 X O4

Oc Oc Oc Oc

PRE-TEST POST-TEST REMOVING THE TREATMENT

O1 X O2 X O3 X 04

CHANGES TO LOOK FOR

CONVERGENCE-DIVERGENCE

1. Positive change in the treatment group without change in the control group

Treatment

Control

Treatment

Pre-test Post-test

Divergence

Positive increments at a different rate

Treatment

Control

Pre-test Post-test

Convergence

The treatment group catches up with the control group

Control

Treatment

Pre-test Postest

Cross pattern

The treatment group overpass the control group

Treatment

Control

Pre-test Post-test

Research design with more power (time series) Pre-test post-test

O1 O2 O3 X O4 O5 O6

Pre-test post-test with control group

O1 O2 O3 X O4 O5 O6

Oc Oc Oc Oc Oc Oc

Changes to look for in time series

TIMEX

No effect

Change on the rate or slope

Change on the intercept

PRE-TEST POST-TEST