multilevel modeling 1.overview 2.application #1: growth modeling break 3.application # 2:...

Multilevel Modeling1. Overview

2. Application #1: Growth Modeling

Break

3. Application # 2: Individuals Nested Within Groups

4. Questions?

Overview

1. What is multilevel modeling?2. Examples of multilevel data structures3. Brief history4. Current applications5. Why multilevel modeling?6. What types of studies use multilevel

modeling?7. Computer Programs (HLM 6

SAS Mixed8. Resources

Multilevel Question

What effects do the following variables have on 3rd grade reading achievement?

School SizeClassroom Climate

Student Gender

What is Multilevel or Hierarchical Linear Modeling?

Nested Data Structures

Several Types of Nesting

1. Individuals Nested Within Groups

Individuals Undivided

Unit of Analysis = Individuals

Individuals Nested Within Groups

Unit of Analysis = Individuals + Classes

… and Further Nested

Unit of Analysis = Individuals + Classes + Schools

Examples of Multilevel Data Structures

Neighborhoods are nested within communities

Families are nested within neighborhoods

Children are nested within families

Examples of Multilevel Data Structures

Schools are nested within districts

Classes are nested within schools

Students are nested within classes

Multilevel Data Structures

Level 4 District (l)

Level 3 School (k)

Level 2 Class (j)

Level 1 Student (i)

2nd Type of Nesting

Repeated Measures Nested Within Individuals

Focus = Change or Growth

Time Points Nested Within Individuals

Repeated Measures Nested Within Individuals

CarlosDay Energy LevelMonday = 0 98Tuesday = 1 90Wednes. = 2 85Thursday = 3 72Friday= 4 70

Repeated Measures Nested Within Individuals

DAY

543210

EN

ER

GY

100

90

80

70

60

Repeated Measures Nested Within Individuals

DAY

543210

EN

ER

GY

100

90

80

70

60 Rsq = 0.9641

Changes for 5 Individuals

0 1.00 2.00 3.00 4.000

25.00

50.00

75.00

100.00

Time

En

erg

y L

evel

Changes in Energy Level Over the Week

3rd Type of Nesting (similar to the 2nd)

Repeated Measures Nested Within Individuals

Focus is not on change

Focus in on relationships between variables within an individual

Repeated Measures Nested Within Individuals

CarlosDay Hours of Sleep Energy

LevelMonday 9 98Tuesday 8 90Wednesday 8 85Thursday 6 72Friday 7 70

Repeated Measures Nested Within Individuals (Not Change)

HOURS

9.59.08.58.07.57.06.56.05.5

EN

ER

GY

100

90

80

70

60

Repeated Measures Nested Within Individuals (Not Change)

HOURS

9.59.08.58.07.57.06.56.05.5

EN

ER

GY

100

90

80

70

60

Repeated Measures Nested Within Individuals

2.00 4.50 7.00 9.50 12.000

25.00

50.00

75.00

100.00

Hours of Sleep

En

erg

y L

evel

Repeated Measures Nested Within Individuals (3 Individuals)

Repeated Measures Within Persons

Level 2 Student (i)

Level 1 Repeated Measures Over Time (t)

Nested Data

Data nested within a group tend to be more alike than data from individuals selected at random.

Nature of group dynamics will tend to exert an effect on individuals.

Nested Data

Intraclass correlation (ICC) provides a measure of the clustering and dependence of the data

0 (very independent) to 1.0 (very dependent)

Details discussed later

Brief Historyof Multilevel Modeling

Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. Sociological Review, 15, 351-357.

Burstein, Leigh (1976). The use of data from groups for inferences about individuals in educational research. Doctoral Dissertation, Stanford University.

Table 1

Frequency of HLM application evidenced in Scholarly Journals

Journal 1999 2000 2001 2002 2003 Total by journal

American Educational Research Journal 3 5 4 3 ? ~15

Child Development 3 2 6 5 13 29

Cognition and Instruction 1 0 0 0 0 1

Contemporary Educational Psychology 0 0 0 0 0 0

Developmental Psychology 2 1 2 5 7 17

Educational Evaluation and Policy Analysis 2 1 5 2 2 12

Educational Technology, Research and Development 0 0 0 0 0 0

Journal of Applied Psychology 1 1 5 7 6 20

Journal of Counseling Psychology 0 2 1 0 0 3

Journal of Educational Computing Research 0 0 0 0 0 0

Journal of Educational Psychology 1 2 3 6 1 13

Journal of Educational Research 2 0 3 3 5 13

Journal of Experimental Child Psychology 0 0 0 0 0 0

Journal of Experimental Education 0 0 0 0 1 1

Journal of Personality and Social Psychology 4 4 6 5 13 32

Journal of Reading Behavior/Literacy Research 0 0 0 0 0 0

Journal of Research in Mathematics Education 0 0 0 0 0 0

Reading Research Quarterly 0 0 0 1 0 1

Sociology of Education 1 2 5 2 1 11

Total by Year 20 20 40 39 49 ~168

Multilevel ArticlesFrequency of Studies Employing HLM in Education or Related Journals

0

25

50

1999 2000 2001 2002 2003

Year

Fre

qu

ency

Total for 19 Journals Reviewed

Journal of Personality and Social Psychology

Child Development

Journal of Educational Research

Some Current Applications of Multilevel Modeling

Growth Curve Analysis Value Added Modeling of

Teacher and School Effects Meta-Analysis

Multilevel Modeling Seems New But….

Extension of General Linear Modeling

Simple Linear RegressionMultiple Linear Regression

ANOVAANCOVA

Repeated Measures ANOVA

Multilevel Modeling

Our focus will be on observed variables (not Latent Variables as in Structural Equation Modeling)

Why Multilevel Modelingvs.

Traditional Approaches?

Traditional Approaches – 1-Level

1. Individual level analysis (ignore group)

2. Group level analysis (aggregate data and ignore individuals)

Problems withTraditional Approaches

1. Individual level analysis (ignore group)

Violation of independence of data assumption leading to misestimated standard errors (standard errors are smaller than they should be).

Problems withTraditional Approaches

1. Group level analysis (aggregate data and ignore individuals)

Aggregation bias = the meaning of a variable at Level-1 (e.g., individual level SES) may not be the same as the meaning at Level-2 (e.g., school level SES)

Multilevel Approach

2 or more levels can be considered simultaneously

Can analyze within- and between-group variability

What Types of Studies Use Multilevel Modeling?

Quantitative

Experimental *Nonexperimental

(Survey, Observational)

How Many Levels Are Usually Examined?

2 or 3 levels very common

15 students x 10 classes x 10 schools

= 1,500

Types of Outcomes

Continuous Scale (Achievement, Attitudes)

Binary (pass/fail) Categorical with 3 + categories

Software to do Multilevel Modeling

SPSS Users2 SAV Files: Level 1

Level 2

HLM 6 (Menu Driven) (Raudenbush, Bryk, Cheong, &

Congdon, 2004)

HLM 6

Software to do Multilevel Modeling

SAS Users

Proc Mixed

Resources (Sample…see handouts for more complete list)

Books Hierarchical Linear Models: Applications and

Data Analysis Methods, 2nd ed. Raudenbush & Bryk, 2002.

Introducing Multilevel Modeling. Kreft & DeLeeum, 1998.

Journals Educational and Psychological Measurement Journal of Educational and Behavioral

Sciences Multilevel Modeling Newsletter

Resources (cont)(Sample…see handouts for more complete list)

Software HLM6 SAS (NLMIXED and PROC MIXED) MLwiN

Journal Articles See Handouts for various methodological

and applied articles Data Sets

NAEP Data NELS:88; High School and Beyond

Self-Check 1

A teacher with 1 classroom of 24 students used weekly curriculum-based measurements to monitor reading over a 14 week period. The teacher was interested in individual students’ rates of change and differences in change by male and female students.

Self-Check 1

How would you classify this situation?

(a) not multilevel(b) 2-level(c) 3-level

Self-Check 2 A researcher randomly selected

50 elementary schools and randomly selected 30 teachers within each school. The researcher was interested in the relationships between 2 predictors (school size and teachers’ years experience at their current school) and teachers’ job satisfaction.

Self-Check 2

How would you classify this situation?

(a) not multilevel(b) 2-level(c) 3-level

Self-Check 3 60 undergraduates from the research

participant pool volunteered for a study that used written vignettes to manipulate the interactional style (warm, not warm) of a professor interacting with a student.  30 randomly assigned students read the vignette depicting warmth and 30 randomly assigned students read the vignette depicting a lack of warmth.  After reading the vignette students used a questionnaire to rate the likeability of the professor.

Self-Check 3

How would you classify this situation?

(a) not multilevel(b) 2-level(c) 3-level

(Select ONLY one)

Growth Curve Modeling

Studying the growth in reading achievement over a two year period

Studying changes in student attitudes over the middle school years

Research Questions

What is the form of change for an individual during the study?

Research Questions

What is an individual’s initial status on the outcome of interest?

Run

Research Questions

How much does an individual change during the course of the study?

Rise Riseb

Run

Research Questions

What is the average initial status of the participants?

Research Questions

What is the average change of the participants?

Research Questions

To what extent do participants vary in their initial status?

Research Questions

To what extent do participants vary in their growth?

Research Questions

To what extent does initial status relate to growth?

Research Questions

To what extent is initial status related to predictors of interest?

Research Questions

To what extent is growth related to predictors of interest?

Design Issues

How many waves a data collection are needed? >2 Depends on complexity of growth

curve

Design Issues

Can there be different numbers of observations for different participants?

Examples Missing data Planned missingness

Design Issues

Can the time between observations vary from participant to participant?

Example: Students observed 1, 3, 5, & 7 months 1, 2, 4, & 8 months 2, 4, 6, & 8 months

Design Issues

How many participants are needed?

More is better Power analyses > 30 rule of thumb

Design Issues

How should participants be sampled?

What you have learned about sampling still applies

Design Issues

What is the value of random assignment?

What you have leaned about random assignment still applies

Design Issues

How should the outcome be measured?

What you have learned about measurement still applies

Example

Context description

A researcher was interested in changes in verbal fluency of 4th grade students, and differences in the changes between boys and girls.

ID    Gender         Time______                              t0    t4    t7

1    0  20    30    302     0          40    44    493 0          45    40    604     0         50    55    595     0          42    48    536 1          45    52    617 1          39    55    638 1          46    58    689 1          44    49    59

Example

Level-1 model specification

0 1 1*( )fluencyY Time error

Example

Level-2 model specification

0 00 01 2

1 10 11

*( )

*( )

G G Gender error

G G Gender

Example

Combined Model

00 01 10

11 2 1

*( ) *( )

*( ) *( )fluencyY G G Gender G Time

G Gender Time error error

Example

SAS program

proc mixed covtest;

class gender;

model score = time gender time*gender/s;

random intercept / sub=student s;

Example

SAS output – variance estimates

Covariance Parameter Estimates  Standard ZCov Parm Subject Estimate Error Value Pr Z Intercept Student 62.5125 35.9682 1.74 0.0411Residual 14.1173 4.9912 2.83 0.0023

Example SAS output – fixed effects

Solution for Fixed Effects  StandardEffect Gender Estimate Error DF t Value Pr > |t| Intercept 39.8103 3.7975 7 10.48 <.0001time 1.5077 0.3295 16 4.58 0.0003Gender F 5.7090 5.6962 16 1.00 0.3311Gender M 0 . . . .time*Gender F 1.0692 0.4943 16 2.16 0.0460time*Gender M 0 . . . .

Example Graph – fixed effects

0

25.00

50.00

75.00

100.00

SC

OR

E

0 2.50 5.00 7.50 10.00

TIME

GENDER = 0

GENDER = 1

Example

Conclusions

Fourth grade girl’s verbal fluency is increasing at a faster rate than boy’s.

Persons Nested in Contexts

Studying attitudes of teachers who are nested in schools

Studying achievement for students who are nested in classrooms that are nested in schools

Research Questions How much variation occurs within

and among groups?

To what extent do teacher attitudes vary within schools?

To what extent does the average teacher attitude vary among schools?

Research Questions What is the relationship among selected

within group factors and an outcome?

To what extent do teacher attitudes vary within schools as function of years experience?

To what extent does student achievement vary within schools as a function of SES?

Research Questions What is the relationship among

selected between group factors and an outcome?

To what extent do teacher attitudes vary across schools as function of principal leadership style?

To what extent does student math achievement vary across schools as a function of the school adopted curriculum?

Research Questions To what extent is the relationship

among selected within group factors and an outcome moderated by a between group factor?

To what extent does the within schools relationship between student achievement and SES depend on the school adopted curriculum?

Design Issues

Consider a design where students are nested in schools

How should schools should be sampled?

How should students be sampled within schools?

Design Issues

Consider a design where students are nested in schools

How many schools should be sampled?

How many students should be sampled per school?

Design Issues

What kind of outcomes can be considered?

Continuous Binary Count Ordinal

Design Issues How will level-1 variables be

conceptualized and measured?

SES

How will level-2 variables be conceptualized and measured?

SES

Terminology Individual growth trajectory – individual

growth curve model A model describing the change process for

an individual Intercept

Predicted value of an individual’s status at some fixed point

The intercept cold represent the status at the beginning of a study

Slope The average amount of change in the

outcome for every 1 unit change in time

I ntercept & Slope I llustration

0

5

10

15

20

25

0 1 2 3 4 5 6 7 8 9 10

Time

Sco

re

RiseRise

bRun

Run

intercept

Curvature =Acceleration=Quadratic Component

0

5

10

15

20

25

30

35

0 1 2 3 4

Time

Sco

re

HLM

Hierarchical Linear Model The hierarchical or nested

structure of the data For growth curve models, the

repeated measures are nested within each individual

Levels in Multilevel Models

Level 1 = time-series data nested within an individual

0 1 *( )Y Time error

Levels in Multilevel Models

Level 2 = model that attempts to explain the variation in the level 1 parameters

0 00 01

1 10 11

*( )

*( )

G G Sessions error

G G Sessions error

More terminology

Fixed coefficient A regression coefficient that does

not vary across individuals Random coefficient

A regression coefficient that does vary across individuals

More terminology Balanced design

Equal number of observations per unit Unbalanced design

Unequal number of observation per unit Unconditional model

Simplest level 2 model; no predictors of the level 1 parameters (e.g., intercept and slope)

Conditional model Level 2 model contains predictors of level 1

parameters

Estimation Methods

Empirical Bayes (EB) estimate “optimal composite of an estimate

based on the data from that individual and an estimate based on data from other similar individuals” (Bryk, Raudenbush, & Condon, 1994, p.4)

Estimation Methods

Expectation-maximization (EM) algorithm An iterative numerical algorithm

for producing maximum likelihood estimates of variance covariance components for unbalanced data.