Experimental Research I

Day 3

Business

For Tomorrow•One article from Day 2 reading list.•Prepare to summarize & comment•Examine 1 Thesis proposal online•List of 12 research articles in APA format related you your study (bibliography)

Reference List APA example:Kotora, E. J. (2005). Assessment practices in the choral music classroom: A

survey of Ohio high school choral music teachers and college choral methods professors. Contributions to Music Education 32(2), 65-80.

Spaces b/w reference elements = 1; b/w sentences in text = 2.

Experimental Research• Only type of research with an intervention• A direct attempt to influence a particular variable• Only method that can truly begin to untangle cause

and effect hypotheses

• Directional Hypothesis = Theory statement predicting the outcome [directional] (There will be a significant difference…). Reflects researcher’s expectations.– Bilingual 3rd graders taught with the Kodaly method will

demonstrate significantly higher musical achievement than bilingual 3rd graders taught with a traditional eclectic method.

Null Hypothesis• Null Hypothesis = Theory statement predicting the

outcome stated in the negative [non-directional] (There will be no significant difference…) The statistical hypothesis that states that there are no differences between observed and expected data. Does not reflect researcher’s expectations (value free)– There will be no significant difference in musical achievement

of bilingual 3rd graders taught with the Kodaly methods and bilingual 3rd graders taught with a traditional eclectic method.

– The goal is to REJECT the Null Hypothesis based (on 95% Confidence level or above)

– Cannot prove the null hypothesis (a negative)• E.g. not guilty does not = innocent• Reject/not reject vs. accept

Type I and Type II Error• Type I Error is erroneously claiming statistical

significance or rejecting the null hypothesis when in fact, it’s true (claiming success when experiment failed to produce results)– Possible w. incorrect statistical test

• Type II Error is when a researcher fails to reject the null hypothesis when it is in fact false (claiming failure when successful)– The smaller the sample size, the more difficult it is to

detect statistical significance– In this case, a researcher could be missing an

important finding because of study design

Group Comparisons• Experimental Group

– Receives a particular treatment specified by the researcher

• Control/Comparison Group– Does not receive that particular treatment

• Sometimes difficult in educational research to have a strict no-treatment, control– Example: Any instruction is likely to be more effective

than no instruction

Randomization• Random assignment to groups

– Every individual has an equal chance of being in the experimental or control/comparison group

• Supposed to help eliminate extraneous sources of variance– For example… if the groups are sufficiently large, any

differences between groups on extraneous variables are likely to be due to chance or randomly distributed among the groups

• Quasi-Experimental=non-randomized groups– Most ed. research– Intact classes & convenience samples– Impacts ability to generalize to whole population

Variables• Independent variable (IV)

– The experimental or treatment variable– This variable is manipulated by the researcher– Examples: instructional approach, environmental condition, the

introduction of a particular musical element– Participant attribute

• Dependent variable (DV) – Compared b/w groups– The criterion or outcome variable– Examples: student attitudes, student achievement, teacher

effectiveness as measure by ?

• Experiments can be expressed as “The effect of the ‘IV’ on the ‘DV’”

• Extraneous Variables– Those that are not specifically included in the study but never the less

may effect the outcome– Object is to control for extraneous variables– The researcher may not know them all

Manipulating the IV

• Presence of the variable vs. absence of the variable– Kodaly instruction (treatment group) vs. no Kodaly

instruction (control group)• One form of the variable vs. another

– modeling vs. verbal music instruction (vs. control group?)

• Varying degrees of the same variable– 100% positive feedback, no negative feedback vs.

50% positive feedback, no negative feedback

Controlling for Extraneous Variables• Best case scenario – all individuals are as

similar as possible on all variables other than the independent variable

• Methods to control:– Randomization & large sample– Holding variables constant (freeze private lessons)– Build variable into the design (compare private

lessons w/ no private lessons)– Matching pairs – one to control, other to exper.– Statistical control – analysis of covariance

(ANCOVA)

Design and Experiment[Effect of Colored note heads on Music Reading]

• State Hypothesis and Null Hypothesis• Select sample and assign to group (control

and treatment). How many in each?• Identify independent and dependent

variables. Any possible extraneous variables?

• Describe experiment. What will you do w/ each group and for how long? How will you know what they already know?

Discussion of Projects

• On task• Practice explaining your project

– Background; State the problem– Purpose statement– Research questions– Methodology (research design)

Experimental Research Designs

Nomenclature/Abbreviations

• When looking at the symbols used to describe various experimental design approaches:– R = random assignment– O = testing (pre- or post-)– X = treatment– C = control/comparison– M = matched

Pre-Experimental Designs[Pilot Studies – Generally Weak]

• One Shot Case Study (X O)– No random assignment, No control/comparison, no pre-

test• One-Group Pre-test, Post-test (O X O)

– No random assignment, No control/comparison group• Static/Intact-group Comparison X O

– No random assignment O• Static/Intact-group Pre-test, Post-test

– No random assignment, possible pre-test effects O X O O

True Experimental DesignsStronger – not always possible in educ.

• Randomized Post-test Only, R X O Control Group R O

– Still not sure about pre-test levels• Randomized Pre-Test, R O X O Post-test, Control Group R O O

– Good for checking if groups are actually similar at the start of the study and possible effects of pretest

– If you do an experiment – probably this one except w/ non-randomized groups

Randomized Solomon Four-Group Design

• Solomon 4–Group controls for possible sensitization effects due to testing or maturation.

1. R O X O 2. R O O (maturation or pretesting?) 3. R X O (effect of pretest?) 4. R O (control group)

In a successful experiment, what would we expect for each group?What if the Post Test scores for group 2 were as high as the Post Test for group 1?What if the Post Test scores for group 3 were lower than group 1?What if the Post Test scores for group 4 were the same as groups 1 & 2?

Quasi-Experimental Design• So called b/c there is no randomization…• Matching Only

– Participants matched in pairs to control for an extraneous variable rather than randomly assigned

• Equivalent Materials Design (next slide)• Counterbalanced Design (2 slides down)

– Multiple groups receive all treatment types in different order– Average post-test scores across groups are compared to

determine effectiveness/effect of the treatment order– Vulnerable to multiple-treatment interference

• Time-series Design– Outcome measured several times before and after

introduction of the treatment O O O O X O O O O

Equiv. Mat. Design: TuningTuba vs. Clarinet vs. oboe vs. drone stimulus

• Band (N = 1): OX(tu)O; OX(clar)O; OX(ob)O; OX(drone)O

• Pre/Post tests = indiv. tuning to different stimuli. Success mes. w/ a tuner.

• Research questions: Which tuning process leads to most growth? • Most interested w/ growth w/i group per treatment

• Other questions: Could introducing drone second be more effective? How do we know that using just one would be just as or more effective? [next slide]

Counterbalanced Design(Latin Square) Order effect

• All groups take a pretest tuning to the piano A440 – no difference in groups; after time 1, post test serves as next pretest

• Each treatment = 2 weeks w/ post-test at the end• 1 = tuba; 2 = clarinet; 3 = oboe; 4 = drone

Quasi-Experimental Design• Factorial Design (Manipulate 2 or more factors

[IVs] at different levels)– Allows for examination of attribute (vs. manipulated)

variables (i.e. gender, age) and interaction effects b/w combinations of IVs

– Example: Effect of teaching method on beat competency of ELL and non-ELL students

• Possible outcome showing interaction of two IV’s• Non-ELL students may do equally well w/ Kod. and Gordon methods, while

bilingual students may do better w/ Kod. vs. Gordon.• What if you had not separated these groups out?

Factorial Example(Two Way - 2x2)

• Beat competency improvement using Gordon or Kodaly among biling. and non-biling students

• IVs = Language classification (biling. vs. non-biling.) & method (Kodaly vs. Gordon)

• DV = rhythm pre- post-tests• Groups (Six 3rd gr. Sections-3 Kodaly; 3 Gordon;

Bilingual & Non-Bilingual in all groups.)– Bilingual & Kodaly– Non-Bilingual & Kodaly– Bilingual & Traditional– Non-Bilingual & Traditional

2 Way Factorial Designs (2 independent variables [often one manipulated, one attribute)

2X2 (2 levels of both variables)

METHOD

Language Classification

Kodaly Gordon

Bilingual 1

Bilingual 2

Non-Bilingual 1

Non-Bilingual 2

Internal Validity - Effectiveness of Exp. DesignControl of Extraneous Variables: Time Bound Factors

• What happens within the experiment– History – What happens b/w pretest and

posttest (private lessons, change in practice routine)

– Maturation – is change result of treatment natural result of repetition and improvement over time?)

– Mortality – Loss of participants may cause imbalance b/w groups

Internal Validity – Control of Extraneous Variables: Sampling & Measurement

Factors• Testing – pretest affect posttest. Ceiling and floor

effects (eliminate outliers?)• Instrumentation – changes in measurement or

observers (judges at contest from one site to the next)

• Statistical regression – students who score extremely high (ceiling) or low (floor) on pretest may regress to the mean on posttest

• Selection – participants do not represent normal population (also affects external validity)

• Interactions – influence of a combination of the above factors

External Validity – Generalizability

• Population Validity– Extent sample is representative of the population to

which the researcher wishes to generalize the results.• Ecological

– Study conditions and setting are representative of the setting in which the researcher would like to apply the findings (e.g. university lab school)

• Replication– Results cannot be reproduced (problem w/ Mozart

effect)• Detailed description of the sample needed in study

– Important regardless of sampling method

Other Threats to External Validity

• Effect or interaction of testing (testing will not occur in natural setting

• Reactive effects of sample– Hawthorne Effect

• Effects due simply to subjects’ knowledge of being in a study

– John Henry Effect• Control group performs beyond usual level because

they perceive they are in competition with the experimental group

– Teacher or Researcher interactions different than in population

• Subconsciously encouraging or discouraging a group

Review: Effect of Intensive Instruction on Elementary Students’ Memory for Culturally

Unfamiliar Music (2013)• Previous researchers have found that both adults and children demonstrate better

memory for novel music from their own music culture than from an unfamiliar music culture. It was the purpose of this study to determine whether this “enculturation effect” could be mediated through an extended intensive instructional unit in another culture’s music. Fifth-grade students in four intact general music classrooms (two each at two elementary schools in a large U.S. city) took part in an 8-week curriculum exclusively concentrated on Turkish music. Two additional fifth-grade classes at the same schools served as controls and did not receive the Turkish curriculum. Prior to and following the 8-week unit, all classes completed a music memory test that included Western and Turkish music examples. Comparison of pretest and posttest scores revealed that all participants (N = 110) were significantly more successful overall on the second test administration. Consistent with previous findings, participants were significantly less successful remembering items from the unfamiliar music culture, a result that was consistent across test administrations and between instruction and control groups. It appears that the effect of enculturation on music memory is well established early in life and resistant to modification even through extended instructional approaches.

Identify or State:

• Independent Variable• Dependent Variable• Treatment Group• Control Group• Diagram experimental design (O & X)• Write a hypothesis & null hypothesis• Paraphrase findings• Implications for the classroom? Did the authors

reject or not reject the null hypothesis?

Sampling

Samples of individuals/entities

• Sample vs. population– Some vs. All– Examples where entire population could be sampled?

• Relationship between sample specificity and generalizability

• Representative sample– Captures relevant and essential characteristics of the

population– What about a sample of teachers? What should the

sample look like?

Sampling Methods• Systematic

– Random start and sampling interval• i.e., Randomly select pages from IHSA directory• choose every ? Name (random number b/w 1-X)

• Convenience– not as valuable but frequent in ed. research – why?

• i.e., intact classes, pre-service teachers from one institution, conference session attendees

• Purposive– Participants fit a particular profile (female band

directors in small towns)– Exclude those who do not fit profile– Often consists of volunteers (problematic)

Types of Samples• Simple Random

– Everyone has equal chance of selection– Reduce systematic bias – error created by sampling

method– Phone book, MENC membership list (But??)

• Stratified Random– Similar proportions between sample and population

• Gender, race, age, instrument, etc.• Cluster Random

– Groups rather than individuals• i.e., classes or ensembles in CPS• Then groups can be assigned randomly

– Two-stage random - groups then individuals• i.e., choose classes then assign individual students or

groups to control or treatment group

Sample Size• As large as possible given reasonable expenditure of

time and energy– Most likely to get significant results– More statistically powerful (more likely to find a significant

difference b/w groups)• Sample size relative to:

– the size of population (50 Cook Co. band directors vs. 50 band students throughout US)

– variability within population (years of teaching, gender, etc.)– sampling method (need a large enough pool from which to

draw)– study design (qualitative vs. quantitative)