evaluating research claims. evaluating tv ad claims an ad on tv says, “use cocoa creme. you’ll...

Evaluating Research Evaluating Research ClaimsClaims

Evaluating TV Ad ClaimsEvaluating TV Ad Claims

An ad on TV says,An ad on TV says,

““USE Cocoa Creme. You’ll have USE Cocoa Creme. You’ll have youthful skin in 10 minutes. All youthful skin in 10 minutes. All natural ingredients.”natural ingredients.”

You say, “Yeah, sure! Youthful You say, “Yeah, sure! Youthful skin. skin.

I bet your face falls off in 10 I bet your face falls off in 10 minutes.”minutes.”


Another ad tells you to,Another ad tells you to,

““Use Motor Magician. Your Use Motor Magician. Your engine will purr like a kitten. And engine will purr like a kitten. And you’ll get 80 miles per gallon!”you’ll get 80 miles per gallon!”

You say, “I don’t want a motor You say, “I don’t want a motor that sounds like a cat. 80 miles that sounds like a cat. 80 miles per gallon? HA! Baloney.”per gallon? HA! Baloney.”


In other words…. In other words….

You aren’t You aren’t fooled fooled by ads by ads promising what is very promising what is very unlikelyunlikely, , and that give and that give no EVIDENCEno EVIDENCE to to support the inflated promises.support the inflated promises.

Evaluating Medical ClaimsEvaluating Medical Claims

The same goes with The same goes with medicinemedicine..

Let’s say your child is ill with an infection.Let’s say your child is ill with an infection.The physician says, “Here’s a prescription The physician says, “Here’s a prescription for Megabiotic.”for Megabiotic.”You ask, “Will it work, Doctor?”You ask, “Will it work, Doctor?”Physician. “Oh, yes. It will fix her right Physician. “Oh, yes. It will fix her right up.”up.”You. “Does it have bad side effects?”You. “Does it have bad side effects?”Physician. “Oh, no. No side effects.”Physician. “Oh, no. No side effects.”

You. You. “How do you know, “How do you know, Doctor?”Doctor?”


““How do you know, DoctorHow do you know, Doctor?”?”That is That is THE THE question. question. ““HowHow do you know?” do you know?”

““What What makes you so sure?”makes you so sure?”

““Do you have Do you have trustworthy trustworthy evidenceevidence to back up your claim?” to back up your claim?”


Will you let your child take the prescribed Will you let your child take the prescribed medication if these (below) are the medication if these (below) are the physician’s answers?....physician’s answers?....

• ““Well, I and my doctor friends Well, I and my doctor friends think it think it worksworks very well.” very well.”

• ““In a study of In a study of five personsfive persons, everyone got , everyone got well.”well.”

• ““In a study of 1000 persons, In a study of 1000 persons, many said many said they felt much betterthey felt much better.”.”

You wouldn’t be satisfied by these You wouldn’t be satisfied by these answers! answers!

In fact, you’d tell the physician, “See you In fact, you’d tell the physician, “See you later!”later!”


What What DODO you want the physician to tell you you want the physician to tell you??

How about these?...How about these?...• Megabiotic was Megabiotic was tested in 10 experimental tested in 10 experimental

studiesstudies..• The research was The research was notnot conducted by the drug conducted by the drug

company, but by company, but by independent researchersindependent researchers who had nothing to gain whether who had nothing to gain whether Megabiotic was effective or not effective.Megabiotic was effective or not effective.

• Each study lasted from Each study lasted from one to five yearsone to five years..• Each study had at least Each study had at least 1000 persons1000 persons in it. in it.

[More][More]


• Each study Each study samplesample had had a mix of different kinds a mix of different kinds of personsof persons in it: younger/older, more ill/less ill, in it: younger/older, more ill/less ill, male/female.male/female.

• Each study Each study compared the effects of Megabiotic compared the effects of Megabiotic with with other medications medications (alternative treatments) (alternative treatments) and with and with no no treatmentstreatments (control group). (control group).

• Each study Each study measured the exact amountmeasured the exact amount of of infection infection beforebefore treatment, each week treatment, each week duringduring treatment, and at the treatment, and at the endend of the course of of the course of treatment. The measurement was treatment. The measurement was objectiveobjective------the amount of infection in the blood.the amount of infection in the blood.

• Side effects were also measuredSide effects were also measured, such as blood , such as blood pressure, skin rash, and kidney function.pressure, skin rash, and kidney function.

[More][More]


• All of the studiesAll of the studies found that persons who found that persons who received Megabioticreceived Megabiotic had had NO NO serious side serious side effects. effects.

• Persons who received MegabioticPersons who received Megabiotic sshowedhowed directly observable directly observable (in the blood)(in the blood) reductionsreductions in infection each week until, in infection each week until, by the end of one year, by the end of one year, 95%95% of the of the persons were cured.persons were cured.

Now Now would you let your child take the would you let your child take the medication?medication?


Probably.Probably.

Why?Why?

Why is the second set of answers Why is the second set of answers from the doctor convincing?from the doctor convincing?

[Think about it.][Think about it.]


The second answers are more The second answers are more convincing because: convincing because:

• The second answers are The second answers are more more believablebelievable..

• The second answers give you The second answers give you solid solid REASONSREASONS to believe that the to believe that the medication is likely to cure your child’s medication is likely to cure your child’s infection and not cause harm.infection and not cause harm.

• The physician gave you The physician gave you evidenceevidence that is that is hard to dismiss as just baloney.hard to dismiss as just baloney.

[More][More]


• You can’t say, “It won’t work with You can’t say, “It won’t work with mymy child,” child,” because the research on Megabiotic was because the research on Megabiotic was tested with thousandstested with thousands of persons---many of of persons---many of whom were whom were just like your childjust like your child..

• You can’t say, “Maybe persons got well by You can’t say, “Maybe persons got well by themselves. It wasn’t Megabiotic,” because themselves. It wasn’t Megabiotic,” because persons in the control grouppersons in the control group (who got no (who got no Megabiotic) Megabiotic) did did NOTNOT get wellget well..

• You can’t say, “They didn’t measure You can’t say, “They didn’t measure improvement,” because in fact they improvement,” because in fact they measured what can be SEENmeasured what can be SEEN (the amount of (the amount of infection in the blood)---an infection in the blood)---an objective objective measuremeasure of improvement, not an opinion. of improvement, not an opinion.

[More][More]


• You can’t say, “The effects are probably You can’t say, “The effects are probably temporarytemporary; persons will get sick again,” ; persons will get sick again,” because some of the because some of the studies lasted five studies lasted five yearsyears, and no one got sick again with , and no one got sick again with the infection.the infection.

• You can’t say, “The research was You can’t say, “The research was riggedrigged to get good results. For example, to get good results. For example, researchers dropped from the study all researchers dropped from the study all persons who were NOT getting well.” persons who were NOT getting well.” You can’t say this because the drug You can’t say this because the drug company (that wanted the drug to company (that wanted the drug to work) work) did not DO the research.did not DO the research.

Summary So FarSummary So Far

Let’s summarize. Let’s summarize.

What makes research What makes research believablebelievable? ?

What makes the What makes the conclusionsconclusions of of research…research…

””The (drug, treatment, or The (drug, treatment, or method) is method) is effective. Therefore, effective. Therefore, it’s a good idea to it’s a good idea to use it.”…use it.”…

……so so trustworthytrustworthy that that you can be you can be confident in your decisionconfident in your decision to use it? to use it?


Here is what makes research believable and Here is what makes research believable and the conclusions trustworthy.the conclusions trustworthy.

• The The effect effect of the “thing” being tested of the “thing” being tested (program, materials, treatment, or method) is (program, materials, treatment, or method) is defined and measured defined and measured objectivelyobjectively..

• The thing being tested is The thing being tested is tested against tested against alternativesalternatives (to see which works (to see which works betterbetter) and ) and against against nono treatment treatment (to see if the thing works (to see if the thing works at all-at all---that is, improvement is much greater --that is, improvement is much greater than when there is NO treatment).than when there is NO treatment).

• The research is The research is done many timesdone many times ( (replicatedreplicated), ), to ensure that results are not chance, not a to ensure that results are not chance, not a fluke.fluke.

• The research is done The research is done for a long period of timefor a long period of time ((longitudinallongitudinal), to see if the effects are long-), to see if the effects are long-lasting.lasting.

[More][More]


• The research is done with The research is done with samples of personssamples of persons who have the who have the same featuressame features as persons in the as persons in the larger populationlarger population who might who might later receive the program, later receive the program, treatment, or method. treatment, or method. [[representative samplerepresentative sample]]

• The researchers do not have a The researchers do not have a stake in the outcomes. stake in the outcomes. [[independent researchindependent research]]

Let’s Apply This to EducationLet’s Apply This to Education• Just as you feel responsible for the care of Just as you feel responsible for the care of

persons you love, you also feel persons you love, you also feel responsible responsible for the children you teachfor the children you teach..

• You don’t want students to be taught with You don’t want students to be taught with curriculum materials or with teaching curriculum materials or with teaching methods that methods that don’t work and that cause don’t work and that cause harm-harm---for example, teaching students the --for example, teaching students the wrong way to read or to solve math problems.wrong way to read or to solve math problems.

• Therefore, you want to Therefore, you want to ask tough questionsask tough questions about researchabout research done done on curriculum on curriculum materials and teaching methods that are materials and teaching methods that are advertised, or that a school district wants you advertised, or that a school district wants you to use, or that your colleagues say are to use, or that your colleagues say are terrific.terrific.What tough questions should you ask about What tough questions should you ask about

research?research?

Questions About Questions About PurposePurpose.. Selling?Selling?

What tough questions should you ask about What tough questions should you ask about research?research?

Let’s say you are reading a research report, or a Let’s say you are reading a research report, or a description of a new program on a website, or an description of a new program on a website, or an advertisement for curriculum materials. advertisement for curriculum materials.

Does the ad, report, or program description Does the ad, report, or program description sound as if the authors are sound as if the authors are selling an idea or a selling an idea or a product—to persuade youproduct—to persuade you that they are right; that they are right; that they have the answer; that they can fix that they have the answer; that they can fix schools and raise achievement? schools and raise achievement?

””Just (believe us, join us, buy our Just (believe us, join us, buy our materials).”materials).”If selling is their purpose, this is NOT good.If selling is their purpose, this is NOT good. Why?Why?You can be almost certain that the You can be almost certain that the only only data data (evidence) the writers will show are data that (evidence) the writers will show are data that supportsupport their claims. They will not show data their claims. They will not show data that that do NOT support their claims. [This is called do NOT support their claims. [This is called disdishonest.]honest.]

Purpose:Purpose: SellingSellingIn other words, you should be In other words, you should be highly highly suspicioussuspicious of their claims about of their claims about effectivenesseffectiveness. .

““Our method works.” Our method works.” ““Our ideas are sound.” Our ideas are sound.” ““Use our ideas and program.”Use our ideas and program.”

These claims These claims do not come from objective do not come from objective scientific researchscientific research. They come from persons . They come from persons and groups who are making up a story (a and groups who are making up a story (a sales pitch) that will get you to buy. sales pitch) that will get you to buy. If they If they had solid research evidencehad solid research evidence that their that their ideas, materials, or procedures were ideas, materials, or procedures were effective, their writing would not sound like effective, their writing would not sound like a sales pitch. a sales pitch.

Purpose:Purpose: ReportingReportingHowever, the purpose of a journal article or a However, the purpose of a journal article or a program description may simply be to program description may simply be to REPORT REPORT the findings of research.the findings of research. For example, the For example, the authors are authors are telling telling you what they learned---you what they learned---not not what they want you to believe or to dowhat they want you to believe or to do..

For instance, authors may report which one of For instance, authors may report which one of three methods of peer tutoring produces the three methods of peer tutoring produces the highest achievement.highest achievement.

Or what percentage of fourth graders are Or what percentage of fourth graders are proficient in reading on grade level after a new proficient in reading on grade level after a new reading program was used.reading program was used.

Simple reporting is good.Simple reporting is good. If authors are If authors are notnot trying to sell you an idea or materials or trying to sell you an idea or materials or programs, then they are probably going to programs, then they are probably going to report honestlyreport honestly what they found; they have what they found; they have nothing to gain simply by passing on nothing to gain simply by passing on information.information.

Questions About the Questions About the Research StrategyResearch Strategy

There are different KINDS of There are different KINDS of research research questionsquestions; for instance, there are ; for instance, there are questions….questions….

• About persons’ About persons’ opinionsopinions. “How do . “How do teachers feel about….?teachers feel about….?

• About what persons About what persons dodo. “How do students . “How do students respond when teachers….?”respond when teachers….?”

• About About factors that cause or predict factors that cause or predict changeschanges. “Does timely correction of . “Does timely correction of errors reduce those same errors errors reduce those same errors happening in the future?”happening in the future?”

• About what is About what is effectiveeffective. “Which program . “Which program (A, B, or C) is more effective in raising the (A, B, or C) is more effective in raising the reading achievement of diverse learners?”reading achievement of diverse learners?”

Questions About the Questions About the Research StrategyResearch Strategy

There are different kinds of There are different kinds of research questionsresearch questions, and there are , and there are different kinds of different kinds of research research strategiesstrategies---or WAYS of trying to ---or WAYS of trying to answer the questions. For answer the questions. For instance, there are: instance, there are:

• Surveys.Surveys.• Observation.Observation.• Experimentation.Experimentation.

Questions About the Questions About the Research StrategyResearch StrategyHere’s a big ideaHere’s a big ideaCertain kinds of research Certain kinds of research questionsquestions require require certain certain research research strategiesstrategies, , or else the research will NOT give evidence or else the research will NOT give evidence (data, findings) that you can trust.(data, findings) that you can trust.

Do you have high blood pressure? Do you have high blood pressure? Which is the better research strategy to get an Which is the better research strategy to get an answer you will trust? answer you will trust?

• Do a Do a surveysurvey of the of the opinionsopinions of 20 of your of 20 of your friends. friends.

• Use a Use a blood pressure cuffblood pressure cuff (every day for a (every day for a week, at different times of day) week, at different times of day) to get exact to get exact measurement.measurement.

Blood pressure is Blood pressure is NOT an NOT an opinionopinion-kind-of--kind-of-thingthing! ! It is an It is an objective thingobjective thing. PRESSURE. . PRESSURE. Therefore, it requires an Therefore, it requires an OBJECTIVE OBJECTIVE measurement---measurement---with the cuff.with the cuff.

Questions About the Research StrategyQuestions About the Research StrategyHere are different kinds of research questions Here are different kinds of research questions and the proper research strategies.and the proper research strategies.

• What are teachers’ What are teachers’ opinionsopinions about a new math about a new math program that they are using?program that they are using?

• How well do students How well do students think think they read three they read three months after they’ve been taught with a new months after they’ve been taught with a new reading program?reading program?

• How How satisfiedsatisfied are teachers with the are teachers with the supervision and professional development supervision and professional development they receive?they receive?

• What are the What are the evaluationsevaluations of parents on the of parents on the helpfulness of their children’s teachers?helpfulness of their children’s teachers?

These are opinion questions.These are opinion questions.

Is the Research Strategy Appropriate Is the Research Strategy Appropriate for the Research Questions?for the Research Questions?

These are opinion questions.These are opinion questions.

The researcher wants information The researcher wants information on how teachers, students, and on how teachers, students, and parents parents SEE SEE things.things.

Therefore, a useful strategy Therefore, a useful strategy would be would be SURVEY researchSURVEY research. .

Example of Example of surveysurvey research… research…

Is the Research Strategy Appropriate Is the Research Strategy Appropriate for the Research Questions?for the Research Questions? Survey Survey

ResearchResearchConsider the question on parents’ Consider the question on parents’

satisfaction.satisfaction.• The researcher selects a The researcher selects a samplesample of families. of families. • This sample is This sample is representative of the whole representative of the whole

population of familiespopulation of families of children at of children at schools; for example, small and large schools; for example, small and large families; two-parent and single-parent families; two-parent and single-parent families; families with one child and families; families with one child and families with more children; poor families families with more children; poor families and affluent families. and affluent families.

• Then the researcher either uses a Then the researcher either uses a questionnaire questionnaire (that parents fill out); or uses (that parents fill out); or uses interviews (face-to-face) interviews (face-to-face) in which the in which the researcher asks the same kinds of questions researcher asks the same kinds of questions as on the questionnaire; or gives as on the questionnaire; or gives questionnaires to all families and questionnaires to all families and also also interviews a smaller sampleinterviews a smaller sample to obtain more to obtain more detailed information.detailed information.

Is the Research Strategy Appropriate Is the Research Strategy Appropriate for the Research Questions?for the Research Questions? Survey Survey

ResearchResearchThe researcher can examine (The researcher can examine (analyzeanalyze) the survey ) the survey

information (information (datadata) from the questionnaires or ) from the questionnaires or

interviews, and can then determine:interviews, and can then determine:• What kinds of help parents feel is What kinds of help parents feel is adequateadequate..• What kinds of help parents feel is What kinds of help parents feel is inadequateinadequate..• What What percentagepercentage of parents are satisfied vs. of parents are satisfied vs.

not satisfied with teachers’ helpfulness.not satisfied with teachers’ helpfulness.

These findings could assist school These findings could assist school administrators to administrators to

improve family-school relations.improve family-school relations.


Here is a different kind of research question.Here is a different kind of research question.• What are the different What are the different waysways that teachers that teachers

respond or try to handle students’ respond or try to handle students’ disruptive behavior? What are the disruptive behavior? What are the effects effects of each wayof each way??

• When students are doing peer tutoring, do When students are doing peer tutoring, do students learn to play the role of tutor and students learn to play the role of tutor and the role of tutee? If so, the role of tutee? If so, what are the what are the behaviors in these rolesbehaviors in these roles??

• When a class is organized into learning When a class is organized into learning centers (e.g., reading, math, science), and centers (e.g., reading, math, science), and students are allowed to come and go as students are allowed to come and go as they wish, they wish, what do students actually DO in what do students actually DO in the centersthe centers? ?


These are questions about what These are questions about what persons persons DODO---about observable ---about observable activityactivity---NOT about how persons ---NOT about how persons see see things---their opinions.things---their opinions.

Therefore, a useful research Therefore, a useful research strategy is strategy is OBSERVATIONAL OBSERVATIONAL researchresearch..

Example of Example of observationobservation research…research…

Is the Research Strategy Appropriate for the Is the Research Strategy Appropriate for the Research Questions?Research Questions? Observational ResearchObservational Research

Consider the question of how teachers respond to Consider the question of how teachers respond to disruptive disruptive

behavior.behavior.• The researcher The researcher selects a sampleselects a sample of classrooms. of classrooms.• Standing out of the way, the researcher Standing out of the way, the researcher takes running takes running

notesnotes (like a sportscaster) on teacher-student (like a sportscaster) on teacher-student interaction. interaction.

• After several weeks, the researcher examines the notes After several weeks, the researcher examines the notes and and a. Identifies a. Identifies different kinds of different kinds of disruptive behaviordisruptive behavior; ; e.g., e.g., making noises, talking out of turn, making noises, talking out of turn, noncompliance with noncompliance with teacher requests.teacher requests.b. Identifies b. Identifies how the teacher typically how the teacher typically respondsresponds to to each kind each kind of disruptive behaviorof disruptive behavior; e.g., telling a ; e.g., telling a student to stop, student to stop, threatening to send the student to the threatening to send the student to the office, ignoring the office, ignoring the disruptive behavior but praising disruptive behavior but praising students when they are students when they are engaging in desirable engaging in desirable behavior.behavior.c. c. Determining what generally happensDetermining what generally happens when when teachers teachers respond in different ways; e.g., students do respond in different ways; e.g., students do the disruptive the disruptive behavior more often vs. less often.behavior more often vs. less often.

Is the Research Strategy Appropriate for Is the Research Strategy Appropriate for the Research Questions?the Research Questions?

Here is yet another Here is yet another kind kind of research of research questionquestion..

• Is peer tutoring Is peer tutoring more effectivemore effective (does it (does it yield higher achievement) for some yield higher achievement) for some subjectssubjects than other subjects? than other subjects?

• Are teachers’ Are teachers’ opinionsopinions about their new about their new math programs math programs affected byaffected by whether a whether a math program provides written math program provides written procedures for teachers to follow vs. procedures for teachers to follow vs. does not provide written procedures for does not provide written procedures for teachers to follow?teachers to follow?

[More][More]


• Under what conditionsUnder what conditions (e.g., size of (e.g., size of school, recent levels of student school, recent levels of student achievement, quality of teaching staff) is achievement, quality of teaching staff) is it better (there is faster change, less it better (there is faster change, less opposition) when school reform is rapid opposition) when school reform is rapid and extensive vs. slow and in small steps? and extensive vs. slow and in small steps? [In other words, [In other words, what factors predictwhat factors predict which kind of school reform works better.]which kind of school reform works better.]

• Which math programWhich math program yields the most math yields the most math proficiency in students overall and proficiency in students overall and students by subgroups, such as ethnicity students by subgroups, such as ethnicity and social class?and social class?

These are questions about These are questions about causationcausation..


These are questions about These are questions about causationcausation. The idea . The idea

is thatis that

Certain factors Certain factors Predict or Predict or produceproduce

or variables changes in or variables changes in other factorsother factors or variables or variables

Is the Research Strategy Appropriate for the Is the Research Strategy Appropriate for the Research Questions?Research Questions?

Here are examples of questions about which factors or Here are examples of questions about which factors or variables predict or produce (cause) effects in other variables predict or produce (cause) effects in other

variables.variables.• Which Which features of a math programfeatures of a math program (Teaching (Teaching

procedures? Practice? How tasks are sequenced?) procedures? Practice? How tasks are sequenced?) have the largest effects on (cause the largest have the largest effects on (cause the largest changes in) changes in) learninglearning? ?

• Which Which features of a school features of a school affect (cause or inhibit) affect (cause or inhibit) the the success of effortssuccess of efforts to reform the school to reform the school curriculum?curriculum?

• How does How does subject mattersubject matter affect (cause or inhibit) affect (cause or inhibit) learning through peer tutoringlearning through peer tutoring??

• Do Do written procedureswritten procedures or the absence of written or the absence of written procedures in a math program affect (cause) procedures in a math program affect (cause) teachers’ opinionsteachers’ opinions of the program? of the program?


This kind of question…. This kind of question…. • Which factors cause what Which factors cause what

changeschanges??• Which curriculum, program, or Which curriculum, program, or

procedure is procedure is more effectivemore effective?.... ?....

…….requires the strategy of .requires the strategy of experimental experimental

researchresearch..

Is the Research Strategy Appropriate for the Is the Research Strategy Appropriate for the Research Questions?Research Questions? ExperimentExperiment

The essential feature of the strategy of The essential feature of the strategy of experimental experimental researchresearch is that you… is that you…

CompareCompare two or more situations (e.g., schools) two or more situations (e.g., schools) that are that are as similaras similar as you can make them (e.g., % as you can make them (e.g., % of students who are poor, size of school, quality of of students who are poor, size of school, quality of teaching staff), but these schools are teaching staff), but these schools are different different in in the the few factorsfew factors (e.g., reading materials) (e.g., reading materials) whose whose effectseffects you are trying to find out, or you are trying to find out, or testtest..

IfIf the only main way that the schools differ is the the only main way that the schools differ is the reading materials used, and if one school has reading materials used, and if one school has

consistently consistently higher reading achievement than the others, then higher reading achievement than the others, then it it

isisvery likely that the very likely that the reading materialsreading materials make the make the

difference. difference. What ELSE What ELSE could could it be? After all…. it be? After all….


ExperimentExperiment• The factor that makes the difference in The factor that makes the difference in

reading achievement reading achievement can’t becan’t be class size-class size-----because this is because this is the samethe same across all the across all the schools.schools.

• The factor that makes the difference in The factor that makes the difference in reading achievement reading achievement can’t becan’t be teacher teacher qualityquality---because this is ---because this is the samethe same across all across all the schools.the schools.

• The factor that makes the difference in The factor that makes the difference in reading achievement reading achievement can’t be thecan’t be the percentage of students who are poorpercentage of students who are poor------because this is because this is the samethe same across all the across all the schools.schools.Therefore, the factor that makes the Therefore, the factor that makes the difference in reading achievement difference in reading achievement must be must be the ONE major way the schools are the ONE major way the schools are differentdifferent----reading ----reading curriculumcurriculum..


However, not all experiments give information that you However, not all experiments give information that you can can

trusttrust.. It depends on how the experiment is carried out. It depends on how the experiment is carried out. For For

example…example…1. 1. Maybe students in schools with the highest reading Maybe students in schools with the highest reading

achievement achievement were already reading at a higher levelwere already reading at a higher level.. Therefore, these students don’t have to learn very much Therefore, these students don’t have to learn very much in order to come out first. In other words, maybe in order to come out first. In other words, maybe students’ students’ pre-skills pre-skills (and (and notnot the curriculum materials) the curriculum materials) account for higher achievement.account for higher achievement.

To rule out this possibility, an experiment should have To rule out this possibility, an experiment should have pre-pre-

test measurestest measures of the variables whose change we are of the variables whose change we are interested interested

in. For instance, we should measure reading skills in all of in. For instance, we should measure reading skills in all of the the

schools schools BEFOREBEFORE new reading materials are introduced, to new reading materials are introduced, to see if some classes are ahead of other classes.see if some classes are ahead of other classes.


So, if an article says that a program or So, if an article says that a program or method produced more change or is more method produced more change or is more effective than another program or method, effective than another program or method,

ask ask if the researchers did if the researchers did pre-testspre-tests to show that to show that students all students all started at the same levelstarted at the same level.. If they If they did not do pre-tests, then you cannot trust did not do pre-tests, then you cannot trust

the the claim about effectiveness. claim about effectiveness. The group with The group with

the the highest achievement may have started at a highest achievement may have started at a higher level.higher level.


2. 2. How do you know how much students learned?How do you know how much students learned? The The only way to find out is to do only way to find out is to do postpost-tests-tests (e.g., at the end (e.g., at the end of the semester and year) to see where students’ of the semester and year) to see where students’ reading achievement reading achievement ended upended up. And you have to see . And you have to see how much students how much students changedchanged (learned, achieved) (learned, achieved) betweenbetween the pre-test the pre-test (reading proficiency at the start) (reading proficiency at the start) and the post-testand the post-test (reading achievement at the end). (reading achievement at the end). And you need to And you need to report report how much differencehow much difference there is there is between the pre-test and post-test scoresbetween the pre-test and post-test scores..

If researchersIf researchers do NOT do post-tests, or do not REPORT do NOT do post-tests, or do not REPORT the the

size of differences between the pre-test and post-testsize of differences between the pre-test and post-test, , then you then you

don’t know don’t know HOW MUCHHOW MUCH students learned as a result of students learned as a result of the the

curriculum that was used curriculum that was used betweenbetween the pre-test and the the pre-test and the post-post-

test. test. Therefore, you can’t trust claims that one Therefore, you can’t trust claims that one curriculum curriculum

produced more achievement than the others.produced more achievement than the others.


3. 3. Were measures objective? Were measures objective? You don’t rely on a physician’s You don’t rely on a physician’s subjective opinionsubjective opinion about about

your your health.health.

““You look fine You look fine to meto me.”.”Maybe the physician is right.Maybe the physician is right.But what if the physician’s opinion is wrong?But what if the physician’s opinion is wrong?Just so, you want Just so, you want objective evidence ofobjective evidence of (in this (in this

experiment) experiment) reading achievementreading achievement. It is nice to know how much . It is nice to know how much

teachers teachers

think think their students learned, but their students learned, but you can’t judge you can’t judge effectiveness based on opinioneffectiveness based on opinion. Reading is a real . Reading is a real

thing. thing. It It cancan be measured be measured objectivelyobjectively---for example, how many ---for example, how many words students read correctly per minute. words students read correctly per minute.


So, if researchers So, if researchers do NOT use do NOT use objective measuresobjective measures of (for example) of (for example) reading and math achievement (e.g., reading and math achievement (e.g., counting correct words students read counting correct words students read or problems students solve per or problems students solve per minute), but instead use the minute), but instead use the subjective opinionssubjective opinions of teachers of teachers (“Students improved a great deal.”) (“Students improved a great deal.”) then then you cannot trust claims about you cannot trust claims about effectivenesseffectiveness----because the ----because the researchers did not give researchers did not give evidence evidence of of effectiveness. They only gave effectiveness. They only gave evidence of how teachers evidence of how teachers see thingssee things. . But But that that was not the question.was not the question.


4. 4. Were Were instruments instruments and and measurement measurement methodsmethods tested tested for for validity validity (they measure (they measure what they are supposed to measure) and what they are supposed to measure) and reliability reliability (different persons measuring the (different persons measuring the same thing would come up with about the same thing would come up with about the same scores)? same scores)? Validity.Validity. Do you think a good measure of Do you think a good measure of reading proficiencyreading proficiency is whether students can is whether students can name the parts of a book and can turn the name the parts of a book and can turn the pages? pages?

If your school principal is selecting a new If your school principal is selecting a new reading program, is that the information you reading program, is that the information you want the principal to use?want the principal to use?““Students taught with Weird Phonics learned Students taught with Weird Phonics learned to to turn the pages. So, I’ll use that turn the pages. So, I’ll use that program with 200program with 200 students in my school.”students in my school.”

I don’t think so.I don’t think so.


ExperimentExperiment

Turning pages is not what “READING” Turning pages is not what “READING” means.means.

You want to know if a program teaches You want to know if a program teaches students to: (1) say the sounds that go students to: (1) say the sounds that go with the letters; (2) accurately and with the letters; (2) accurately and quickly read single words; (3) quickly read single words; (3) accurately and quickly read passages; accurately and quickly read passages; (4) learn many new vocabulary words (4) learn many new vocabulary words each week; (5) answer both simple and each week; (5) answer both simple and hard comprehension questions about hard comprehension questions about what they read.what they read.


So, if researchers use instruments and So, if researchers use instruments and measures that measures that do not measuredo not measure what they what they OUGHTOUGHT to measure, then to measure, then the findings have the findings have nothing to do with the research question.nothing to do with the research question.

• The question is whether a curriculum The question is whether a curriculum teaches students to teaches students to readread. But the findings . But the findings have to do with have to do with turning pagesturning pages. . Invalid Invalid measure.measure.

• The question is whether a certain teaching The question is whether a certain teaching procedure increases math procedure increases math achievementachievement. . But the findings have to do with how much But the findings have to do with how much students students enjoyed enjoyed being taught with the being taught with the method. method. Invalid measure.Invalid measure.

• The question is whether peer tutoring is The question is whether peer tutoring is effective at helping students to effective at helping students to spell betterspell better. . But the findings have to do with changes in But the findings have to do with changes in self-esteemself-esteem. . Invalid measure.Invalid measure.


5.5. Is the sample size, composition (kinds of Is the sample size, composition (kinds of participants), and selection/allocation of participants), and selection/allocation of participants to groups appropriateparticipants to groups appropriate for the for the type of study and for how the findings might type of study and for how the findings might be used? be used? Poor.Poor. The researcher claims that a teaching method The researcher claims that a teaching method is effective. However, the method was used in is effective. However, the method was used in only only oneone class, or in one school. class, or in one school. There is no There is no way to tell if it would be effective way to tell if it would be effective anywhere anywhere elseelse..Poor.Poor. The researcher claims that a method is The researcher claims that a method is notnot effective. However, it was used in only effective. However, it was used in only one one class, or in one school. There is no way to tell class, or in one school. There is no way to tell if it would have been effective if it would have been effective elsewhereelsewhere..

[More][More]


Poor.Poor. The researcher claims that an instructional The researcher claims that an instructional procedure or curriculum is more effective procedure or curriculum is more effective than another. However, the than another. However, the comparison comparison groupsgroups (one class used one curriculum; the (one class used one curriculum; the other class used a different curriculum) other class used a different curriculum) were not created by were not created by random assignmentrandom assignment of of children to the groups, or by children to the groups, or by matchingmatching the the groups to make sure they had, for example, groups to make sure they had, for example, the same percentage of boys and girls. the same percentage of boys and girls.

Therefore, Therefore, the groups may have been the groups may have been different in different in other waysother ways besides the besides the curriculumcurriculum; e.g., more children in one ; e.g., more children in one group got reading instruction at home, and group got reading instruction at home, and this (not the curriculum) caused differences this (not the curriculum) caused differences in achievement. [More]in achievement. [More]


ExperimentExperiment

So, if research involves comparing (for example) So, if research involves comparing (for example)

achievement from one group/sample (class, achievement from one group/sample (class,

school) to another, but the researchers did NOT school) to another, but the researchers did NOT

make sure (through make sure (through random assignmentrandom assignment of of

students to the samples, or through students to the samples, or through matchingmatching) ) that that

the groups have the the groups have the same characteristicssame characteristics, then , then you you

can’t trust that differences in achievement are can’t trust that differences in achievement are

caused by the curriculum or teaching method.caused by the curriculum or teaching method.


Here is Here is one kindone kind of experimental research that you of experimental research that you DO want to see.DO want to see. Level-1 research.Level-1 research.The researcher conducts a The researcher conducts a pilot studypilot study to see if a method to see if a method seems to work well enough that it ought to be given a seems to work well enough that it ought to be given a

more more valid testvalid test. The researcher uses a . The researcher uses a small samplesmall sample (one class (one class

or or school). It is not a school). It is not a representativerepresentative sample of the school sample of the school population, but that is alright. [Do you want to use a population, but that is alright. [Do you want to use a

method method that may not work on a lot of kids?] The researcher finds that may not work on a lot of kids?] The researcher finds

that that the new method is followed by significant student the new method is followed by significant student achievement. But since the sample is small and not achievement. But since the sample is small and not representative of the larger school population, and since representative of the larger school population, and since the research was done only once, the researcher cautions the research was done only once, the researcher cautions

that that the findings are very tentative and cannot be generalized the findings are very tentative and cannot be generalized anywhere else. Further research is needed. anywhere else. Further research is needed.

This is called honesty.This is called honesty.


Here is another kind of experimental research that you DO Here is another kind of experimental research that you DO want to see. want to see. Level-2 research.Level-2 research.The researcher The researcher repeats researchrepeats research on the same method in on the same method in different schools. Each new study is called a different schools. Each new study is called a replicationreplication. .

The The sample of schools sample of schools is representativeis representative of the population of of the population of schools---small/large, wealthy/poor, rural/urban. schools---small/large, wealthy/poor, rural/urban. The researcher claims that data from most of the studies The researcher claims that data from most of the studies show that the method is effective in most of the schools, show that the method is effective in most of the schools,

but but the researcher cannot figure out which factors hinder its the researcher cannot figure out which factors hinder its effectiveness or what additional factors may be needed to effectiveness or what additional factors may be needed to make it work better. The researcher advises caution in make it work better. The researcher advises caution in

using using the method. the method. [This is honest and morally responsible.] [This is honest and morally responsible.]


Here is yet another kind of experimental research Here is yet another kind of experimental research that you DO want to see.that you DO want to see. Level-3 research.Level-3 research. The researcher conducts a study in The researcher conducts a study in several school several school districts at oncedistricts at once. The sample of districts is . The sample of districts is representative representative of the larger population of schools and of the larger population of schools and school districts. Three new reading curricula are school districts. Three new reading curricula are tested against each other. The three curricula are tested against each other. The three curricula are randomly assignedrandomly assigned to the districts. to the districts. Objective and Objective and validated measuresvalidated measures of essential reading skills are of essential reading skills are made before the new curricula are introduced (pre-made before the new curricula are introduced (pre-test), every two weeks during the use of the curricula test), every two weeks during the use of the curricula (progress assessment). and at the end of the year (progress assessment). and at the end of the year (post test, or outcome assessment. [More](post test, or outcome assessment. [More]


At the end of the year, researchers At the end of the year, researchers examine the amount of change in examine the amount of change in students’ skills students’ skills from pre-test to post-testfrom pre-test to post-test. . They find that curriculum A produced They find that curriculum A produced significantly more achievement (and significantly more achievement (and objectively high achievement) in all of the objectively high achievement) in all of the reading skills (compared to achievement reading skills (compared to achievement produced by curricula B and C) produced by curricula B and C) in 8 out of in 8 out of the 10 schoolsthe 10 schools in which it was used. The in which it was used. The researchers therefore claim that researchers therefore claim that curriculum A is more likely than the other curriculum A is more likely than the other curricula to produce high reading curricula to produce high reading achievement, and that it can be used with achievement, and that it can be used with at least modest confidenceat least modest confidence. However, they . However, they say that more research is needed to say that more research is needed to determine the school conditions in which determine the school conditions in which curriculum A works best.curriculum A works best.

Would you use curriculum A?Would you use curriculum A?

evaluating research claims. evaluating tv ad claims an ad on tv says, “use cocoa creme. you’ll...

Documents

effects of megabiotic

medical claimsthe

medical claimswill

medical claimshow

medical claimsprobably

medical claimswhat

tv ad claimsanother

doctor convincing