Asking the right questions:

Collecting and validating data –

refining, rephrasing, and

refocusing

Dr. Corinne Haigh, The School of Education, Bishop’s

University, chaigh@ubishops.ca

PROCEDE 2014: People, Passion, Proof – The Data

Difference

PROCEDE 2014

1

Session Outline

What makes a good research question?

Where do I begin?

What is the danger of a single story?

How can I be sure that the measurements I take match

the question I am trying to answer?

How do we define what we wish to measure?

What are reliability and validity?

What are some ethical issues to consider?

PROCEDE 2014

2

Pollev.com/chaighbu

From any browser

(code or keyword) <your response>

From a text message

Participating with Poll Everywhere: How

to vote via the web or text messaging

22333

Participating with Poll Everywhere

How to vote via the web

Participating with Poll Everywhere

How to vote via texting

Reflection

What types of questions was I able to ask using the

polling software?

What type of information was I able to gather?

How could I analyze this information to answer a

specific question?

How might I present it to others?

Take a moment to reflect on these questions

individually.

PROCEDE 2014

10

Reflection

What types of questions was I able to ask using the

polling software?

What type of information was I able to gather?

How could I analyze this information to answer a

specific question?

How might I present it to others?

Take a moment to reflect on these questions

individually.

PROCEDE 2014

13

Now, think of a recent “problem” you

have encountered in your

professional life…

Briefly describe it.

How did/might you approach it?

What did/do you hope to learn as a result?

Take a few minutes to reflect on these questions

individually, and then compare notes with a

colleague

PROCEDE 2014

14

The danger of a single story

Asking the right questions involves considering

multiple perspectives

We will now watch one video that illustrates the

danger of being exposed to only a single story

And I’ve included a link to another for you to

watch at another time…

PROCEDE 2014

15

Chimamanda Adichie

Novelist Chimamanda Adichie tells the story of

how she found her authentic cultural voice --

and warns that if we hear only a single story

about another person or country, we risk a

critical misunderstanding.

The Danger of a Single Story

PROCEDE 2014

16

Ben Goldacre

Doctor and epidemiologist Ben Goldacre shows

us, at high speed, the ways evidence can be

distorted, from the blindingly obvious nutrition

claims to the very subtle tricks of the

pharmaceutical industry.

Battling Bad Science

PROCEDE 2014

17

Checkpoint

Take 2-3 minutes and write about an

educational example of the danger of a single

story

Try to write for the full time

You will then discuss your writing with a partner

(or in a trio) and then with the whole group

PROCEDE 2014

18

Asking a question often starts

from necessity

How do I know when…?

Think up some endings to my sentence starter.

PROCEDE 2014

19

The Assessment for Learning

(AL) Cycle 1. What is

known

2. What might be happening

and why

3. What you need to find

out

4. What you think is

happening

5. What you say

6. What you need to do

7. Did what you chose to

do work?

Gathering

referral

information

Generating

hypotheses

Checking

your

hypotheses

by collecting

information

Arriving at

tentative

judgement

Reporting

findings

Planning

instruction

Evaluating

success

Further

information

needed?

20 PROCEDE 2014

Refining Questions

Initial questions may be vague, but vague

questions can never be answered

Refine questions by:

Consulting past research in the area

Speaking with colleagues

Applying theory

Replicating someone else’s study

Clarifying contradictory findings

PROCEDE 2014

21

What is a variable?

A variable is something that can be changed,

such as a characteristic or value

We are generally looking at whether change in

one variable results in change in another

variable

Can you generate some examples of variables

from your practice? What kind of relationships

between these variables might be interesting to

investigate?

PROCEDE 2014

22

Learning environment

Academic

Physical

Emotional

Social

Initial Concern

Gaps in learning;

opportunity to learn;

history of

successes/failures;

teachers; past and

present programs

Vision; hearing; health

history

Reading profiles;

academic profile;

experiences; cognitive

development; aptitude

(IQ); Language profile

Family; self concept;

motivation; security

Culture; friends; maturity

23 PROCEDE 2014

Operationally defining a variable

Specifies exactly how to measure and/or

manipulate the variables in a study

Go back to one of the examples you

generated for the previous question – how

would you define it?

Remember, it must be measureable (think

about setting SMART goals for IEPs)

PROCEDE 2014

24

Types of variables

Independent vs. dependent

Independent: precedes, influences, or predicts the

dependent variable (e.g., teaching method)

Dependent: affected or predicted by the

independent variable (e.g., student engagement)

Extraneous and confounding

Extraneous: source of error affecting the results (e.g., time of day)

Confounding: varies systematically with the independent variable (e.g., the teacher)

PROCEDE 2014

25

26

How can we make sense of a

complex process? We make

inferences…

1.

2.

3.

PROCEDE 2014

27

BALLOON

- Recognize the picture as balloons

- Select the appropriate word

- Produce the appropriate articulation

What mental activity needs

to occur to identify the

picture and say what it is?

How is making an inference about learning

different than blowing up a balloon?

- we cant see the “mental activity”

-Therefore, we must make inferences

about the mental activity from

observable behaviours.

PROCEDE 2014

Qualitative research

A means for exploring and understanding the

meaning that individuals or groups ascribe to a

social or human problem

Emphasizes natural settings, understanding,

verbal narratives, and flexible designs

28

PROCEDE 2014

Qualitative techniques and data

Data: Verbal descriptions, field notes,

observations, documents, photographs,

people’s own words, narrative

Techniques: observation, participant

observation, open-ended interviewing, review

of documents and artifacts

PROCEDE 2014

29

Quantitative research

A means for testing objective theories by

examining the relationship among variables

Emphasizes numbers, measurements, deductive

logic, control, and experiments

30

PROCEDE 2014

Quantitative techniques and

data

Data: quantiles, counts, numbers, statistics,

measures/instruments

Techniques: experiments, quasi-experiments,

structured observations, structured interviews,

surveys

PROCEDE 2014

31

How do I know which to use?

It is critical to match the reason for the research with the

appropriate method

Determined by the purpose and the research question

Using each method will be a challenge

Either can be used well, or poorly

Depth of understanding is important, regardless of the

design

Leads to greater impact

PROCEDE 2014

32

Checkpoint

Think about some of the potential research

questions you generated earlier – what sort of

approach would be best suited to answering

them?

Consider the data you’d want to collect (or

existing data you’d like to access) and the

techniques you could use to do so

PROCEDE 2014

33

Reliability

Test-retest reliability:

Agreement between scores on the same test,

given to the same people, at different times

Inter-rater reliability

Agreement among different raters or observers

Inter-item reliability

Are all of the items on the test equally good at

assessing the same underlying ability? 34

PROCEDE 2014

Validity

Content validity

The range and relevance of the items

Can be very difficult to measure for some more

complex aspects of learning

Construct validity

Concerns the underlying theoretical construct: is the

test measuring the target skill or behaviour

Test this by comparing scores in two different measures

– should be highly related if they’re measuring the

same construct

35

PROCEDE 2014

Reliability and validity

36

PROCEDE 2014

Some broad ethical questions to

consider:

Does answering this question benefit the individuals being studied? Will the answer be meaningful for others?

Are participants being marginalized or disempowered?

Are participants being deceived about the intent of the study?

Are participants at more than minimal risk?

Minimal risk: “Research in which the probability and magnitude of possible harms implied by participation in the research is no greater than those encountered by participants in those aspects of their everyday life that relate to the research.” (TCPS 2, Chapter 2)

PROCEDE 2014

37

Some broad ethical questions to

consider:

Did all participants benefit from the “treatment”?

Is there a possibility that participants will disclose harmful

and/or intimate information? What will be done with this

information?

How will you protect the anonymity of participants?

PROCEDE 2014

38

Checkpoint

Describe an ethical question or concern you’ve had

when trying to collect data to answer a specific

question (or speculate about a potential ethical

issue that may arise in future).

How did/could you address it?

First reflect on this individually and then have a

discussion at your table.

PROCEDE 2014

39

Some final points for reflection…

Sometimes coming up with a good research problem or

question is the hardest part

The approach you take to answering your question will

depend on:

The problem you’re trying to address

Your worldview or assumptions about research

The specific inquiry strategies you wish to use

Your personal experience

There are multiple ways to approach data collection and

analysis and no one is better than the other as long as you aware of their strengths and limitations

PROCEDE 2014

40

Thank you!

chaigh@ubishops.ca

PROCEDE 2014

41