Qualitative Assessment: A Hands-

On Introduction

Miriam Segura-TottenBSP Research Residency

Institute July 23, 2015

Qualitative data is information which does not present itself in numerical form and is descriptive, appearing mostly in conversational or narrative form.

Words, phrases, text…

Definition

What are some sources of qualitative data?

Notebooks Open-ended questions Papers Journal entries On-line discussions, blogs Email Twitter/ ‘tweets’ Notes from observations Responses from interviews and focus groups

What are some sources of qualitative data?

Qualitative analysis is the “interplay between researchers and

data.”

Researcher and analysis are “inextricably linked.”

Qualitative Data Analysis

Inductive process◦ Grounded Theory (“Open coding”)

Unsure of what you’re looking for, what you’ll find No assumptions No literature review at the beginning

Deductive process◦ Theory driven

Know the categories or themes using rubric, taxonomy Looking for confirming and disconfirming evidence Question and analysis informed by the literature,

“theory”

Qualitative Data Analysis

Coding process: ◦ Conceptualizing, reducing, elaborating and

relating text– i.e., words, phrases, sentences, paragraphs.

Building themes:◦ Codes are categorized thematically to describe or

explain phenomenon.

Definitions: Coding and Themes

What have students learned from a particular activity?

What is the mental process of a student engaged in a particular intervention?

What are the key transitions that occur as undergraduates acquire metacognitive-regulation skills? (Stanton et al., 2015)

Example Research Questions

Read through the student reflection paper and highlight words, parts of sentences, and/or whole sentences with some “code” attached and identified to those sections.

Let’s Code #1

What did you highlight?Why?

Read through this reflection paper and code based on this question:

What does the term "scientific research" mean?

Let’s Code #2

What did you highlight?Why?

Our prompt for students’ reflective papers:

What does the term "scientific research" mean to you? What do you

think doing scientific research entails?

Terms that we agreed upon when coding:

Thematic analysis procedure:

1. Several individuals code material (triangulation).

2. Meet to decide on common set of terms.

3. Code again using common set.

4. Can go through steps 1-3 several times.

5. Analyze codes: qualitatively and/or quantitatively.

Let’s look at the Stanton paper:

What is the research question of this study?

What are the key transitions that occur as undergraduates acquire metacognitive-regulation skills? (Stanton et al., 2015)

Why is qualitative analysis suited for looking at this research question?

Let’s look at the Stanton paper:Background

Self-regulated learning (metacognition imp):• understand what the task involves• ID strengths and weaknesses• Create plan for completing task (planning)• Monitor how well plan is working (monitoring)• Evaluate and adjust plan as needed

(evaluating)

Let’s look at the Stanton paper:

What was the experimental approach of this study?

Self-evaluation assignments after two tests

E1-SEE2-FT

Prompts

The questions attempt to get at whether students are planning, monitoring and evaluating.

Code student answers.

Let’s look at the Stanton paper:How did the authors approach the coding of student answers?

With “metacognitive regulation in mind”.Theory-driven, not open coding.

Let’s look at the Stanton paper:

How did the authors approach the coding of student answers?

E1-SE: They developed a coding system (“sufficient/provides evidence” or “insufficient/provides no evidence”).

Applied this to planning, monitoring, evaluating (self-regulated learning/metacognition).

Let’s look at the Stanton paper:

How did the authors approach the coding of student answers?

E2-FT: Coded for evidence that students followed study plan developed. Coding system (“yes/followed plan” or “no/did not follow plan”).

Applied this to planning, monitoring, evaluating (self-regulated learning/metacognition).

Let’s look at the Stanton paper:

Example of results

Let’s look at the Stanton paper:

Example of results

Let’s look at the Stanton paper:

Analysis of student answers led authors to propose a continuum of metacognitive regulation

Let’s look at the Stanton paper:

What does this mean for your study?• Don’t be scared! Most projects do not require this type

of depth of qualitative analysis.• Many studies can benefit from mixed methods (qual &

quant). • If you want to know student opinions, analyze long

answer questions in exams, or essays, you may want to consider qualitative analysis.

Use mixed methods, multiple sources. Triangulate your data whenever possible. Ask others to review your design

methodology, observations, data, analysis, and interpretations (e.g., inter-rater reliability).

Note limitations of your study whenever possible.

Ensuring “validity” and “reliability” in your research

Read the literature: What is accepted in your field?

Read books or articles on qualitative research

Collaborate with a qualitative researcher Or, just pick their brain over coffee

How to get started in qualitative research

• Designing and Conducting Mixed Methods Research, Creswell, J.W., and Plano Clark, V.L., 2006, Sage Publications.

• Discipline-Based Education Research: A Scientist’s Guide, Slater, S.J., Slater, T.F., and Bailey, J.M., 2010, WH Freeman.

• Educational Researchers: Living with a Lesser Form of Knowledge, Labaree, D.L., 1998, Educational Researcher, 27(8), 4-12.

• Processing Field Notes: Coding and Memoing. In Writing Ethnographic Fieldnotes, 1995, Emerson, R.M., Fretz, R.I., and Shaw, L.L. (pp. 142-168). Chicago: The University of Chicago Press.

References