using focus groups for evaluating learner experiences by tunde varga-atkins and luciane vieira...

Using focus groups for evaluating learner experiences

Tünde Varga-Atkins - eLearning Unit, CLLLuciane Vieira de Mello Rigden – School of Life Sciences

University of Liverpool6th June 2013

Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

Outline1. Why focus group?2. What is a focus group? 3. Focus groups in context of other methods4. Viewpoints: focus groups as a – Commissioner (stakeholder)– Observer– Facilitators

5. Focus group analysis and reporting6. Alternatives / enhancements:– Nominal Group Technique– Visual and any other methodsCreative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

Your experience

• Have you had experiences of focus group, either as:– Commissioner?– Facilitator?– Observer?– Participant?

What is your research question?Appropriate/Not to explore in a

focus group?


1. Why focus groups?


Rationale for focus groups

“When people gather to talk about something, their contributions and understandings will be

enriched by the group dynamic.” (Cousins 2009)

“To display and discuss differences in a group. ...” (Lewis 2003)

Focus groups are for sharing and comparing … and remembering


2. What is a focus group?


Focus group process

Neutral facilitatorSize: 6-8 (4-12) participants

Incentive

Sample selection!

Ethics: informed consent and reporting

Audio recorder & other resources or stimulus material

1-2 hours

an open group discussion with a focus


What is a focus group?

Study skills

International student experienceFocus:

Question(s):

English language Peer

support

Can you tell me about the support that you have received on campus?

A focused but open discussion with a group of 6-8 participants, moderated by a neutral facilitator usually lasting 1-2 hours. The facilitator usually has a topic (or question) guide which may or may not be covered in sequence.

Introduction Closure


The evaluation/research cycle* 1. Set out purpose

2. Agree what questions

to ask

3. Agree on method

(focus group)

5. Compiling draft report

6. Draft report sent to students for confirmation

10. Action & feedback to

students

7. Production of final report

4. Conduct of focus groups

8. Circulation of report to staff

9. Presentation and discussion

of report


*of not just focus groups but any participant consultation!

3. Focus groups in context of other methods


Some possible combinations

Exploring depth

survey

survey

focus group(s)

focus group(s)

Generate ideas

focus group(s)

a

b

c


Methods: from individual to group

Surveys

Focus groups

Interviews

Researcher present

individual

group

Researcher not present


Methods: from individual to group

Surveys

Focus groups

Interviews

Researcher present

individual

group

Researcher not present

How many hours do you spend studying?*

What makes you study (more)?

Can you tell me a bit more about what you do when you study?

Idea development

aided by others?

Participant motivation

Bias by others?

Shared experience?

* BUT: in a focus group setting, participants can have a discussion about what constitutes study, which may make them adjust their own understanding of what study is.Creative Commons: Attribution-NonCommercial-Sharealike By Tunde Varga-Atkins, University of Liverpool.

Surveys Focus groups Interviews

Bias? Useful when you don’t want others to bias your participant

Potential of ‘group bias’ – participants say what they think you or other participants want to hear

Useful when you don’t want others to bias your participant

Shared experience? Participant relies on own experience – may be more difficult to recall.

Useful to talk about shared experiences. ‘You bounce off ideas from one another’.

Useful when you want to explore individual experiences.

Idea development aided by others?

Relies on participant able to formulate their ideas without prompts.

FGs are useful when the development of ideas and understanding is aided by other group members.

FGs are useful when the development of ideas and understanding is aided by other group members.

Motivation/Enjoyment ? Non-response and quality of response as a problem.

Participants generally enjoy the process.* They learn something new.

Participants generally enjoy the process. They learn something new.

BUT: feasibility and practicality may be the largest factors(of getting volunteers, organising them together etc.)


Sample questions from a UoL curriculum evaluation focus group

• 1.What aspects of the X best facilitated your learning?

• 2. What aspects of the X inhibited your learning ?• 3. What aspects of the X caused the most

stress/anxiety and did this stress/anxiety worsen or lessen as the X progressed?

• 4. Optional specific questions.• 5. Can you suggest one (two) different ways in

which the X could be improved?


4. ViewpointsFocus groups as a:

commissioner

observerfacilitator


As a commissioner of a focus group• Experiences of Dr Mello (Lu):

– Asking the right questions• After a questionnaire: realisation of non-properly covered aspects guided question

formulation• Questions were discussed with the facilitators (but I would do it more extensively

next time….)

– Coverage of questions• Some really short answers from students• Group dynamic

– What is important? • Have clear what you want• Take the first one as an experience

– Expectations • If you have an open mind, you will surprise yourself!

As an observer of a focus group

• Experiences of Dr Mello (Lu):– What is going on?

• Got to know the questions just before meeting the students

– Benefits• Better appreciation & understanding of the whole process• Confidence to plan another focus group• How I learned from the students!!!

– Challenges• Hard to just observe! • Have a good chat with the facilitator in advance

As a facilitator of a focus group

• Design• Introduction, scene setting• Questioning techniques• Managing group dynamics• Managing schedule / topics / coverage

Slides supporting focus group process (facilitation)

The schedule of a Focus Group

1. Introduction = setting scene and establishing trust:– purpose – ground rules (conformity / convergence)– ethics – reporting

1. Introduction

2. The ‘Focus’ = questions

3. Closure


2. The ‘Focus’ = aka questionsLow vs high moderation

2 questions 6 questions: exploring international student experience on campus

6 questions

(Cousins 2009, p63)

Note: Participants do not normally see the questions – the discussion is good if it seems to flow naturally.

These will not necessarily be covered in order.

Anatomy of a good focus group question

with ready probes

(Cousins 2009, p63)

Preparation and good question (and prompts, probes) design is very important!

English language

Peer support

Probing for range

Probing for depth

Study skills

Can you say something

about further support..?

Open question

5. Analysis and reporting


Analysis

TranscriptReport

Analytical process

Various decisions and challenges!• Emphasis – key points• Focus• Agreement• Majority vs single voices

Research question dependent

6. Alternative methods


It doesn’t have to be just words!Tasks e.g. projection techniques

Source: http://bit.ly/JZRz1T

Visual techniquesStimulus

Source http://bit.ly/Mc0kSu

Bringing as many stimulus material related to the topic as possible to help participants recall e.g. bring assignment samples for a discussion on feedback.

Asking participants to take photos, draw diagrams or drawings to elicit their experiences e.g. ask them to draw a timeline of their time at university

Asking participants to undertake an activity, e.g. using projective techniques or write down their top five priority in the given topic etc.


http://lb-qr-english.blogspot.co.uk/2010/06/qualitative-projective-techniques.html

http://bit.ly/JZRz1T

http://bit.ly/JZRz1T

http://bit.ly/Mc0kSu

The Nominal Group Technique • Good decision-making technique with no need for

transcription!• Brings in a quantitative element through the ranking stages.

• More useful for evaluation and less useful for ‘full’ research.– See references for more details.


The Nominal Group Technique: an alternative focus group method

Surveys

Focus groups

Interviews

individual

group

Nominal Group Technique

The Nominal Group Technique is a useful technique that is structured in a way that although it is a group session, it does focus on individual experiences and opinions rather than group consensus. Participants at various points in the session are asked to give their individual opinions which are ranked at the end.

Summary• Focus groups are useful when:

– exploring a shared experience.– participants (and the research!) benefits from the development of

understanding and ideas in a group environment.– group bias or conformity is not likely/important.

• Focus groups:– need to be facilitated by a neutral facilitator.– produce quality of the output (if focused and run well).– can be enriched by various techniques (visuals, stimulus) or tasks.

• Analysis involves various decisions on what/how to report.• A useful alternative is the Nominal Group Technique.

– Just writing up a combined ‘Nominal Focus Group’ as a useful method.


References

• Focus groups– Cousin, G. (2009). Focus Group Research. IN:

Researching learning in higher education : an introduction to contemporary methods and approaches. London: Routledge, pp.51-69.

– Ritchie, J., & Lewis, J. (2003). Qualitative research practice : a guide for social science students and researchers. London; Thousand Oaks, Calif.: Sage Publications.


References

• Nominal Group Technique– Practical guide: Varga-Atkins, T., with contributions from Bunyan, N;

McIsaac, J; Fewtrell J. (2011) The Nominal Group Technique: a practical guide for facilitators. Written for the ELESIG Small Grants Scheme. Liverpool: University of Liverpool. October. Version 1.0. Available at http://slidesha.re/s5KPUr

– Project report: Varga-Atkins, T. with contributions from Bunyan, N; McIsaac, J and Fewtrell, R. (2011) Using the nominal group technique with clickers to research student experiences of e-learning. Project Report written for the ELESIG Small Grants Scheme. Liverpool: University of Liverpool. Available from slideshare.net at http://slidesha.re/sc8gwT


http://slidesha.re/s5KPUr

http://t.co/0r5BDRbW

using focus groups for evaluating learner experiences by tunde varga-atkins and luciane vieira...

Education