An Overview of Virginia 4-H An Overview of Virginia 4-H Camping Program Evaluation: Camping Program Evaluation:

Process and OutcomesProcess and Outcomes

2002 National 4-H Camping Institute, Ashley, OH

February 13-17, 2002

Barry A. Garst, Extension Specialist4-H Staff Development and Camping

Becky Gilles, Program DirectorSmith Mountain Lake 4-H Educational Center

Bryan Branch, Program DirectorHoliday Lake 4-H Educational Center

Ruben Brown, Urban 4-H Program AssistantChesapeake Cooperative Extension

Increasing importance of demonstrating the value and benefits of youth programs

Increasing emphasis on the development of rigorous, systematic evaluation procedures and “good” instruments

Importance of sharing information and approaches to evaluating the impacts/outcomes of youth camping

Why this session?

What are your expectations?

What information are you hoping to gain through this session?

Why is this session important to you?

Overview Overview of Virginia 4-H Camping 4-H camp evaluation prior to 2001 2001 data sources Results and analysis Limitations What did we learn? Where do we go from here?

ASK QUESTIONS AT ANY TIME!ASK QUESTIONS AT ANY TIME!

Virginia 4-H Camping Involved in 4-H camping since 1917 Recognized as one of Virginia’s primary

delivery modes for 4-H educational programs

Conducted at the unit-level (107) and 4-H Center (6) level

Approximately 30,000 served annually From 1995-2001, approximately 4,000

more youth served

Virginia’s 4-H Educational Centers

Virginia’s six 4-H Educational Centers are owned and operated by their respective Boards of Directors

Each 4-H Center has a Memorandum of Understanding with Virginia Tech

Virginia Cooperative Extension provides funding for (1.5 FTE) positions and operations

Virginia Tech provides insurance for facilities Each 4-H Center is accredited through the

American Camping Association

4-H Educational CentersOperation and Management

4-H Camp Evaluation Prior to 2001

Parent/guardian study (1995 and 1998) Some 4-H Centers were evaluating post-

camp satisfaction (ratings, etc.), Some 4-H Centers were not conducting

regular evaluation No state-wide standardized instruments No state-wide standardized questions

Evaluation Needs Leading into Summer 2001

Needed to be able to collect similar data across 4-H Centers Assess overall program quality Data to assist resource development Baseline data for future comparison Comparing 4-H Centers was not a goal

Data CollectionData Collectionin Summer 2001in Summer 2001

Data Sources Virginia 4-H Camping Participation

Form Standardized 4-H Center Survey Parent/Guardian Survey

Virginia 4-H Camping Participation Form Distributed to 113 units in Virginia 4-H

107 counties/cities Six (6) 4-H Centers

Questions WHO is participating? WHAT programs are being offered? WHEN are programs taking place? WHERE are programs being conducted?

2001 Virginia 4-H CampingParticipation Form*

(NOTE: Complete a separate form for each unit and each program**)

H UNIT: NAME OF 4-H CAMPING EVENT:

ATE OF 4-H CAMPING EVENT: PERSON WITH 4-H REPONSIBILITIES :

AMP TYPE: [Check () all that apply] ___Day Camp ___Residential Camp (2-3 days) ___Residential Camp (4+ days)

___4-H EFNEP/SCENEP ___Special Interest Camp

STRUCTIONS: In the table below, please identify the number of participants in your 4-H camp event by placing a number in the correct box for eachrticipant group (youth, teen counselors, adults) and each ethnic group. If ethnic group information is missing or cannot be identified because therticipant chose not to identify his/her ethnic group, then indicate that number in the “Missing/Unidentified” column.

WHITE BLACK HISPANIC ASIAN AMERICANINDIAN

MULTI-CULTURAL

OTHER MISSING/UNIDENTIFIED

YOUTHAges 9-13)

MalesFemales

TotalEEN COUNSELORS

Ages 14-18)Males

FemalesTotal

ADULTSAges 19 and older)

MalesFemales

Total-H CENTER

PROGRAM STAFFMales

FemalesTotal

GRAND TOTALS

Return to Barry A. Garst by September 15, 2001Include all camps conducted/sponsored by 4-H: Junior 4-H Camps, special interest camps, Cloverbud camps, teen weekends, etc.)

Standardized 4-H Center Survey

Population sample Every Junior 4-H Camp participant Post-camp survey (last day of camp) 11,254 surveys collected (82% response)

Questions Are youth, teen/adult volunteers, and Agents

receiving a positive experience during Junior 4-H Camp?

Are youth campers (ages 9-13) receiving life skill benefits as a result of participation in Junior 4-H Camp?

Parent/Guardian Survey Random sample of youth selected Survey mailed to parents/guardians

Post camp survey (2 weeks after camp) Survey/Reminder/2nd Survey 363 surveys returned (46%)

Questions What are parents’ perceptions of Junior 4-H Camp? Are youth campers (ages 9-13) receiving (short-

term) life skill benefits as a result of participation in Junior 4-H Camp?

Virginia 4-H Virginia 4-H Participation FormParticipation Form

2001 Virginia 4-H CampingParticipation Form*

(NOTE: Complete a separate form for each unit and each program**)

H UNIT: NAME OF 4-H CAMPING EVENT:

ATE OF 4-H CAMPING EVENT: PERSON WITH 4-H REPONSIBILITIES :

AMP TYPE: [Check () all that apply] ___Day Camp ___Residential Camp (2-3 days) ___Residential Camp (4+ days)

___4-H EFNEP/SCENEP ___Special Interest Camp

STRUCTIONS: In the table below, please identify the number of participants in your 4-H camp event by placing a number in the correct box for eachrticipant group (youth, teen counselors, adults) and each ethnic group. If ethnic group information is missing or cannot be identified because therticipant chose not to identify his/her ethnic group, then indicate that number in the “Missing/Unidentified” column.

WHITE BLACK HISPANIC ASIAN AMERICANINDIAN

MULTI-CULTURAL

OTHER MISSING/UNIDENTIFIED

YOUTHAges 9-13)

MalesFemales

TotalEEN COUNSELORS

Ages 14-18)Males

FemalesTotal

ADULTSAges 19 and older)

MalesFemales

Total-H CENTER

PROGRAM STAFFMales

FemalesTotal

GRAND TOTALS

Return to Barry A. Garst by September 15, 2001Include all camps conducted/sponsored by 4-H: Junior 4-H Camps, special interest camps, Cloverbud camps, teen weekends, etc.)

30,175 servedThrough 4-H Camping/Environmental Education

7,359

22,816

4-H Camp Participation4-H Environmental Education Participation

4-H Camping ParticipationYouth, Teen Counselors, Adult Volunteers*

650

16,939

3,1612,066

0

4,000

8,000

12,000

16,000

20,000

Youth (ages 5-8)(Cloverbud)

Youth (ages 9-13) Teens (ages 14-18) Adult Volunteers(ages 19+)

* Based on a total of 22,816 participants

Percentage of 4-H Camping Unit-based vs. 4-H Center based

4-H Center Based49.7%

Unit Based50.3%

150 programs*

148 programs*

* Based on a total of 298 camping programs conducted

Holiday Lake19%

Jamestown13%

Airfield12%

Southwest12%

Smith Mountain Lake27%

Northern Virginia17%

* Based on a total of 22,816 participants

4-H Camp Participation By 4-H Center*

1% <1%4% 3%

23%

6% <1%2%1%

20%

39%

0%

20%

40%

60%

80%

100%

Janu

ary

Februa

ry

March

April

MayJu

ne July

Augus

t

Septem

ber

Octobe

r

Novem

ber

Decem

ber

* Based on a total of 298 camping programs conducted

4-H Camp Participation By Month*

Cloverbud (ages 5-8) Junior 4-H Camp (ages 9-13) Teen Camps (ages 14-18) Day Camps (ages 9-18) Special Interest Camps (ages 9-18)

Forestry Camp, Shooting Education, Outdoor Adventure, Science and Technology, Family Camp, Creative Arts, Nutrition Education

Types of 4-H Camps Offered

37% 33%

13%

7% 5% 4% <1%

0%

20%

40%

60%

80%

100%Residential Junior 4-H Camps

Day Camps

Special Interest Camps

Teen Camps

Cloverbud Camps

4-H EFNEP/SCNEP Camps

Other residential camps

4-H Camping ProgramsPercentage of 4-H Camping Programs Conducted

* Based on a total of 298 camping programs conducted

Camp ThemeNumber ofPrograms

OfferedGeneral (more than one theme offered) 147

Nutrition Education/Foods 14

Environmental Education 12

Science 10

Fishing 8

Outdoor Adventure / Survival 7

Clothing/Fashion/Fabrics 6

Shooting Education 5

4-H Camping ProgramsMost Common 4-H Camp Themes

* Total of 22,816 participants

267 385

7,579 9,374

1,276 1,883885 1,164

1

10

100

1,000

10,000

Youth 5-8(Cloverbuds)

Youth 9-13 Teen Counselors 14-18

Adult Leaders19+

Male

Female

NOTE: This is a logarithmic scale.

4-H Camp ParticipationGender

* Total of 22,816 participants

1

10

100

1,000

10,000

100,000

Youth 5-8(Cloverbuds)

Youth 9-13 Teen Counselors 14-18

Adult Leaders19+

WhiteBlackHispanicAsianAmerican IndianMulti-CulturalOther/Missing

NOTE: This is a logarithmic scale.

4-H Camp ParticipationEthnic Groups Represented

* Total of 22,816 participants NOTE: This is a logarithmic scale.

0%

10%

20%

30%

40%

50%

60%

70%

1stYear

2 Years

3 Years

4 Years

5 Years

6 Years

7+Years

CampersCITsTeen CounselorsAdult LeaderExtension Agents

Previous 4-H Camping Experience

Standardized 4-H Standardized 4-H Camp Survey ResultsCamp Survey Results

Description Mean

4-H summer program staff 4.53Teen counselors 4.47Classes 4.35How would you rate 4-H Camp overall? 4.31Adult leaders 4.27Safety at 4-H Camp 4.25Counselors-In-Training (CITs) 4.18Afternoon recreation 4.12Evening programs 4.03Campfires 3.77Reflections/vespers 3.57Food 3.25

* Scale of 1-5, where 1=poor and 5= excellent. (n=9,296)

2001 Junior 4-H Camp Ratings By Youth Campers Ages 9-13*

* Scale of 1-5, where 1=poor and 5= excellent. (n=1,958)

Description Mean

Helpfulness of 4-H Center Summer Staff 4.28Safety at 4-H Camp 4.27Classes 4.17How would you rate 4-H Camp overall? 4.17Evening Programs 4.14Cooperation between your group / 4-H Center staff 4.09Afternoon Recreation 4.08Campfires 3.96Facilities 3.75Reflections / Vespers 3.74Get-Acquainted Activities 3.68Flag Ceremonies 3.60Food 3.09

2001 Junior 4-H Camp Ratings By CITs, Teen Counselors, Adult Volunteers, and Extension Agents*

YouthCampers

9-13 years old(n=9,296)

CITs, Teens,Adults,

ExtensionAgents

(n=1,958)

Parents(n=363)

YES 69% 84% 93%

NO 5% 3% 1%

Maybe / I’mNot Sure

26% 13% 6%

2001 Junior 4-H Camp Interest in Future 4-H Camp Participation*

LIFE SKILL DOMAIN FOUR-FOLD DOMAINSURVEY ITEM

Based on Iowa State’s “Targeting Life Skills Model”Helped me become moreresponsible

Self-Responsibility HEALTH (Being)

Helped me to see the worldbeyond my hometown

Accepting Differences HEART (Relating)

Helped to me make new friends Social Skills HEART (Relating)

Helped me to develop closerfriendships with people Ialready know

Nurturing Relationships HEART (Caring)

Helped me to learn more aboutdifferent subjects

Learning to Learn HEAD (Thinking)

Helped me improve myleaderships skills

Leadership HANDS (Giving)

Helped me to be moreindependent and take care ofmyself

Self-Responsibility HEALTH (Beining)

Helped me improve mycommunication skills

Communication HEART (Relating)

Helped me improve my self-confidence

Self-esteem HEALTH (Being)

Helped me to develop new skillsin an area that I enjoy

Marketable SkillsLearning to Learn

HANDS (Working)HEAD (Thinking)

4-H Camp Standardized Survey Life Skill Items and Related Domains

* Youth campers ages 9-13 were asked to identify whether or not attending camp helped them on a scale of 1-4, where 1= helped me very little and 4=helped me very much.

Has attending Junior 4-H Camp helped you?* Mean

Make new friends 3.34Develop new skills in an area that I enjoy 3.25Be more independent and take care of myself 3.06Learn more about different subjects 3.05Develop closer friendships with people I already know 3.01Improve my self-confidence 2.97Improve my communication skills 2.83Improve my leaderships skills 2.81Become more responsible 2.70See the world beyond my hometown 2.69

2001 Junior 4-H Camp Life Skill Rankings by Youth Campers Ages 9-13*

Additional Analysis of Life Skill Items

Significant difference between the life skill mean response of campers ages ages 8-11 and campers ages 12-14

“helped me to learn more about different subjects” “helped me to develop new skills in an area that I enjoy”

Suggests that younger campers perceive more educational benefits from 4-H camp than older campers.

This trend is noticeable across most life skill areas (except life skills related to friendship development) suggesting that the Virginia 4-H camping program may be geared towards younger campers.

Parent/Guardian Parent/Guardian Survey ResultsSurvey Results

* This figure (43%) represents 89 other camps in which 4-H youth participated.

11%

11%

11%

9%6%5%4%

43%

Basketball CampGirl Scout CampDay CampChurch CampBoy Scout CampSoccer CampAthletic/Sports CampOther camps*

Other Camps in Which 4-H Youth Campers ParticipatedSummer 2001

* Scale of 1-4, where 1=much worse and 4= much better. (n=363)

How did 4-H Camp compare with your child’sother camping experiences?

Mean

Overall Camp Experience 3.96

Recreational Activities 3.92

Teen Counselors and Counselors in Training 3.78

Classes/Educational Programs 3.78

Camp Staff and Instructors 3.75

Camp Fee 3.60

Registration Process 3.41

2001 Junior 4-H Camp Ratings By Parents/Guardians of 4-H Youth Campers*

LIFE SKILL DOMAIN FOUR-FOLD DOMAIN*SURVEY ITEM

Based on Iowa State’s “Targeting Life Skills Model”…takes responsibility for his/her own actions. Self-Responsibility HEALTH (Being)…handles success and failure. Managing Feelings HEALTH (Being)…shares work responsibilities. Contributes to Group Effort HANDS (Giving)…has a good mental attitude. Managing Feelings HEALTH (Being)…listens to the opinions of others. Communication/Social Skills HEART (Relating)…participates in a discussion. Sharing/Communication HEART (Caring/Relating)…follows directions. Cooperation HEART (Relating)…shares own thoughts and ideas verbally. Sharing/Communication HEART (Caring/Relating)…asks questions. Critical Thinking HEAD (Thinking)…tries to find solutions to problems. Problem Solving HEAD (Thinking)…participates in learning experiences. Learning to Learn HEAD (Thinking)…tries to find answers to questions. Critical Thinking/Problem Solving HEAD (Thinking)…makes good decisions. Decision Making HEAD (Thinking)…considers choices before making a decision. Critical Thinking HEAD (Thinking)…adapts to change. Resiliency HEAD (Managing)…sets goals for himself / herself. Goal Setting HEAD (Managing)…takes care of his / her own things. Wise Use of Resources HEAD (Thinking)…takes initiative / is a self-starter. Self-Motivation HANDS (Working)…sets priorities. Goal Setting HEAD (Managing)…Manages money well for his / her age group. Wise Use of Resource HEAD (Thinking)…cooperates and works in a group. Cooperation HEART (Relating)…gets along with people around him / her. Nurturing Relationships/Social Skills HEART (Caring/Relating)…is a team player. Teamwork HANDS (Working)…deals effectively with conflict. Conflict Resolution/Managing

FeelingsHEART (Relating)HEALTH (Being)

4-H Camp Parent/Guardian Survey Life Skill Items and Related Domains

Data Analysis of Life Skill Items

Conducted paired t-test comparison of “before-camp” and “after-camp” life skill averages (means)

Life Skill ItemN

sizeBefore1

CampMean

SDAfter1

CampMean

SD Gain2

Takes care of his/her ownthings

317 3.61 1.08 3.98 .95 .37

Shares workresponsibilities

320 3.63 2.06 3.89 .88 .26

Takes initiative and is aself-starter

315 3.58 1.01 3.82 .93 .24

Takes responsibility forhis/her own actions

319 3.72 .96 3.95 .83 .23

Handles success and failures 318 3.73 .87 3.96 .79 .23

Has a good mental attitude 318 4.10 .87 4.33 .75 .23

Adapts to change 317 3.89 .89 4.12 .80 .23

1 Parents were asked to rate their child’s life skill behavior before and after the Junior 4-H Camp experience on a scale of 1-4, where 1= almost never and 5= almost always.

2 Paired t-test analysis found significant differences (p<.05) between all of the before and after camp mean comparisons.

2001 Junior 4-H Camp Life Skill Rankings by Youth Campers Ages 9-13 Before and After Attending Junior 4-H Camp*

Additional Analysisof Life Skill Items

Additional t-test analysis of life skill rankings among male and female campers indicated individual benefits (rather than gender-related benefits) as perceived by campers’ parents/guardians

“Gender” differences do not explain significant results

Suggests that Junior 4-H camp programs/activities are appropriate for both genders.

Summary of Outcomes Youth (ages 9-13) perceived that Junior 4-H

Camp participation increased their own skills (immediate outcomes)

Developing social relationships with new friends Developing new skills

Parents/guardians perceived that Junior 4-H Camp participation increased their child’s life skill behaviors (short-term outcomes)

“takes care of his/her own things” “shares work responsibilities” “takes initiative and is a self-starter”

Limitations Standardized survey

Response bias (tendency to respond favorably) Lack of pre-test (How did campers’ perceive

their life skills before camp?) Lack of analysis of open-ended responses

Parent/guardian survey Response bias (Who returned the survey? /

What about non-respondents?) Can outcomes be attributed to something else?

Others (What do you think??)

What did we learn?What did we learn?

Value of Systematic Evaluation

1st step has been taken!! Standardized evaluation allowed for state-wide

reporting of 4-H camp outputs and outcomes Standardized evaluation can provide data for

meaningful comparisons Staff/volunteers learned the importance of

evaluation in the programming process Evaluation results are assisting resource

development 4-H Centers Virginia 4-H Foundation

Examine 4-H camp components that received lower ratings

Explore the 12-14 year old camp experience

Where do we go from here?Impact on Virginia 4-H Camping- ProgramsPrograms

Utilize methods that are less labor-intensive

Review life-skills that were assessed in 2001

Qualitative study to explore results CIT and teen leadership 12-14 year old experience Long-term outcomes of 4-H camp involvement

Where do we go from here?Impact on Virginia 4-H Camping- Program Program EvaluationEvaluation

As a 4-H camping community, we should explore 4-H camping outcomes together

COMMUNICATE- Increase communication about our instruments, methods, and results

COLLABORATE- Work together to develop a multi-state approach to evaluating 4-H camp program outcomes

Where do we go from here?Impact on National 4-H Camping

Virginia invites other states to work with us to develop a National 4-H Camp Research Consortium by 2005.

Where do we go from here?Impact on National 4-H Camping

Summary Overview of Virginia 4-H Camping 4-H camp evaluation prior to 2001 2001 data sources Results and Analysis Limitations What did we learn? Future evaluation

State and national levels

What do you still need? Were any of your questions

unanswered? If so, what are they?

What would help you to better understand or apply the information learned in this session?

For additional information:For additional information:

Barry A. Garst Extension Specialist

4-H Staff Development and CampingState 4-H Office

Virginia Techbgarst@vt.edu(540) 231-6373