leaps workforce development cluster study

43
Leadership & Evaluation to Advance Program Success June 2015 Prepared by LEAPS for Environmental Literacy: Engaging Adolescents in Workforce Development

Upload: blwenger

Post on 25-Jan-2016

84 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: LEAPS Workforce Development Cluster Study

Leadership & Evaluation to Advance Program Success June 2015

Prepared  by  

LEAPS  for  Environmental  Literacy:    Engaging  Adolescents  in  Workforce  

Development      

!

Page 2: LEAPS Workforce Development Cluster Study

Cluster  Study  Overview                                                                                                                                                                                                                                              3  

Par<cipant  Overview                                                                                                                                                                                                                                                  12  

Pa@erns  in  Outcomes  Across  Organiza<ons                                                                                                                                                                  16  

Connec<ons  between  Program  Components  and  Outcomes                                                                                                      22                                                              

Compensa<on                                                                                                                                                                                                                                                                          30                                                              

Avenues  for  Future  Evalua<on  and  Learning                                                                                                                                                              34                                                              

Takeaways  for  the  Field                                                                                                                                                                                                                                        36                                                              

Appendices                                                                                                                                                                                                                                                                                    38                                                              

Table  of  Contents  

LEAPS  Cluster  Studies:  Workforce  Development  

Page 3: LEAPS Workforce Development Cluster Study

Cluster  Study  Overview  

Page 4: LEAPS Workforce Development Cluster Study

4  

The  Path  to  Cluster  Studies  

§  Seventeen  LEAPS  organizaBons  have  engaged  in  ongoing  technical  assistance  to  develop  their  evaluaBon  capacity  since  October  2012.  During  the  first  phase  of  this  iniBaBve,  organizaBon  representaBves  aJended  trainings  focused  on  foundaBonal  evaluaBon  and  learning  concepts:    theories  of  change,  logic  models,  evaluaBon  plans,  tool  development,  data  analysis  techniques,  and  communicaBng  effecBveness.  They  also  engaged  in  a  range  of  leadership  capacity  building  acBviBes.  

§  In  the  phase  that  followed  and  with  conBnued  support  from  the  S.D.  Bechtel,  Jr.  FoundaBon,  Learning  for  AcBon  worked  with  clusters  of  organizaBons  to  develop  shared  measures  around  three  cross-­‐cuSng  areas  of  interest.  The  cluster  study  work  arose  out  of  an  opportunity  for  the  cohort  to  begin  to  document  its  collecBve  story  –  which  was  further  emerging  aUer  Phase  I’s  work  –    and  the  desire  to  advance  organizaBons’  evaluaBon  capacity  through  the  hands-­‐on,  group  learning  experience  this  type  of  project  would  provide.  OrganizaBons  worked  together  to  define  and  form  the  clusters,  collect  data,  and  analyze  the  emerging  stories.  

LEAPS  Cluster  Studies:  Workforce  Development  

Page 5: LEAPS Workforce Development Cluster Study

Cluster  Study  Topic  

5   LEAPS  Cluster  Studies:  Workforce  Development  

OrganizaBons  in  the  Workforce  cluster  uBlize  workforce  development  models  centered  on  culBvaBng  youth  leadership  and  voice  in  local  environmental  and  social  jusBce  issues  as  plaYorms  for  preparing  youth  for  future  job  opportuniBes,  and  fostering  greater  connecBon  with  the  environment.  They  have  adapted  to  their  local  communiBes,  engaged  many  different  partners,  and  drawn  from  diverse  funding  sources  in  the  development  of  their  models.    

We  used  the  following  unifying  If-­‐Then  statement  to  test  assump<ons  about  our  models:    If  we  create  relevant,  supporBve  early  employment  experiences  for  youth  that  demonstrate  appreciaBon  and  value  through  compensaBon,  then  youth  will:    §  Develop  transferrable  professional  &  leadership  skills;    §  Increase  commitment  to  environmental  stewardship;  and,    §  Experience  increased  self-­‐efficacy,  including  workplace  self-­‐efficacy.    

Photo  Credit:  CommunityGrows  

Page 6: LEAPS Workforce Development Cluster Study

LEAPS  Cluster  Studies:  Workforce  Development  6  

Cluster  Study  Organiza<ons  

Student  Conserva<on  Associa<on’s  (SCA)  

mission  is  to  build  the  next  generaBon  of  conservaBon  leaders  and  inspire  lifelong  

stewardship  of  the  environment  and  

communiBes  by  engaging  young  people  in  hands-­‐on  

service  to  the  land.  

Rising  Sun  Energy  Center’s    (Rising  Sun)  mission  is  to  empower  individuals  to  achieve  environmental  and  

economic  sustainability  for  themselves  and  their  

communiBes.  

CommunityGrows  culBvates  healthy  youth  

through  growing  gardens  in  low-­‐income,  diverse  communiBes.  

Page 7: LEAPS Workforce Development Cluster Study

LEAPS  Cluster  Studies:  Workforce  Development  7  

Cluster  Study  Organiza<ons  

I   will   apply   the   criBcal   thinking   skills   I've  learned   here   to   almost   every   aspect   of   my  life,   including   my   educaBon   and   my  consumpBon   of   media.   I   will   also   apply   my  new  consciousness  of  professional  skills  to  all  professional  spaces  in  my  life.  

-­‐Youth  Radio  ParBcipant1  

We  would  also  like  to  recognize  Youth  Radio  for  its  parBcipaBon  and  thought  partnership  !

Youth  Radio’s  mission  is  to  launch  young  people  on  career  and  educaBon  pathways  by  engaging  them  in  work-­‐based  learning  opportuniBes,  creaBve  expression,  professional  development,  and  health  and  academic  support  services.  

throughout  the  cluster  work.  Their  program  model  and  approach  share  similar  components  and  intended  impact  as  the  other  organizaBons  within  the  cluster.  While  Youth  Radio  could  not  parBcipate  in  data  collecBon  with  the  cluster,  the  organizaBon  made  significant  contribuBons  to  every  stage  of  this  work,  parBcularly  survey  instrument  development.  

1While  Youth  Radio  did  not  parBcipate  in  cluster  data  collecBon,  this  parBcipant  quote  comes  from  Youth  Radio’s  exisBng  evaluaBon  efforts.    

Photo  Credit:  Youth  Radio  

Page 8: LEAPS Workforce Development Cluster Study

Our  Approach  

8  

Research  Ques<ons  §  What  pa@erns,  if  any,  emerge  across  organizaBons  with  regard  to  rela<ve  gains  

in  professional  skills,  connecBon  to  the  environment,  and  self-­‐efficacy?  §  What,  if  anything,  can  we  learn  about  the  relaBonship  between  different  parts  of  

our  if-­‐then  statement?  §  How  important  is  compensa<on  to  youth?  §  Where  we  have  qualitaBve  informaBon,  in  what  ways  do  youth  describe  the  

rela<onship  between  what  they  did  in  program  and  the  changes  they  experienced?  

• Shared  post-­‐program  survey  aUer  the  summer  or  fall  work  experience.  • OrganizaBons  uBlized  many  common  measures,  with  occasional  customizaBon.  

Methods  

LEAPS  Cluster  Studies:  Workforce  Development  

Page 9: LEAPS Workforce Development Cluster Study

9  

§  Our  survey  was  designed  to  test  our  if-­‐then  statement  in  a  way  that  is  based  on  youth’s  experience  of  a  program,  and  less  conBngent  upon  program  models  or  specific  program  acBviBes.    

§  The  survey  tool  built  upon  other  measurement  tools  and  knowledge  in  the  field  for  a  range  of  survey  secBons  including  the  program  atmosphere,  connecBon  to  the  environment,  self-­‐efficacy,  and  support  secBons.  

§  The  survey  was  administered  anonymously,  increasing  the  likelihood  of  candid  responses.  

 

§  While  surveys  are  useful  for  answering  quesBons  like  “how  much?”  and  “to  what  extent?”,  they  are  less  useful  for  answering  quesBons  like  “how?”  or  “why?”.  While  some  qualitaBve  data  was  collected  in  the  survey,  the  lack  of  addiBonal  qualitaBve  methods  –  such  such  as  interviews  or  focus  groups  –  is  an  inherent  limitaBon  of  the  design.  

§  Youth  responses  were  not  triangulated  with  other  forms  of  program  data.  

§  Due  to  the  Bming  of  cluster  work,  both  survey  design  and  data  collec<on  took  place  in  the  fall.  For  the  programs  that  operate  in  the  summer,  this  was  a"er  the  interven<on,  limiBng  the  types  of  data  that  could  be  collected  and  likely  influencing  response  rates.  

Strengths   Limita<ons  

Strengths  &  Limita<ons  of  Our  Approach  

LEAPS  Cluster  Studies:  Workforce  Development  

Page 10: LEAPS Workforce Development Cluster Study

1Response  rates:  CG  =  9/12,  SCA  =  18/63,  Rising  Sun  =  55/~300    

10  

§  Low  response  rates.  We  had  a  low  response  rate  to  the  survey,1  which  creates  a  fewchallenges:    o  It  may  introduce  bias.  In  parBcular,  we  are  most  

concerned  about  voluntary  response  bias.    o  On  an  organizaBon  level,  too,  a  low  response  rate  

creates  an  issue  with  precision,  reducing  our  confidence  in  whether  the  data  collected  from  our  sample  represents  the  program  populaBon  more  broadly.    

 Nevertheless,  we  find  the  data  valuable  for  looking  for  

pa9erns  and  exploring  our  if-­‐then  statement.  

Addi<onal  Limita<ons  

LEAPS  Cluster  Studies:  Workforce  Development  

Page 11: LEAPS Workforce Development Cluster Study

1Response  rates:  CG  =  9/12,  SCA  =  18/63,  Rising  Sun  =  55/~300    

11  

Use  of  Aggregated  Data  

§  We  only  aggregated  data  when  looking  at  assumpBons  and  beliefs  that  the  programs  share  in  common,  when  looking  for  cross-­‐program  pa@erns,  and  when  we  could  explore  our  if-­‐then  statement  looking  at  the  rela<onship  between  survey  measures.  

§  In  cases  where  we  aggregated  data  for  analysis,  we  have  different  levels  of  representa<on  for  each  organiza<on,  based  on  different  numbers  of  respondents.    o  Specifically,  there  are  9  respondents  from  CommunityGrows,  18  

respondents  from  SCA,  and  55  respondents  from  Rising  Sun.    

LEAPS  Cluster  Studies:  Workforce  Development  

Page 12: LEAPS Workforce Development Cluster Study

ParBcipant  Overview  

Page 13: LEAPS Workforce Development Cluster Study

13  

3%  

11%  

14%  

19%  

23%  

31%  

Pacific  Islander  /  NaBve  Hawaiian  

MulBracial  

White  /  Caucasian  

LaBno  /  Hispanic  

Black  /  African  American  

Asian  

Overall  Sample  

20%  

20%  

40%  

20%  

Pacific  Islander  /  NaBve  Hawaiian  

MulBracial  

White  /  Caucasian  

LaBno  /  Hispanic  

Black  /  African  American  

Asian  

CommunityGrows  

2%  

9%  

11%  

21%  

21%  

36%  

Pacific  Islander  /  NaBve  Hawaiian  

MulBracial  

LaBno  /  Hispanic  

White  /  Caucasian  

Black  /  African  American  

Asian  

Rising  Sun  

13%  

31%  

31%  

25%  

Pacific  Islander  /  NaBve  Hawaiian  

MulBracial  

White  /  Caucasian  

LaBno  /  Hispanic  

Black  /  African  American  

Asian  

SCA  

Survey  Respondent  Demographic  Data:  Ethnicity  

LEAPS  Cluster  Studies:  Workforce  Development  

5   44   16  65  

The  majority  of  the  survey  sample  is  youth  of  color.  Asian/Asian  American  youth  (31%)  make  up  the  largest  proporBon  of  youth  that  submiJed  the  survey.    

Page 14: LEAPS Workforce Development Cluster Study

Survey  Respondent  Demographic  Data:  Gender  

14  

60%  

40%  

Female   Male  

Overall  Sample  

75%  

25%  

Female   Male  

CommunityGrows  

61%  

39%  

Female   Male  

Rising  Sun  

53%  47%  

Female   Male  

SCA  

LEAPS  Cluster  Studies:  Workforce  Development  

4   49   17  70  

Females  comprise  the  majority  of  the  sample  (60%).    

Page 15: LEAPS Workforce Development Cluster Study

Survey  Respondent  Demographic  Data:  Free  or  Reduced  lunch  

15  

I  do  not  know  7%  

No  49%  

Yes  44%  

I  do  not  know  13%  

No  25%  

Yes  63%  

LEAPS  Cluster  Studies:  Workforce  Development  

Overall  Sample   CommunityGrows   Rising  Sun   SCA  

5   48   16  69  

The  sample  is  almost  evenly  split  by  youth  who  qualify  for  free  or  reduced  lunch  (44%)  and  those  who  do  not(49%).  Seven  percent  of  youth  were  unsure  if  they  qualify.    

I  do  not  know  6%  

No  60%  

Yes  33%  

No  20%  

Yes  80%  

Page 16: LEAPS Workforce Development Cluster Study

PaJerns  in  Outcomes  Across  OrganizaBons  

Page 17: LEAPS Workforce Development Cluster Study

17  

Average    (5-­‐point  scale)  (n=53-­‐54)  

a.  Dressing  appropriately     4.7  

b.  Following  direc<ons   4.5  

c.  Showing  up  on  <me     4.4  

d.  Knowing  my  schedule   4.3  

e.  Working  well  with  a  supervisor   4.3  

f.  Working  well  with  others  on  a  team   4.3  

g.  Rela<ng  well  with  other  people   4.2  

h.  Communica<ng  effec<vely  with  a  supervisor   4.2  

i.  Managing  my  <me   4.0  

j.  Being  responsive  to  texts  and  email   4.0  

k.  Handling  disagreements  produc<vely     3.9  

l.  Leading  my  peers   3.9  

LEAPS  Cluster  Studies:  Workforce  Development  

§  Across  programs  surveyed,  dressing  appropriately  and  following  direc<ons  were  consistently  among  the  top  three  professional  skills  with  which  youth  rate  greatest  proficiency  (on  average).    

§  Similarly,  leading  peers,  handling  disagreements  produc<vely,  and  being  responsive  to  text  and  email  consistently  scored  within  the  boJom  four  skills.  

Youth  surveyed  report  varying  levels  of  proficiency  in  professional  skills  

Photo  Credit:  Youth  Radio  

Photo  Credit:  Student  ConservaBon  AssociaBon  

Page 18: LEAPS Workforce Development Cluster Study

18  

Average    (4-­‐point  scale)  (n=62-­‐72)  

a.  Communica<ng  effec<vely  with  a  supervisor   3.1  

b.  Working  well  with  others  on  a  team   3.1  

c.  Working  well  with  a  supervisor   3.0  

d.  Rela<ng  well  with  other  people   2.9  

e.  Managing  my  <me   2.9  

f.  Following  direc<ons   2.9  

g.  Showing  up  on  <me     2.8  

h.  Leading  my  peers   2.8  

i.  Knowing  my  schedule   2.8  

j.  Handling  disagreements  produc<vely     2.8  

k.  Dressing  appropriately   2.7  

l.  Being  responsive  to  texts  and  email   2.6  

LEAPS  Cluster  Studies:  Workforce  Development  

In  my  Bme  with  working  with  BEETS  I've  seen   myself   grow   in   communicaBon  skills.  

-­‐CommunityGrows  ParBcipant  

§  Across  programs  surveyed,  no  clear  pa@ern  emerged  regarding  top  four  and  boJom  four  areas  of  improvement.  

§  Communica<ng  effec<vely  with  a  supervisor  and  managing  my  <me  consistently  appeared  within  the  top  half  of  improvement  averages.    

§  Leading  my  peers  and  knowing  my  schedule  consistently  appeared  in  the  boJom  half.  

Youth  Surveyed  Report  Increases  in  Professional  Skills  As  A  Result  of  Par<cipa<ng  

Page 19: LEAPS Workforce Development Cluster Study

19  

Average    (4-­‐point  scale)  

(n=78)  

a.  It  makes  me  happy  to  see  people  take  care  of  the  environment   3.2  

b.  I  feel  a  personal  responsibility  to  help  protect  the  environment   3.1  

c.  I  am  interested  in  learning  more  about  the  environment   3.1  

d.  I  am  aware  of  environmental  issues  facing  my  community   3.0  

e.  I  feel  personally  impacted  by  environmental  condi<ons  in  my  community   2.9  

f.  I  speak  up  and  talk  with  friends  and  family  about  issues  that  affect  the  environment   2.8  

g.  I  par<cipate  in  ac<vi<es  that  help  the  environment   2.7  

h.  I  am  interested  in  pursuing  a  career  that  has  to  do  with  the  environment   2.6  

LEAPS  Cluster  Studies:  Workforce  Development  

The   program   exposed  me   to   issues   that   I   had   not   previously  known…  I  will  use  what   I've   learned  to  spread  knowledge  and  parBcipate  in  acBviBes  that  help  create  a  beJer  community.    

-­‐Student  ConservaBon  AssociaBon  ParBcipant  

§  “It  makes  me  happy  to  see  people  take  care  of  the  environment”  and  “I  am  aware  of  environmental  issues  facing  my  community”  consistently  appeared  within  the  top  half  of  improvement  averages.  

§  “I  speak  up  and  talk  with  family  and  friends  about  issues  that  affect  the  environment”  and  “I  par<cipate  in  ac<vi<es  that  help  the  environment”  consistently  appeared  in  the  boJom  half.  

Youth  Surveyed  Report  Increased  Connec<on  to  the  Environment  

Photo  Credit:  Student  ConservaBon  AssociaBon  

Page 20: LEAPS Workforce Development Cluster Study

20   LEAPS  Cluster  Studies:  Workforce  Development  

I   am  more   familiar   with   the   technical  aspects   of   simple   eco-­‐friendly   choices,  though   I   s<ll   struggle   in   sugges<ng  these  changes  in  my  friends  and  family.    

-­‐Rising  Sun  ParBcipant  

§  Within  SCA,  we  found  there  is  a  strong  staBsBcally  significant  relaBonship  between  number  of  sessions  in  the  program  and  improvement  in  speaking  up  and  talking  with  family  and  friends  (p<.01).  

§  In  aggregate  we  found  that  greater  prior  frequency  of  par<cipa<on  in  environmentally-­‐focused  ac<vi<es  was  correlated  with  greater  improvement  in  speaking  up  and  talking  with  family  and  friends  (p<.05).  

Digging  Deeper:  Speaking  Up  with  Friends  and  Family  

§  Speaking  up  with  family  and  friends  appears  to  be  an  outcome  that  may  take  longer  to  develop.  

Photo  Credit:  CommunityGrows  

Page 21: LEAPS Workforce Development Cluster Study

§  Youth  surveyed  report  improvement  in  self-­‐efficacy,  including  improvement  in  their  beliefs  about  future  workforce  success.  

21  

Average    (4-­‐point  scale)  

(n=71-­‐74)  

a.  Even  when  I  don’t  know  how  to  do  something  right  away,  I  can  learn  how   3.3  

b.  I  believe  I  will  succeed  in  my  future  jobs   3.2  

c.  I  am  confident  that  I  can  do  well  on  many  different  professional  tasks   3.1  

d.  I  can  reach  my  goals  if  I  try   3.1  

e.  I  can  remain  calm  when  facing  difficul<es   3.1  

f.  When  I  am  confronted  with  a  problem,  I  can  usually  find  several  solu<ons   3.0  

g.  When  I  need  help,  I  know  where  to  go   2.9  

LEAPS  Cluster  Studies:  Workforce  Development  

§  “Even  when  I  don’t  know  how  to  do  something  right  away,  I  can  learn  how”  consistently  appeared  within  the  top  two  improvement  averages.  

§  “When  I  need  help,  I  know  where  to  go”  and  “I  can  remain  calm  when  facing  difficul<es  consistently  appeared  in  the  boJom  half  of  improvement  averages.  

Youth  Report  Improvement  in  Self-­‐Efficacy  

Photo  Credit:  CommunityGrows  

Page 22: LEAPS Workforce Development Cluster Study

ConnecBons  between  Program  Components  and  Outcomes  

Page 23: LEAPS Workforce Development Cluster Study

Rela<onship  between  Relevancy  and  Increased  Connec<on  to  the  Environment  

§  The  more  youth  had  the  opportunity  to  work  on  a  relevant  project  or  job  skill,  the  greater  growth  they  reported  in  connec<on  to  the  environment.  

23  

Correla<on  with  Relevancy  (n  =  78)  

a.  It  makes  me  happy  to  see  people  take  care  of  the  environment   **  b.  I  feel  a  personal  responsibility  to  help  protect  the  environment   ***  c.  I  am  interested  in  learning  more  about  the  environment   ***  d.  I  am  aware  of  environmental  issues  facing  my  community   *  e.  I  feel  personally  impacted  by  environmental  condi<ons  in  my  community     ***  f.  I  speak  up  and  talk  with  friends  and  family  about  issues  that  affect  the  environment     **  g.  I  par<cipate  in  ac<vi<es  that  help  the  environment   ***  h.  I  am  interested  in  pursuing  a  career  that  has  to  do  with  the  environment   ***  

In  the  chart  above:    •  Darkness  of  shading  corresponds  to  the  esBmated  effect  size  of  the  relaBonship:  n  =  small,  n  =  

moderate,  n  =  large  •  Asterisks  (*)  represent  the  p-­‐value  (indicaBng  level  of  significance)  of  the  relaBonship:  *  =  p<.05,  

**  =  p<.01,  ***  =  p<.001    

LEAPS  Cluster  Studies:  Workforce  Development  

§  All  eight  connecBon  to  the  environment  measures  were  posiBvely  correlated  with  having  the  opportunity  to  work  on  a  relevant  project  or  job  skill.  

Photo  Credit:  Rising  Sun  Energy  Center  

Page 24: LEAPS Workforce Development Cluster Study

I've  always  wanted  to  work  with  animals  growing  up  and  I  always  thought  that  it  had  to  be  direct  contact  jobs  such  as  a  veterinarian,  but  during  my  job  at  SCA,  I  learned  that  we  were  restoring  habitat  because  we  wanted  to  rebuild  the  homes  of  different  animals.  Restoring  habitat  was  an  indirect  way  to  help  animals  and  when  I  learned  that,  I  felt  a  lot  more  passionate  about  what  I  was  doing.  In  all,  the  job  experience  opened  my  eyes  to  the  different  career  possibili<es  that  have  the  same  goal  in  mind,  which  is  caring  for  and  protecBng  animals.    

   -­‐SCA  ParBcipant  

Page 25: LEAPS Workforce Development Cluster Study

25  

§  Youth’s  raBngs  of  supporBve  adults  were  correlated  with  some  environment  and  self-­‐efficacy  measures.1    

1See  Appendix  A  for  more  details  

LEAPS  Cluster  Studies:  Workforce  Development  

The  thing  that  helped  me  the  most  to  boost  my  confidence   and   self   esteem   were   my  encouraging  and  understanding  managers.  

-­‐Rising  Sun  ParBcipant  

Prior   to   the   program   I   thought   I   couldn't   deal  with  problems  on  a  crew.  But  this  past  summer  showed   me   that   not   all   crews   were   nice   and  calm   and   problems   can   and   will   happen.   So  even  with   the  problems   in  my  crew   I  was  able  to   deal  with   them  well.   I   also   discussed   these  issues   with   my   supervisors   and   learned   a   lot  about  conflict  mediaBon.    

-­‐Student  ConservaBon  AssociaBon  ParBcipant  

Youth  Ra<ngs  of  Adult  Support  and  Modeling  Were  Posi<vely  Correlated  with  Various  Outcomes  

Photo  Credit:  CommunityGrows  

§  Most  notably,  Bmely  enforcement  of  rules  and  agreements  was  posiBvely  correlated  with  seven  of  eight  dimensions  of  improvement  in  connecBon  to  the  environment.    

§  In  addiBon,  adults  appears  to  be  one  of  the  few  program  model  characterisBcs  correlated  with  dimensions  of  self-­‐efficacy.  

Page 26: LEAPS Workforce Development Cluster Study

§  Par<cipants  who  reported  greater  agreement  that  they  regularly  collaborate  and  share  opinions  also  tended  to  report  greater  improvement  in  connecBon  with  the  environment.  

26  

In  this  program,  we  regularly  collaborate,  share  our  opinions  with  

others,  and  work  in  teams  (n  =  78)  

a.  It  makes  me  happy  to  see  people  take  care  of  the  environment  

b.  I  feel  a  personal  responsibility  to  help  protect  the  environment   **  c.  I  am  interested  in  learning  more  about  the  environment   *  d.  I  am  aware  of  environmental  issues  facing  my  community  

e.  I  feel  personally  impacted  by  environmental  condi<ons  in  my  community   **  f.  I  speak  up  and  talk  with  friends  and  family  about  issues  that  affect  the  environment   *  g.  I  par<cipate  in  ac<vi<es  that  help  the  environment     **  h.  I  am  interested  in  pursuing  a  career  that  has  to  do  with  the  environment   *  

In  the  chart  above:    •  Darkness  of  shading  corresponds  to  the  esBmated  effect  size  of  the  relaBonship:  n  =  small,  n  =  

moderate,  n  =  large  •  Asterisks  (*)  represent  the  p-­‐value  (indicaBng  level  of  significance)  of  the  relaBonship:  *  =  p<.05,  

**  =  p<.01,  ***  =  p<.001    

LEAPS  Cluster  Studies:  Workforce  Development  

Posi<ve  Connec<on  between  Collabora<on  and  Environmental  Outcomes  

Photo  Credit:  Student  ConservaBon  AssociaBon  

Page 27: LEAPS Workforce Development Cluster Study

27  

Is  improvement  in  one  set  of  outcomes  -­‐  such  as  professional  skills,  connec@on  to  the  environment,  or  self  efficacy  –  correlated  with  

improvement  in  other  outcome  measures?  

§  We  saw  many  sta<s<cally  significant  posi<ve  correla<ons  between  different  types  of  outcomes,  telling  us  that  youth  who  tended  to  report  greater  improvement  in  one  dimension,  also  tended  to  report  greater  improvement  in  others.1  

1See  Appendix  B  for  more  details  

LEAPS  Cluster  Studies:  Workforce  Development  

Connec<ons  Between  Different  Types  of  Outcomes  

Photo  Credit:  CommunityGrows  

Page 28: LEAPS Workforce Development Cluster Study

§  Improvement  in  professional  skills  emerged  as  related  to  three  specific  dimensions  of  improvement  in  self  efficacy:    o  “I  believe  I  will  succeed  in  

my  future  jobs.”  o  “I  am  confident  that  I  can  

do  well  on  many  different  professional  tasks.”  

o  “I  can  reach  my  goals  if  I  try.”    

I  believe  I  will  succeed  in  my  future  jobs  (n=59-­‐68)  

I  am  confident  that  I  can  do  well  on  many  different  

professional  tasks  (n  =  58-­‐67)  

I  can  reach  my  goals  if  I  try  (n=59-­‐69)  

a.  Communica<ng  effec<vely  with  a  supervisor   **   **   **  b.  Working  well  with  others  on  a  team   *   *  c.  Working  well  with  a  supervisor   **   ***  d.  Rela<ng  well  with  other  people   **   *   ***  

e.  Managing  my  <me   **   *   **  

f.  Following  direc<ons   ***   *   ***  

g.  Showing  up  on  <me     ***   **   ***  

h.  Leading  my  peers   *   **   **  

i.  Knowing  my  schedule   **   *   ***  j.  Handling  disagreements  produc<vely     *   *   *  

k.  Dressing  appropriately   ***   **   ***  

l.  Being  responsive  to  texts  and  email   ***   **   ***  

In  the  chart  above:    •  Darkness  of  shading  corresponds  to  the  esBmated  effect  size  of  the  relaBonship:  n  =  small,  n  =  moderate,  n  =  large  •  Asterisks  (*)  represent  the  p-­‐value  (indicaBng  level  of  significance)  of  the  relaBonship:  *  =  p<.05,  **  =  p<.01,  ***  =  p<.

001    

LEAPS  Cluster  Studies:  Workforce  Development  

Being   able   to   communicate  and   work   beJer   with   clients  and   customers.   This   gave   me  higher  self  esteem.  

-­‐Rising  Sun  ParBcipant  

28  

Connec<ons  between  Improvement  in  Professional  Skills  and  Improvement  in  Self  Efficacy  

Page 29: LEAPS Workforce Development Cluster Study

29  

§  In  various  open-­‐ended  responses,  youth  commented  on  the  way  in  which  certain  elements  of  the  program  boosted  their  self  esteem  and  confidence.  

LEAPS  Cluster  Studies:  Workforce  Development  

Before   I  began  the  program,   I  was  very  nervous  about  facing   challenges,   especially   ones   that   seemed  overwhelming.   Now   I   feel   much   more   confident   in  myself   and   the   people   around   me   when   taking   on  heavy   tasks.   Having   the   group   set   a   goal   helped   set   a  goal-­‐driven   atmosphere   and   that   is   essenBal   towards  geSng   tasks   done.   I   became  moBvated   to   accomplish  the  tasks  as  I  did  not  want  to  let  the  team  down.    

-­‐Rising  Sun  ParBcipant  

The   overall   work   experience,  which   I've  never  had  before  this   is  what   increased   my   confidence   in  my   abiliBes.   I   was   able   to   finally  see   for   this  first  Bme  how   I  would  actually  do.  

-­‐Student  ConservaBon  AssociaBon  ParBcipant  

Digging  Deeper:  Youth  Comments  Related  to  Self  Efficacy  

Photo  Credit:  Rising  Sun  Energy  Center  

Page 30: LEAPS Workforce Development Cluster Study

CompensaBon  

Page 31: LEAPS Workforce Development Cluster Study

8%  

25%  

33%   33%  

Not  At  All  Important  

Somewhat  Important  

Important   Very  Important  

31  

How  important  was  it  to  you  to  have  a  paid  internship/posi<on?  (First  Year)  

2%   13%  20%  

64%  

Not  At  All  Important  

Somewhat  Important  

Important   Very  Important  

Average  =  3.5  (n  =  45)  

Average  =  2.9  (n  =  24)  

How  important  was  it  to  you  to  have  a  paid  internship/posi<on?  (Returning)  

LEAPS  Cluster  Studies:  Workforce  Development  

§  While  84%  of  first  year  respondents  report  that  compensaBon  was  important  or  very  important  to  them,  only  66%  of  those  who  have  been  in  the  program  for  more  than  a  year  rate  it  as  important  or  very  important.  

How  Important  Is  Compensa<on?  

Page 32: LEAPS Workforce Development Cluster Study

29%  

63%  

4%   4%  

Yes   Maybe   Probably  Not  

No  

32  

If  your  posi<on  were  not  paid,  would  you  s<ll  have  chosen  to  par<cipate?  (First  Year)  

Average  =  2.6  (n  =  45)  

Average  =  1.8  (n  =  24)  

If  your  posi<on  were  not  paid,  would  you  s<ll  have  chosen  to  par<cipate?  (Returning)  

16%  

33%   31%  

20%  

Yes   Maybe   Probably  Not  

No  

LEAPS  Cluster  Studies:  Workforce  Development  

§  While  just  over  half  (51%)  of  first  year  parBcipants  say  that  they  either  would  not  or  probably  would  not  have  parBcipated  if  the  posiBon  were  not  paid,  that  percent  drops  to  8%  of  those  who  have  been  in  the  program  for  more  than  a  year.  

Compensa<on  A@racts  Par<cipants  Who  May  Not  Have  Par<cipated  Otherwise  

Page 33: LEAPS Workforce Development Cluster Study

It  gave  me  confidence  and  told  me  that  yes,  I  can  make  money,  and  that  I'm  more  than  just  a  high  school  student.  

-­‐Rising  Sun  ParBcipant        

Page 34: LEAPS Workforce Development Cluster Study

Avenues  for  Future  EvaluaBon  and  Learning  

Page 35: LEAPS Workforce Development Cluster Study

Avenues  for  Future  Evalua<on  and  Learning    

35  

As  organizaBons  conBnue  to  explore  their  effecBveness,  they  may  choose  to:    §  ConBnue  to  collect  program  data,  re-­‐running  important  analyses  to  beJer  

understand  these  relaBonships  on  an  organizaBon-­‐specific  level  (here  we  looked  at  aggregate  results).  Moreover,  having  a  greater  range  of  the  organizaBon's  parBcipants  involved  may  change  the  relaBonships  organizaBons  see.    

§  Look  at  composite  scores  for  some  of  the  core  concepts  –  professional  skills,  connecBon  to  the  environment,  and  self-­‐efficacy  –  that  we  explored  through  individual  items.  

§  Consider  more  qualitaBve  evaluaBon  to  explore  the  “how”  and  “why”  of  some  of  the  links  that  emerge  in  the  quanBtaBve  data.  

§  Explore  potenBal  drivers  of  change  for  workforce  and  self-­‐efficacy  outcomes.  §  Collect  data  alongside  program  delivery  to  ensure  high  response  rates.    

LEAPS  Cluster  Studies:  Workforce  Development  

Page 36: LEAPS Workforce Development Cluster Study

Takeaways  for  the  Field  

Page 37: LEAPS Workforce Development Cluster Study

§  For  environmental  organiza<ons  looking  to  increase  the  connecBon  to  the  environment,  creaBng  opportuniBes  for  adolescents  to  work  on  projects  or  skills  they  care  about  may  support  greater  increases  in  connec<on  to  the  environment,  including  greater  interest  in  environmentally-­‐focused  careers.  

§  For  environmental  educaBon  programs  that  serve  adolescents,  providing  a  workforce  opportunity  may  a@ract  youth  who  might  otherwise  not  be  inclined  to  par<cipate,  and  may  further  support  their  sense  of  self-­‐efficacy.  

37   LEAPS  Cluster  Studies:  Workforce  Development  

Takeaways  for  the  Field  

Page 38: LEAPS Workforce Development Cluster Study

Appendices  

Page 39: LEAPS Workforce Development Cluster Study

Seven  of  eight  connec<on  to  the  environment  dimensions  were  posi<vely  correlated  with  <mely  enforcement  of  rules  and  agreements.    

Adults  in  this  program  

want  to  see  me  succeed.  

(n=72)  

I  would  feel  comfortable  asking  a  program  staff  

member,  adult  mentor  or  professional  if  I  needed  a  

reference  for  school  or  a  job  (n=71)  

There  is  someone  in  this  program  –  a  staff  person,  mentor,  professional,  or  other  adult  –  who  I  can  go  to  for  help  when  

making  decisions  that  are  important  to  me  

(n=72)  

Instructors,  staff,  and/or  other  leaders  in  this  program  “walk  

their  talk”  by  modeling  the  

behavior  they  expect  of  us  (n=78)  

When  someone  breaks  our  rules  or  agreements  in  this  program,  the  issue  

is  addressed  rather  than  ignored  (n=78)  

a.  It  makes  me  happy  to  see  people  take  care  of  the  environment   **   **   **  b.  I  feel  a  personal  responsibility  to  help  protect  the  environment   ***  c.  I  am  interested  in  learning  more  about  the  environment   *   *   ***  d.  I  am  aware  of  environmental  issues  facing  my  community   **   *   *  e.  I  feel  personally  impacted  by  environmental  condi<ons  in  my  community  

**  

f.  I  speak  up  and  talk  with  friends  and  family  about  issues  that  affect  the  environment  

**  

g.  I  par<cipate  in  ac<vi<es  that  help  the  environment   *   *   ***  h.  I  am  interested  in  pursuing  a  career  that  has  to  do  with  the  environment  

*   *   *  

Suppor<ve  Adults   Adult  Modeling  

Appendix  A:  Tables  Related  to  Suppor<ve  Adults  

Page 40: LEAPS Workforce Development Cluster Study

Adults  in  this  program  want  to  see  me  succeed  

(n=69-­‐72)  

I  would  feel  comfortable  asking  a  program  staff  member,  

adult  mentor  or  professional  if  I  needed  a  reference  for  school  

or  a  job  (n=69-­‐71)  

There  is  someone  in  this  program  –  a  staff  person,  mentor,  professional,  or  other  adult  –  who  I  can  go  to  for  help  when  making  

decisions  that  are  important  to  me  

(n=69-­‐72)  

Instructors,  staff,  and/or  other  leaders  in  this  program  “walk  their  talk”  by  modeling  the  behavior  they  expect  of  

us  (n=71-­‐74)  

When  someone  breaks  our  rules  or  agreements  in  this  program,  the  issue  is  addressed  rather  than  ignored  (n=71-­‐74)  

a.  Even  when  I  don’t  know  how  to  do  something  right  away,  I  can  learn  how   *   *  

b.  I  believe  I  will  succeed  in  my  future  jobs.   *  

c.  I  am  confident  that  I  can  do  well  on  many  different  professional  tasks   *   **   *  

d.  I  can  reach  my  goals  if  I  try  

e.  I  can  remain  calm  when  facing  difficul<es   *   *  

f.  When  I  am  confronted  with  a  problem,  I  can  usually  find  several  solu<ons.   *   *  

g.  When  I  need  help,  I  know  where  to  go   **   **  

To  some  extent,  par<cipants  who  reported  greater  agreement  with  various  statements  about  suppor<ve  adults  also  tended  to  report  greater  improvement  in  self-­‐efficacy.     Suppor<ve  Adults   Adult  Modeling  

Appendix  A:  Tables  Related  to  Suppor<ve  Adults  

Page 41: LEAPS Workforce Development Cluster Study

Even  when  I  don’t  know  how  to  do  something  right  away,  I  can  learn  

how  (n=72)  

I  believe  I  will  succeed  in  my  future  jobs  (n=74)  

I  am  confident  that  I  can  do  well  on  many  different  professional  

tasks  (n=73)  

I  can  reach  my  goals  if  I  

try  (n=74)  

I  can  remain  calm  when  

facing  difficul<es  (n=71)  

When  I  am  confronted  with  a  problem,  I  can  usually  find  several  

solu<ons  (n=73)  

When  I  need  help,  I  know  where  to  go  

(n=72)  

a.  It  makes  me  happy  to  see  people  take  care  of  the  environment  

**   **   ***   *   **   ***   ***  

b.  I  feel  a  personal  responsibility  to  help  protect  the  environment  

*   *   *   *   *  c.  I  am  interested  in  learning  more  about  the  environment  

**   ***   *   **   **   **  d.  I  am  aware  of  environmental  issues  facing  my  community  

**   **   **   **   *   *   ***  e.  I  feel  personally  impacted  by  environmental  condi<ons  in  my  community  

*   **   *   **   **   *  f.  I  speak  up  and  talk  with  friends  and  family  about  issues  that  affect  the  environment  

**   *   *   **   ***  

g.  I  par<cipate  in  ac<vi<es  that  help  the  environment   **   *   *   *  

h.  I  am  interested  in  pursuing  a  career  that  has  to  do  with  the  environment  

***   **   **   *   *   *   **  

Par<cipants  who  reported  greater  improvement  on  self  efficacy  measures  also  tended  to  report  greater  improvement  in  connec<on  to  the  environment.    

Appendix  B:  Connec<ons  Between  Outcomes  

Page 42: LEAPS Workforce Development Cluster Study

Youth  who  tended  to  report  greater  improvement  in  professional  skills  also  tended  to  report  greater  improvement  in  connec<on  to  the  environment  (and  vice  versa).    

It  makes  me  happy  to  see  people  take  care  of  the  environment  (n=62-­‐72)  

I  feel  a  personal  responsibility  to  help  protect  the  environment  (n=62-­‐72)  

I  am  interested  in  learning  

more  about  the  environment    (n=62-­‐72)  

I  am  aware  of  environmental  issues  facing  

my  community    (n=62-­‐72)  

I  feel  personally  impacted  by  environmental  condi<ons  in  my  

community  (n=62-­‐72)  

I  speak  up  and  talk  with  friends  and  family  about  issues  that  affect  the  environment  

(n=62-­‐72)  

I  par<cipate  in  ac<vi<es  that  help  the  environment  (n=62-­‐72)    

 

I  am  interested  in  pursuing  a  career  that  has  to  do  with  the  environment  (n=62-­‐72)  

a.  Communica<ng  effec<vely  with  a  supervisor   **   **   **   **   **   *   *  b.  Working  well  with  others  on  a  team   *   **  c.  Working  well  with  a  supervisor   **   **   *   ***   **   *   **  d.  Rela<ng  well  with  other  people   *   ***   ***   **   ***   ***  e.  Managing  my  <me   **   ***   ***   *   ***   **   **   **  f.  Following  direc<ons   *   **   ***   **   ***   **   *  g.  Showing  up  on  <me     **   **   **   *   **  h.  Leading  my  peers   **   **   *   ***   **   **   **  i.  Knowing  my  schedule   *   **   **   *   *  j.  Handling  disagreements  produc<vely    

**   **   ***   **   **   ***  k.  Dressing  appropriately   *   ***   **   *   **   **  l.  Being  responsive  to  texts  and  email   **   *   *   *   *   *  

Appendix  B:  Connec<ons  Between  Outcomes  

Page 43: LEAPS Workforce Development Cluster Study

www.learningforacBon.com  170  Capp  Street,  Suite  C        |        San  Francisco,  CA  94110  

For  more  informaBon  about  the  Workforce  Development  Cluster  Study,  please  contact  Frances  Tompkins  ([email protected])  or  Jessica  Xiomara  García  ([email protected]).