program evaluation of the commonwealth seminar · program evaluation of the commonwealth seminar...

Program Evaluation of theCommonwealth SeminarOpening the doors of the State House

The report was prepared by Whitney Donaldson (class of 2012), Victor

Wong (class of 2011), and Shadman Zaman (class of 2012), under the

supervision of project manager Kim Pernell (PhD Candidate in Sociology),

Professor Christopher Winship and Dr. David Ager.

All are members of Harvard University’s Department of Sociology.

This evaluation was sponsored by the Barr Foundation.

Special thanks to Rahn Dorsey for his guidance on the evaluation scope and

his insights on the program.

2

Executive Summary

The Commonwealth Seminar aims to “open the doors of the Massachusetts State

House” to diverse community leaders, with the hope that greater access to the

legislative process will enable participants to become better advocates for their

communities and increase minority representation in the public service sector. To

achieve this mission, the Seminar employs three initiatives: (1) a six‐week

legislative training seminar, (2) networking opportunities with participants and

policymakers, and (3) an online job bank. Between June 2010 and May 2011, we, a

research team from Harvard University's Department of Sociology, conducted 33

in‐depth interviews with randomly selected Seminar graduates, administered an

internet‐based survey to all graduates (N = 172 responses), and observed

Commonwealth Seminar‐sponsored events and legislative training seminar

sessions to assess the extent to which existing Commonwealth Seminar programs

have achieved their stated objectives.

Overall, we find that the Commonwealth Seminar has been overwhelmingly

successful in fulfilling its mission. Almost 80% of respondents agree that they have

become better advocates for their communities, and considerable evidence

suggests that the Commonwealth Seminar generates beneficial outcomes, both for

the organizations participants represent and for participants’ own career

advancement.

3

We undertake an in‐depth evaluation of five program aspects: respondent

characteristics, recruiting, the legislative training seminar, networking, and the

online job bank. We summarize findings for each of these aspects below.

Respondent Characteristics

The Commonwealth Seminar serves a diverse group of community leaders:

participants vary in age, gender, racial/ethnic identification, occupation,

and leadership activity.

Recruiting

The Commonwealth Seminar relies primarily on word‐of‐mouth, informal

recruiting strategies. We find that this strategy has been largely successful

in attracting interested and qualified candidates. However, we recommend

incorporating additional formal recruitment strategies to reach a wider pool

of diverse leaders.

Legislative Training Seminar

Respondents were highly satisfied with the legislative training seminar. The

vast majority of respondents considered each topic covered during the

seminar to be useful or extremely useful. Respondents cited policymakers’

presentations and the location of the seminar in the Massachusetts State

house as particularly effective aspects of the legislative training seminar.

The majority (61%) of respondents have contacted a Massachusetts political

representative or policymaker since graduating from the seminar, and a full

4

(77%) of respondents agree that they have become better advocates for

their communities.

Networking

This report evaluates the Commonwealth Seminar’s provision of two types

of networking opportunities: opportunities to network with fellow

participants and opportunities to network with policymakers. Only 34% of

respondents were satisfied with available opportunities to network with

Seminar participants. Relative dissatisfaction stemmed from insufficient

opportunities to network with fellow participants during the seminar and a

desire for more formal networking events after the seminar ended.

Relationships formed through the Commonwealth Seminar proved valuable

for many: 32% of respondents had collaborated on a work‐related project

with someone they met through the seminar; 43% learned about or

obtained resources for their organization via someone they met through the

seminar; and 43% learned about a job opportunity from someone they met

during the seminar.

Job Bank

The majority (71%) of survey respondents were aware of the job bank, and

most (57%) of these respondents had used the job bank in some way. The

majority (69%) of respondents were satisfied with this program aspect.

Respondents explained that two features of the job bank were responsible

5

for their satisfaction: (1) available jobs are compiled into a single, easy‐to‐

find list, and (2) the opportunities posted are of high quality.

The report concludes with three overarching program recommendations:

(1) improve outreach through additional formal recruiting tactics and

through strengthening alumni relations; (2) increase the number of formal

networking events, both during and after the seminar; and (3) ensure

continued success and enhance the sustainability of the Seminar by hiring a

part‐time outreach and alumni relations coordinator.

6

TABLE OF CONTENTS

EXECUTIVE SUMMARY ............................................................................................. 2 RESPONDENT CHARACTERISTICS ............................................................................................... 3 RECRUITING ............................................................................................................................... 3 LEGISLATIVE TRAINING SEMINAR ............................................................................................... 3 NETWORKING ............................................................................................................................ 4 JOB BANK ................................................................................................................................... 4

THE COMMONWEALTH SEMINAR ....................................................................... 7

THE PROGRAM EVALUATION ................................................................................. 8

METHODOLOGY ............................................................................................................ 9

BROAD OVERVIEW ................................................................................................... 11

RESPONDENT CHARACTERISTICS .................................................................... 12

RECRUITMENT ........................................................................................................... 16

LEGISLATIVE TRAINING SEMINAR .................................................................... 18 BACKGROUND .......................................................................................................................... 18 Topic Usefulness ................................................................................................................................................. 19

POLICYMAKERS ........................................................................................................................ 20 THE STATE HOUSE ................................................................................................................... 22 SATISFACTION: LEGISLATIVE TRAINING ................................................................................... 23 BENEFITS AND OUTCOMES ...................................................................................................... 24

NETWORKING ............................................................................................................ 27 BACKGROUND .......................................................................................................................... 27 NETWORKING OVERALL .......................................................................................................... 28 NETWORKING WITH SEMINAR PARTICIPANTS .......................................................................... 29 SATISFACTION: NETWORKING WITH PARTICIPANTS ................................................................. 31 BENEFITS AND OUTCOMES ...................................................................................................... 32 NETWORKING WITH POLICYMAKERS ....................................................................................... 35

JOB BANK .................................................................................................................... 38 BACKGROUND .......................................................................................................................... 38 SATISFACTION: JOB BANK ........................................................................................................ 39 BENEFITS AND OUTCOMES ...................................................................................................... 40 SUMMARY AND RECOMMENDATIONS ...................................................................................... 41

CONCLUSION & RECOMMENDATIONS ........................................................... 41 RECOMMENDATION 1: IMPROVE OUTREACH ........................................................................... 41 RECOMMENDATION 2: FORMALIZE NETWORKING ................................................................... 43 RECOMMENDATION 3: ENSURING CONTINUED SUCCESS ......................................................... 44

APPENDIX .................................................................................................................... 46

7

The Commonwealth Seminar

Joel Barrera and Senator Jarrett

Barrios founded the Commonwealth

Seminar in 2003. The stated mission

of the program is to “open the doors

of the State House” to diverse

leaders in Massachusetts.

Participants are encouraged to learn

more about the legislative process

and establish cross‐sector, cross‐

racial/ethnic, bi‐partisan

relationships in the course of becoming better advocates for their respective

communities. Since its inception, the program has served over 650 participants.

There are three major aspects of the Commonwealth Seminar. They include:

(1) A six‐week legislative training seminar

This aspect of the program is designed to demystify the legislative process

through teaching participants about the legislative process in

Massachusetts. Topics range from the role of lobbyists in policymaking to

the importance of networking to the state budget process. Participants

receive information about these topics from various guest speakers,

Commonwealth Seminar Founder Joel Barrera (right) and Governor Deval Patrick (left)

8

including policymakers (i.e. state senators, elected representatives,

legislative aides, and lobbyists).

(2) Networking opportunities

The Commonwealth Seminar provides opportunities for participants to

develop relationships with one another and with policymakers, recognizing

that strong networks enable participants to access valuable resources and

information. The program provides informal networking opportunities

during the seminar (such as break time) and formal networking

opportunities (such as graduations or other formal events) after the

seminar. Participants are also given an opportunity to connect with the

policymakers who speak at the seminar.

(3) The online job bank

In 2005, the Commonwealth Seminar developed an online job bank, which

consists of a centralized listing of public service job opportunities. Jobs are

posted as they become available, and the program advertises the job bank

via a hyperlink in the Commonwealth Seminar newsletter.

The Program Evaluation

The primary objective of this program evaluation is to provide stakeholders with

new and valuable information about how and to what degree the programs of the

9

Commonwealth Seminar effectively achieve the Seminar’s stated objectives.

Specifically, the evaluation will:

(1) Identify the beneficiaries of Commonwealth Seminar programs and

assess how applicants come to learn about the seminar.

(2) Identify the aspects of the program that have been effective for

participants and the aspects that would benefit from improvement.

(3) Increase understanding of the process by which Commonwealth

Seminar programs generate desired outcomes for seminar graduates,

including deeper and more sustained advocacy at the state level, improved

access to resources and information, and professional advancement.

(4) Determine whether the effectiveness of Commonwealth Seminar

programs varies across different groups of seminar participants. Perceptions

may differ according to participant age, gender, race/ethnicity, occupation,

leadership activity, and/or year of graduation from the program.

Methodology

We collected both quantitative and qualitative data in order to evaluate the

effectiveness of the Commonwealth Seminar. The qualitative data consists of 33 in‐

depth interviews with Seminar graduates, completed between October 2010 and

March 2011. We began the interview phase of the evaluation by conducting pilot

interviews with five respondents who were referred to us by executive director Joel

10

Barrera. The remaining 28 respondents were randomly selected from a list of all

Commonwealth Seminar graduates. Interviews were transcribed by Datalyst

transcription service. The research team coded and analyzed all interviews using

the Atlas.ti analytic software. The purpose of collecting interview data was to learn

more about respondents’ Commonwealth Seminar experiences, uncover the

rationales behind respondent satisfaction or dissatisfaction with various program

aspects, and to better understand how the Commonwealth Seminar experience

affected respondents’ advocacy efforts, personal and professional relationships,

and career trajectories. See the Appendix for a copy of the interview questionnaire.

We used the interview data to develop a survey questionnaire. The primary

goal of the internet‐based survey was to identify broad patterns in respondent

satisfaction and to test whether patterns revealed in the interview data were

generalizable to the entire population of Commonwealth Seminar graduates. In

February 2011, we sent an email to all Seminar graduates with up‐to‐date contact

information, inviting them to take the online survey. Executive director Joel

Barrera publicized the survey via an email to the Commonwealth Seminar listserv,

and we sent two additional follow‐up emails before closing the survey in mid‐

February. In both interviews and in the survey questionnaire, we stressed

respondent confidentiality: all participants were assured that identifying

information would not be linked with their responses in the final report. We

received 172 responses from seminar graduates. See the Appendix for a copy of the

survey questionnaire.

11

Broad Overview

Overall, we find that the

Commonwealth Seminar

largely fulfills its

important mission, and

participants have found

the program to be a

rewarding and satisfying

experience.

The following quotes from interview respondents are indicative of this general

and widely shared sentiment:

“I was happy with my Commonwealth Seminar experience and wish that it had lasted longer!” “It's a great primer and…it has impacted the way I think about policy‐making and politics.” “My experience was great. The Commonwealth Seminar was absolutely worth the time, the effort and the travel expense…it is a powerful testament to commitments by all involved.”

Overall satisfaction with the program is very high, with the vast majority of

respondents (86%) indicating that they were satisfied or extremely satisfied with

their Commonwealth Seminar experience. However, respondents were more

satisfied with some aspects of the program than others, as seen in the following

graph:

12

In the sections that follow, we provide an in‐depth evaluation of five

specific program aspects of the Commonwealth Seminar: respondent

characteristics, recruiting, the legislative training seminar, networking, and the job

bank.

Respondent Characteristics

Unsatis2ied 3%

Moderately Satis2ied 12%

Satis2ied 85%

Unsatis2ied

Moderately Satis2ied

Satis2ied

13

In keeping with its mission, the Commonwealth Seminar has cultivated a diverse

community of leaders, attracting participants from a wide range of ages,

racial/ethnic groups, occupations, and leadership activities.

The program draws participants from a broad age range. However, most

respondents were in their late 20’s or early 30’s, which suggests that the

Commonwealth Seminar primarily serves people in the early stages of their

careers. Average participant age has steadily decreased over time.

0 10 20 30 40 50

70‐74 65‐69 60‐64 55‐59 50‐54 45‐49 40‐44 35‐39 30‐34 25‐29 20‐24

Age Range of Participants

Number of Respondents

Percentage

14

Women constituted the majority of respondents, being more than twice as

likely as men to participate in the program.

Each Commonwealth Seminar class includes a diverse mix of racial/ethnic

groups. Respondents who identify as black, white, and Hispanic each comprised a

quarter of respondents. Asians and individuals who identify as mixed‐race

comprised the remainder quarter.

Male 32%

Female 68%

Male

Female

American Indian 1%

Asian 13%

Black 28%

White 23%

Hispanic 25%

Mixed 10% American Indian

Asian

Black

White

Hispanic

Mixed

15

Additionally, respondents hail from a variety of occupational backgrounds,

ranging from non‐profits to government to business. Given the Commonwealth

Seminar’s emphasis on public service, it is worth noting that a significant

proportion of respondents work in the non‐profit sector.

Finally, regardless of their occupation, the majority of respondents are

involved with non‐profit organizations through their leadership activities.

0 5 10 15 20 25 30 35

Business or Professional

Education, Research, or News

Retired

Student

Government

Non‐Pro2it

Unemployed

Number of Participants

Occupation


3%

Education, Research, or

News 4%

Student 8%

Government 20%

Non‐Pro2it 65%


Education, Research, or News

Student

Government

Non‐Pro2it

16

Recruitment

The Commonwealth Seminar employs two distinct strategies to recruit prospective

participants. The first strategy is recruiting through informal, word‐of‐mouth

methods. An example of this type of method is hearing about the program from a

work colleague who encourages one to apply. The second strategy is recruiting

through formal, non‐word‐of‐mouth methods. An example of this type of method

is receiving an email from one’s university that advertises the program. In other

words, formal methods of recruitment involve direct outreach efforts from the

Commonwealth Seminar itself, while informal methods rely on Seminar graduates

or affiliates to pass the word along.

The majority of respondents (72%) report that they first became aware of

the Commonwealth Seminar via informal, word‐of‐mouth means. Among this

group of respondents, the majority (55%) learned about the program from a

Commonwealth Seminar alumnus or from executive director Joel Barrera himself,

while 45% learned about the program from a friend, work colleague, boss/mentor,

or through a chance encounter with someone familiar with the program. The

remainder of the respondents (28%) report that they first learned about the

Commonwealth Seminar through more formal means, including their involvement

in a particular organization or university, a forwarded e‐mail, or an Internet

search.

17

The Seminar’s current emphasis on informal, word‐of‐mouth recruiting

tactics has both positive and negative effects. A major benefit is that this method

of recruitment allows Commonwealth Seminar affiliates to refer the most

appropriate and interested individuals to the program. Program alumni know the

program well; they have a good sense of who would be a good fit for the program

and can target recruitment to ideal candidates. However, the potential downside

of relying on informal recruitment is that it also limits the pool of potential

applicants to particular networks, which may lead the Seminar to overlook leaders

who could potentially benefit from the program.

Summary and Recommendations

We recognize that a large part of the Commonwealth Seminar’s success in

assembling an engaged and talented group of leaders is due to its well‐functioning

system of informal recruitment. Existing channels of recruitment should be

maintained. However, we advocate combining the current informal, word‐of‐

mouth methods with a greater emphasis on formal tactics, including

organizational partnerships and more aggressive use of social media. We explain

this recommendation in greater detail in the Conclusion & Recommendations

section.

18

Legislative Training Seminar

Background

The Commonwealth Seminar’s

legislative training seminar is an

intensive six‐week program held at

the Massachusetts State House. The

explicit goal of the legislative training

seminar is to demystify the state

legislative process by educating

participants about the process and all

that it entails. An implicit goal of this

program aspect is to increase

participants’ comfort and confidence securing resources and advocating for the

organizations and communities they represent.

There are three primary mechanisms through which the legislative training

seminar achieves its goals and enables the Commonwealth Seminar to achieve its

mission of “opening the doors of the State House” to diverse leaders: (1) The

seminar covers a range of topics designed to enhance familiarity with all aspects of

the legislative process, (2) it brings in relevant political leaders and policymakers

to teach particular topics, (3) it holds seminars at the Massachusetts State House.

19

These factors work in tandem to provide participants with the skills,

knowledge, and confidence necessary to make the most of their positions as

community leaders and advocates.

Topic Usefulness

The overwhelming majority of respondents found all of the topics covered

during the legislative training program to be useful or extremely useful.

Topics covered include: the legislative process (94% of respondents

considered this topic useful), the budget process (95%), media relations

(87%), lobbying (89%), and the importance of and strategies for networking

(87%).

Respondents report that coverage of these topics equipped

participants with the tools they need to speak informatively about public

issues, understand the respective roles of different branches of government,

and to advocate effectively for their causes:

I feel like I have a much better understanding of how things work right, how the process works, how the cycles are, how the stuff like the budget works, and during the election seasons and when people have talking points and stuff like that, you understand how certain things come about. And then I feel like I could pick up on the stuff that [is] just false. A lot of people, even myself, really didn’t understand how the local government worked with the federal government, how much power the government locally has and how much it affects your day‐to‐day life because you tend to think…a lot of things affect your life but it’s like the littlest things like how many street lights are in your city, do you get trash pickup, what’s it like, what does your community look like and it’s really affected by the local government. So I think it kind of put everything into perspective for me.

20

Policymakers

Commonwealth Seminar participants appreciated the fact that experts,

including legislators and policymakers, were brought in to teach each

seminar topic. Participants felt that they benefited from learning about how

the legislative process works in practice from the people who deal with

aspects of this process every day, as the following interview quotes

elucidate:

[The legislative training] was very relevant because I ended up lobbying statehouse legislators… We got introduced to some legislators, so before I even started my job, I had relationships with some legislators because I got to meet them through the Commonwealth Seminar. So I would say…for someone who ended up lobbying, it prepared me very, very well and I know that my boss and I both were pretty much like this is a good program and we made relationships out of that that otherwise we wouldn’t have.

I remember how important that was to learn firsthand from a legislator who was passionate about [the seminar topic], who was in that context where he or she wasn’t running for office or like wasn’t put in the spotlight in the media where they can talk honestly about the structures and how the government can do better in spending taxpayer money and so I think that was just really beneficial.

I would say [the most useful takeaway from the program was] being able to hear firsthand…anecdotes from legislators. You know, really hearing about how things really work, how they were successful, how they have been successful in their careers and from people that are advocates that have worked with legislature in order to advance whatever their agenda may be.

Respondents cited another benefit to the presence of political

leaders and policymakers at the seminars: their presence at the seminars

21

signaled that these policymakers were accessible to them and considered

their needs. The following comment from an interview respondent echoes

this common perspective:

I think one of the things that the training imparted around the legislative process was how accessible legislators were. And so it made Beacon Hill a more accessible place that was very demystified, the process was demystified, and I think that that’s what helped people who tend to be disenfranchised from the legislative process feel like they have more access.

The presence of policymakers was particularly powerful when these

speakers were themselves minority leaders who rose to success in the

Massachusetts political arena and now exert substantial influence on local

policy. These speakers are living success stories that reinforce the

aspirations of participants who want to run for elected office someday, as

interview respondents explain:

A majority of folks were legislators and they were legislators of color, which for me was great because like I said I would love to run for office one day and…I remember…a representative and she came to the group and she is a woman of color and she told her story of how she started working as an aide and then ran for office and she’s been in office now and she’s a mom so she’s juggled all of these different roles and it really made me feel like well I can do it too. [A state rep.] actually gave me inspiration in terms of we coming from much more lower income background and family, having the ability to sort of, you know having that dream that says, “Oh I can be, I can run from public office, without loads of money” and it just take small grassroots approach. And when he offered himself to sort of help me in that situation, I felt like that was the biggest uplifting moment (Male, Asian).

22

The State House

The Commonwealth Seminar literally opens the State House doors for

participants by holding its seminars at the Massachusetts State House. The

benefit of holding seminars at the State House is that participants become

more comfortable being in a location that most people find intimidating.

The State House is the symbolic seat of Massachusetts state government; if

participants are to leverage their new knowledge about the legislative

process to achieve their advocacy goals, they must feel comfortable

operating in this location. The Commonwealth Seminar wants participants

to feel a sense of ownership in the State House, and it works to help

participants feel comfortable accessing the State House’s resources. Many

respondents cited “familiarity with the State House” as one of the most

important benefits of participation in the Commonwealth Seminar; the

following comment is representative of the many others who share this

sentiment:

The idea that…the State House, it's my house as well and that my voice counts and that I should be able to feel comfortable going there and talking to people there even though they don’t look like me, I think that was the most important thing.

23

Satisfaction: Legislative Training

The legislative training seminar is one of the most effective aspects of the

Commonwealth Seminar, as evidenced by the high level of respondent satisfaction

and the positive outcomes associated with participation.

The vast majority (86%) of respondents indicated that they were satisfied or

extremely satisfied with the legislative training program. On average, participants

were more satisfied with legislative training than with any other aspect of the

Commonwealth Seminar. Respondent satisfaction primarily stemmed from the

perceived benefits associated with information covered during the seminar,

participants’ ability to learn directly from policymakers, and participants’

increased comfort in navigating the State House.

Satisfaction with the legislative training seminar varied with respondent

gender and race/ethnicity. The average woman was significantly more satisfied

with the legislative training seminar than was the average man. Compared to

other racial/ethnic groups, Asian individuals were significantly less satisfied with

24

this aspect of the program. Despite lower average satisfaction among Asian

respondents, it is worth noting that the majority (75%) of Asian respondents

reported that they were either satisfied or extremely satisfied with the legislative

training program. Satisfaction with the legislative training program did not vary by

respondent occupation, leadership activity, age, or year of graduation.

Benefits and Outcomes

Commonwealth Seminar graduates have clearly benefited from their participation

in the legislative training seminar. Respondents experienced positive outcomes

that are in line with the legislative training seminar’s goal of increasing participant

knowledge and comfort with advocating for their organizations and communities

through effective navigation of the state legislature. The majority (61%) of

respondents have contacted a Massachusetts political representative or

policymaker since graduating from the seminar, and a full (77%) of respondents

agree that they have become better advocates for their communities.


Although most respondents indicated that they were very satisfied with

their legislative training seminar experience, they also believe that there is room

for improvement in this aspect of the program. We tested a list of possible

recommendations and found that at least half of respondents agreed with the

suggestions that there should be more interactive activities during the seminar,

more time for student discussion, and handouts during seminar presentations.

Based on these findings, we provide two recommendations. The first

25

recommendation is to incorporate more interactive activities into the seminar.

Two thirds (66%) of participants agreed that the Commonwealth Seminar should

include more interactive activities. We suggest two ways that the seminar might

incorporate more interactive activities. First, provide more time for students to

discuss topics during the seminar, a recommendation that was supported by 53%

of respondents. The following comment from an interview respondent reveals why

some respondents were eager for more discussion:

I feel like a lot of it was lecturing. It would be interesting to see how it would work out if they brought in a speaker and then you were split up kind of like in workshops, because a lot of it felt like lecturing and then when people got really into it, it was like we weren’t fitting together...But that’s just the only thing that I could think of structurally…I would like to see more discussion amongst the class and not just only of the class to the speaker.

Second, given the symbolic importance of holding the seminars at the State

House, the Commonwealth Seminar might consider including more activities

designed to promote greater immersion in the State House and the events that

take place there, such as tours or legislative meetings. The following comment

from an interview respondent illustrates the reasons why respondents would

support more of these kinds of activities:

The only hindering thing is that we were in the same room for the entire class and we didn’t go anywhere else except during that tour to see the chamber, to see certain things… I thought it was a great idea being there and learning about the [State House]. I think that it has a big effect when we learn about [the government] sitting in the building... So, I thought that was a big plus, and [so that] we can make the most of our time being there, why not maybe divide up a little bit and have us sit in certain sessions that might be more intriguing, and maybe give us a little more of in‐depth look at a certain topic?

26

Our second recommendation is to ask all seminar presenters to provide

handouts. An overwhelming majority (71%) of respondents agreed that the

Commonwealth Seminar should provide students with handouts during

presentations. Seminar participants clearly find the information presented during

the seminar to be useful, but it can also be a lot to digest in a single setting.

Handouts make it easier to recall and reference important information after the

seminar ends. It is worth noting that we received handouts during our visits to the

seminar, which suggests that participants receive handouts at least some of the

time. However, given that so many respondents are eager to receive handouts, we

recommend stressing the importance of providing relevant handouts to all seminar

presenters.

In summary, we believe that the legislative training seminar fulfills its

objectives of educating and empowering participants. The following comment

illustrates the typical respondent attitude towards this aspect of the program:

I think [the most useful takeaway from the legislative training seminar] is the power. It's the message that folks like me need to be more part of political process and I think that that is really the mission of the Commonwealth Seminar to bring people of color to the statehouse, teach them what goes on in the building and let them realize that they can have a stake in what goes on in there like not this world that’s so far removed and where complicated issues take place. You as a citizen, as a stakeholder working in whatever field it is that you are working in, you have a place here and you should be engaged in what's going on here… So I think that that’s the overall mission of the Commonwealth Seminar and it does a good job of really opening the doors and telling people this is your house too as the governor says. This is your house and what goes on in here affects you and you should know and you should be involved.

27

Minor critiques aside, the legislative training seminar should continue to do

exactly what it has been doing.

Networking

Background

The legislative training

seminar increases

participants’ ability to

serve their communities

by information about the

legislative process.

However, in order to effectively advocate for one’s community and secure valuable

resources, whom one knows is often as important as what one knows. In

recognition of this political reality, the Commonwealth Seminar also encourages

participants to forge relationships or professional connections with their fellow

participants and with policymakers and provides opportunities to do so.

We use two criteria to evaluate the effectiveness of the networking aspect of

the Commonwealth Seminar: (1) reported satisfaction with current networking

opportunities and (2) whether connections formed through the Commonwealth

Seminar have lead to beneficial outcomes for participants. Do connections made at

the Commonwealth Seminar actually lead to further collaboration after the

28

seminar? Have these connections increased participants’ access to valuable

resources, information, or career opportunities?

After a brief discussion of satisfaction with this aspect of the

Commonwealth Seminar, we discuss the Seminar’s facilitation of two types of

networking in greater detail: networking with seminar participants (i.e. fellow

classmates and Commonwealth Seminar alumni) and networking with

policymakers. For each type of networking, we evaluate and explain the following:

(1) the methods used to facilitate relationships and connections, (2) respondent

satisfaction, (3) the ways in which seminar participants have benefited from

existing methods, and (4) program recommendations.

Networking Overall

Overall, the majority of respondents were pleased with the networking aspect of

the program: 61% of respondents indicated that they were satisfied or extremely

satisfied with the seminar’s provision of networking opportunities. However, this

is also the aspect of the program with the highest percentage of unsatisfied

respondents (10%).

One notable finding that emerged from the interview data is that many

respondents appreciated the information the Seminar provided about the

importance of networking, not just the provision of networking opportunities. A

key theme of the legislative training seminar is the connection between strong

networks and effective advocacy or career advancement. Respondents explained

29

that they were able to put these lessons learned about networking to good use, as

the following interview quotes reveal:

[A]fter the Commonwealth Seminar I realized that you have to network… you have to be able to engage [people] and they have to be able to speak on your behalf and advocate for you…. [T]he Commonwealth Seminar opened that door for me and allowed me to showcase my skills which then somebody else caught sight of that and allowed me to have my second job (Female, Hispanic).

The power of networking is what I learned, how to effectively put your point across to be heard... To me, the most [important] part [of the Commonwealth Seminar] was networking, which helps you in whatever you do. If you have an organization and you need something to pass, you have to know the right people, you have to do the right thing, know who to target to get something done, that was just worth it (Female, African American).

Networking with Seminar Participants

The Commonwealth Seminar uses two strategies to facilitate connections among

Seminar participants: (1) informal networking opportunities built into the seminar

itself and (2) formal events during and after the seminar. First, unstructured time,

including time during morning refreshments and during breaks between guest

speakers, is built into the seminar schedule. As one respondent describes, these

breaks serve an important function, as participants use this unstructured time to

chat and get to know one another:

[B]efore, after and during [the seminar], there would be like a chance to get coffee or tea or a bagel so people started talking. And then you would have class, and then there would be a ten‐minute break or whatever. And people would funnel together and then you would walk out and you would still be talking about this that and the other thing, oh yeah (Female, Hispanic).

30

Second, the Commonwealth Seminar uses formal events to facilitate

connections between seminar participants. Formal events include reunions,

discussion events (i.e. events framed around a particular policy issue or program),

formal celebrations, and graduations. Respondents who have attended formal

events indicated that they have enjoyed them, particularly because they provide an

invaluable opportunity to network with Commonwealth Seminar alumni and

recent graduates. It is clear that respondents recognize the unique networking

opportunity Commonwealth Seminar‐sponsored events afford to program

participants, as the following interview quote conveys:

[P]eople do keep in touch and people do come back… during the graduations, during a lot other things that Joel has… if it wasn’t for the Commonwealth Seminar, there [would not] be those kind of invitations, there [would not] be that kind of knowledge or that kind of exposure to those things around Boston (Male, Asian).

In short, respondents enjoy informal opportunities to chat with their

classmates during the seminar, and most report that they are quite satisfied with

the nature and content of formal Commonwealth Seminar‐sponsored events.

Respondents’ dissatisfaction with opportunities to network with Seminar

classmates and alumni is less a function of how the seminar facilitates networking,

and more a function of how often the Seminar provides such opportunities. We

discuss this finding in greater detail in the next section.

31

Satisfaction: Networking with Participants

Respondents were the least satisfied with this aspect of the Commonwealth

Seminar. Only 34% of respondents indicated that they were satisfied with existing

opportunities to network with Seminar classmates (33% were unsatisfied), and

only 29% were satisfied with opportunities to network with Seminar alumni (39%

were unsatisfied).

The program currently relies on unstructured time built into the seminar as

the primary mechanism to facilitate networking among classmates. As noted in the

previous section, respondents enjoy these informal networking opportunities.

However, only 16% of respondents agreed with the statement that current

methods for promoting networking among Seminar participants should not be

changed, which suggests that there is room for improvement in this respect. The

primary source of respondent dissatisfaction is a lack of available networking time:

many respondents felt that the time constraints of the seminar made it difficult to

establish sufficient new connections. Essentially, participants both recognize the

value of networking with classmates and desire more opportunities to do so, as the

following respondent explains:

I think the networking notion is really good. We got an opportunity to see all the people who’s come from different walks of lives and different things. And we have emailed each other and exchanged things and I think that is one good thing that I got out of it…. I see that they are giving us some time to just socialize a little bit here and there but because the amount of time is too short, [the seminar] is two hours [long] and we have speakers coming in (Male, Asian).

32

Satisfaction with the Commonwealth Seminar’s provision of opportunities

to network with classmates and alumni varied with respondent gender, age, and

graduation year. The average female respondent was slightly more satisfied with

opportunities to network with participants than the average male respondent.

Older individuals were also more satisfied with opportunities to network with

Seminar classmates. Finally, the more recently the respondent had graduated from

the Commonwealth Seminar, the less satisfied they were with opportunities to

network with classmates or alumni. Satisfaction did not vary by occupation or

leadership activity.


Most respondents

established between 1 and

3 new professional

connections during the

Commonwealth Seminar,

and 56% of respondents

report that they have kept

in touch with 1 to 3 people they met through the Commonwealth Seminar. Our

findings suggest that respondents who developed relationships through the

Commonwealth Seminar were able to leverage them to access various

opportunities that would not have been otherwise available. A third (32%) of

respondents indicated that they had collaborated on a work‐related project with

33

someone they met through the seminar; a significant proportion (43%) learned

about or obtained resources for their organization via someone they met through

the seminar; and another 43% learned about a job opportunity from someone they

met during the seminar. The following comments from an interview respondent

illustrate how connections formed during the Commonwealth Seminar can prove

professionally useful after the seminar ends:

Well for instance, [at my firm], where my role was to make grants to nonprofit organizations in specific areas... One of [my] colleagues from Commonwealth Seminar is the executive director of one of these organizations that I was considering for a grant. So because we had that connection to the Commonwealth Seminar, you know it was easy to sort of foster a relationship there (Male, White).


The Commonwealth Seminar facilitates networking among participants by

providing informal opportunities (e.g., breaks, morning refreshments) and formal

events (e.g., graduations, discussion events). Respondents explained that they

appreciated and benefited from these opportunities to establish relationships with

their fellow program participants and alumni. However, this is also the aspect of

the Commonwealth Seminar with which respondents were least satisfied. Perhaps

unsurprisingly, respondents who developed more network connections during the

seminar were significantly more satisfied with this aspect of the program and with

the Commonwealth Seminar overall.

Based on survey and interview responses, we propose two

recommendations for improving existing opportunities for networking with

34

program participants: (1) increase the number of formal networking events during

the seminar and (2) provide additional avenues for communication between

Seminar classmates and alumni. The overwhelming majority of respondents (81%)

agreed that they would like the Commonwealth Seminar to hold more formal

networking events and activities. Further, most (75%) of respondents agreed that

the program should do more to facilitate communication among Commonwealth

Seminar alumni.

Additional formal networking events during the seminar would address a

key source of respondent dissatisfaction. As one interview respondent explains, it

can be hard to establish enough new connections without designed networking

time during the seminar:

[T]here wasn’t really a lot of formal time for people to… build those networks. [I]f the program doesn’t have anything formal in terms of… fostering those relationship… then naturally we just want to get what we want to get and then we go back to our own silos and work (Male, Asian). We also believe that exploring additional methods of increasing communication among Seminar alumni would increase participant satisfaction. We suggest two ways to encourage interaction between program participants. First, the program might increase the number of formal events, as these events provide opportunities for individuals to meet members of other Seminar classes and develop ties with them. Holding more of these events and holding some of them on weekends may make it easier for people with busy schedules to attend, as one respondent suggests: I mentioned the social events that he puts together. I think they are great, [but] I think it’s so unfortunate that I can never go due to work… [H]e needs to bank more on weekends (Female, Hispanic).

35

Second, the program should take advantage of its presence on social media

websites like as Facebook, LinkedIn, and Twitter. Encouraging more seminar

graduates to “like” the Commonwealth Seminar’s Facebook page provides an easy

and convenient way for participants to obtain each other’s updated contact

information.

Networking with policymakers

The Commonwealth Seminar helps participants connect with policymakers in two

ways: policymakers attend the seminar, which gives participants an opportunity to

meet and interact with them and the seminar’s location in the Massachusetts State

House also brings participants to policymakers. The program invites policymakers

from many levels of government to be guest lecturers. The presence of

policymakers at the seminar allows participants to develop connections that may

prove professionally useful after the seminar ends, as one respondent describes:

There were a lot of people who are speakers for different workshops, I mean Sonia Chang‐Diaz, I think she was running back then [for state senator]. So she came and talked to us…. So from meeting her at the Commonwealth Seminar she is on the board here [where I work] and we have a good relationship because I was in the Commonwealth Seminar... (Female, Hispanic).

Satisfaction: Networking with Policymakers

More individuals were satisfied with opportunities to network with policymakers

than with opportunities to network with fellow participants. Of those surveyed,

80% were moderately to extremely satisfied with this aspect of the program, while

20% of individuals indicated that they were unsatisfied.

36

Respondent satisfaction with this portion of the program stemmed from the

perception that they had established connections to individuals they had

previously viewed as inaccessible through their participation in the

Commonwealth Seminar. Relationships with legislators proved professionally

useful and enabled participants’ advocacy efforts, as the following quote from an

interview respondent illustrates:

We got introduced to some legislators. So before I even started my job, I had relationships with some legislators because I got to meet them through the Commonwealth Seminar. So I would say yes for me for someone who ended up lobbying, it prepared me very‐very well and I know that my boss and I both were pretty much like this is a good program and we made relationships out of that that otherwise we wouldn’t have (Female, Hispanic).

As noted in the legislative training section, respondents also indicated that

the relationships with policymakers that they developed during the seminar also

made them feel more comfortable reaching out to legislators after the seminar

ended. Satisfaction with opportunities to network with policymakers did not vary

by gender, age, race/ethnicity, graduation year, occupation, or leadership activity.


As mentioned in the legislative training seminar section, respondents who formed

relationships with legislators felt that these connections increased their ability to

access resources in state government and increased their confidence in

approaching elected officials. Another important benefit of networking with

37

policymakers was that interactions with diverse policymakers increased the

perceived viability of a career in public service.


The Commonwealth Seminar gives participants opportunities to interact with

policymakers after guest lectures and around the State House. Overall,

respondents were satisfied with the Seminar’s facilitation of this aspect of

networking. Satisfaction stemmed from the program’s ability to provide access to

policymakers they could have not met otherwise. The evidence suggests that this

portion of the program works well; however, we believe that it could be even

better. We provide two recommendations: (1) continue to invite a diverse group of

policymakers to speak and (2) provide more opportunities to interact with

policymakers.

Inviting policymakers to speak at the seminar provides one effective

strategy for facilitating connections between participants and policymakers, and

this practice should be continued or even increased. Furthermore, respondents

clearly felt empowered after meeting policymakers who were similar to them;

therefore, the seminar should continue to invite a diverse group of policymakers.

However, providing opportunities to interact with policymakers beyond guest

lectures may increase participant satisfaction. Respondents indicated that they

particularly valued those interactions with policymakers that went beyond brief

conversations after guest lectures. For example, something as simple as an

interactive State House tour (led by a policymaker) can really enhance the

38

participant experience, as the following quote from an interview respondent

reveals:

[B]ecause this particular state rep… literally showed us the ropes, and so it was just engaging. [I]t just made me feel more comfortable… and… inspired because we get more time to actually get to know this particular person, because sometimes it's not about what they do, it's really about who they are that we identify with (Male, Asian).

Job Bank

Background

The online job bank was created in 2005 in order to provide easy‐to‐access

information about public service job opportunities. This initiative encourages

diverse leaders to consider a career in public service, and it serves as a useful

resource for past and present Seminar participants who are searching for new

public service job opportunities.

Respondents identified two important characteristics that made the job

bank a particularly useful resource: (1) it is a centralized listing, available to the

public on the Commonwealth Seminar webpage and (2) it is consistently

advertised and linked to via the Commonwealth Seminar newsletter. The majority

(71%) of survey respondents were aware of the job bank, and most (57%) of these

respondents had used the job bank in some way. Evidently, the program has been

largely effective in spreading the word about this resource and the majority of

individuals who know about it have used it.

39

Satisfaction: Job Bank

Of those aware of the job bank, the majority (69%) was satisfied or extremely

satisfied with this aspect of the program. Only a tiny fraction of respondents (2%)

were unsatisfied with the job bank. Compared to respondents from other

occupations, students were less likely to be aware of the job bank, but satisfaction

with this resource did not vary by age, gender, occupation, or other respondent

characteristics.

Respondent satisfaction with the job bank stems from its centralized nature

and the quality of the opportunities that are posted. The job bank’s centrality is its

defining feature and key to making the resource both more accessible and useful.

One respondent explains why this feature of the job bank made this an effective

resource for her:

Well I thought [the job bank] was a great idea and I thought, I appreciated having in one place a good selection of available positions especially that there was a sense that people of color were invited to apply was, it was great place for folks to go to look at jobs that were available and accessible and I just appreciated having that in one place (Female, African American).

Other respondents explained that the job bank is an especially useful

resource because the reputation of the program and its executive director added

credibility to the opportunities posted on the job bank. In other words, the job

bank is a valuable resource due to the perceived quality of the opportunities that

are likely to be posted there. Comments from one respondent illustrate this

commonly‐held sentiment:

40

I do know that it’s a good list and it’s always a list with companies that are very reputable and companies that value diversity so obviously the Commonwealth seminar doesn’t just put any job on there… So it's always going to be a good job and it’s always going to be with a good company because I think Joel does a good job of having those relationships and filtering organizations. So I don’t think it’s just any job that you would find on Craigslist or in the paper or anything like that (Female, African American).


Participants use the job bank in a variety of ways, including finding jobs and

passing on information to others about available opportunities. In fact, of those

that used the job bank, a third (34%) told a friend or professional contact about an

opportunity listed on the job bank. Four percent of respondents had applied for a

job and received a job offer, while 13% had applied for a listed job, but did not

receive an offer. 20% of respondents regularly browsed the resource, but had not

applied for any of the opportunities listed.

Interview evidence suggests that the job bank is particularly useful for

participants who are interested in transitioning into a career in public service. One

respondent explains how the job bank helped her to accomplish this goal:

[The job bank is] hugely useful especially in my case it helped because my undergrad [degree] is in finance and I was in the business world for years and then I just wanted to take all that to the non‐profit. It helps your self‐esteem, it helps you feel totally comfortable approaching someone saying “Hey maybe I can go with this at a different angle” (Female, White).

41


The job bank is a valuable resource. The majority of respondents who were aware

of the job bank were also satisfied with it. However, 29% of survey respondents

were not aware of its existence, which suggests that program could still do more to

advertise this important resource.

Promoting the job bank on the Commonwealth Seminar Facebook group

provides another way to reach individuals and remind them about the resource.

Informing people about the resource during formal events would also help to

increase awareness. We recommend that the program continue to maintain this

resource in its current form. The Commonwealth Seminar might also encourage

more participants to use the resource by publicizing it more aggressively, both

online and at events.

Conclusion & Recommendations

Overall, the Commonwealth Seminar has done an excellent job of fulfilling its

mission of opening the doors of the State House to diverse leaders. However, we

believe this program can be even better. In this section, we provide three major

program recommendations.

Recommendation 1: Improve Outreach

We recommend that the Commonwealth Seminar focus on implementing new

strategies for extending its outreach, including the use of formal recruitment

42

tactics to reach a wider array of prospective candidates and strengthening the

alumni network. The Commonwealth Seminar relies primarily on informal, word‐

of‐mouth methods to recruit candidates. The potential downside to this method is

that it may limit the pool of potential applicants to particular networks and miss

leaders who could potentially benefit from the program. We believe that the

Seminar should supplement these informal tactics with a more concerted formal

recruitment strategy. We propose two specific formal recruitment tactics that

would allow the seminar to reach more diverse leaders.

First, advertise the application timeline on the Commonwealth Seminar

website. If possible, determine application deadlines six months to a year in

advance and post this information clearly in an easy‐to‐find location. Second, work

on building formal partnerships with various community organizations, non‐

profits, and universities. Using such partnerships, not just Commonwealth

Seminar graduates and affiliates, for recruiting would have two major benefits.

First, organizations have access to a larger number of people, and using

organizations to advertise the program would help the program reach leaders who

may fall outside of existing Commonwealth Seminar networks. Second, many

organizations have established communication channels like listservs, websites, or

newsletters that could be used to quickly and easily advertise the program.

Establishing relationships with area organizations would potentially be an effective

way to reach out to additional qualified applicants.

43

Additionally, we believe that the Commonwealth Seminar will benefit from

strengthening alumni networks. This should help the Seminar maintain the

community of diverse leaders that it has so successfully created. We propose two

specific ways the Commonwealth Seminar can strengthen its alumni networks and

thereby improve its outreach to program graduates. First, actively leverage social

media channels (e.g., Facebook, LinkedIn, Twitter) to maintain contact with and

facilitate communications among program graduates. Second, organize and host

more formal events (e.g., alumni receptions and reunions) to provide program

graduates with a convenient opportunity to reconnect with one another.

Recommendation 2: Formalize Networking

Networking is a program objective that the vast majority (74%) of respondents

consider very important. The Commonwealth Seminar currently provides both

informal networking opportunities during the seminar and formal events designed

to facilitate networking among classmates and alumni. We believe that increasing

the number of formal events, both during and after the seminar, will increase

satisfaction with this aspect of the program.

During the seminar, the program should devote more formal, structured

time to networking. As it stands, some participants feel that insufficient time is set

aside for networking during the seminar. Formal networking events during the

seminar would enable all participants, including more reticent individuals, to

develop more relationships. The program should also continue to hold formal

events after the seminar, as these events allow Commonwealth Seminar alumni to

44

reunite with their classmates and meet participants they may not have previously

encountered. Holding some of these events on weekends may yield a larger pool of

participants.

Recommendation 3: Ensuring Continued Success

The Commonwealth Seminar addresses an important and enduring gap in political

access and representation. All available evidence suggests that it has truly

changed the lives of many program participants. Given this remarkable success, we

hope that the Commonwealth Seminar program will expand to serve even more

diverse leaders in the future. Furthermore, we believe that implementing the

above recommendations will address sources of respondent dissatisfaction without

affecting features of the program that contribute to the current high level of

overall satisfaction.

However, we recognize that improving outreach and formalizing

networking will require additional time and effort on the part of the

Commonwealth Seminar leadership. These resources are in short supply.

Therefore, our final recommendation is to enhance the sustainability of the

Commonwealth Seminar by spreading leadership tasks across a number of

additional positions. Creating an enduring structure for the management and

oversight of the Commonwealth Seminar would reduce the burden on any single

individual and would enable expansion and growth. Evidence from the interviews,

the survey, and our personal observations indicates that the success of the

Commonwealth Seminar is, in large part, a function of the tireless efforts of its

45

current leadership team. Almost every respondent spoke in glowing terms about

the current Seminar leadership, particularly executive director Joel Barrera.

Indeed, it is unlikely that the Seminar will be able to find a similarly committed

and dedicated individual to replace Mr. Barrera when his tenure as executive

director ends. For this reason, we believe that if the Commonwealth Seminar is to

continue to operate long‐term, leadership tasks must be allocated across multiple

positions.

We recognize that the Commonwealth Seminar is currently a small

program, and that it is limited by both time and funding constraints. Therefore, as

a first step, we propose that the Seminar work towards securing funding to hire a

part‐time outreach and alumni relations coordinator. This individual's

responsibilities might include reaching out to area organizations to advertise the

seminar, organizing and advertising formal networking events, and strengthening

alumni relations through frequent updates to social media sites, soliciting career

information and updates from alumni, and maintaining the alumni contact

information database. Finally, a number of interview respondents mentioned that

they were willing and eager to “give back” to the seminar in some way; this part‐

time employee could also be responsible for coordinating alumni volunteer

initiatives.

In short, we are amazed by what this small‐scale program has been able to

accomplish since its inception. The Commonwealth Seminar addresses an

46

important community need, and we hope that this program will continue to

benefit diverse leaders in Massachusetts for years to come.

Appendix

Table 1: Network Ties Established During the Commonwealth Seminar

Number of New Ties Number of Respondents Percent of Respondents

0 7 4.07

Between 1 and 3 74 43.02



10 10 5.81

More than 10 10 5.81

Did not answer question 9 5.23

Total 172 100.00

Table 2: Network Ties Maintained After the Seminar

Ties Maintained Number of Respondents Percent of Respondents

0 46 26.74


47



10 3 1.74

More than 10 1 0.58

Did not answer question 9 5.23

Total 172 100.00

Exhibit A: Interview Questionnaire

RESPONDENT BACKGROUND

Let’s begin with a couple of questions about you.

1. Since the Commonwealth Seminar targets leaders, why don’t you start by telling

me about your leadership activities?

2. Is this your full time job?

IF NO: Tell me about what you do. What is your current job?

3. Tell me a little bit about your career.

a. Probe: Have you always been at (current job)?

GENERAL INVOLVEMENT WITH COMMONWEALTH SEMINAR

I have a few general questions about your participation in the Commonwealth

Seminar

48

4. What year did you graduate from the Commonwealth Seminar?

5. Tell me about how you first found out about this program.

6. What made you decide to apply?

a. Probe: At the time, what did you expect to gain from participating in the

seminar?

7. What was the single most useful thing you took away from the Commonwealth

Seminar?

8. In your opinion, what’s the number one thing the Commonwealth Seminar

could do to improve?

a. Probe: Why?

LEGISLATIVE TRAINING

9. Was the information covered in the six‐week legislative training seminar useful

for you?

IF YES: How so?

a. Probe: Give me an example of something that you learned from the seminar that

proved useful later on.

IF NO: Why not?

10. The Commonwealth Seminar application asked applicants to identify topics or

issues that they are interested in learning more about. How well did the seminar

address the issues that you were initially interested in?

49

NETWORKING

11. Did you make any friends or professional connections during the

Commonwealth Seminar?

IF YES: Tell me a little bit about them.

Probe: Were you able to meet with any policymakers? What did you talk about

with them?

12. Did the Commonwealth Seminar do anything to foster relationships among

participants, or did this just happen naturally?

13. Have you been in touch with any of these people since you graduated?

IF YES: What do you guys discuss?

IF NO: Why not?

NETWORKING BENEFITS

14. Have the connections you made through the Commonwealth Seminar

improved your ability to advocate for your community on particular issues?

IF YES: Probe: Give me some examples.

15. Have you collaborated with any Commonwealth Seminar graduates on specific

projects?

50

16. What about you, personally? Have your Commonwealth Seminar connections

improved your access to jobs, resources, or other professional opportunities?

CAREER TRAJECTORY

17. Have your career goals changed since you participated in the Commonwealth

Seminar?

IF YES: How so?

18. Are you aware of the Commonwealth Seminar‐sponsored job bank, posted on

the Commonwealth Seminar website?

IF YES: What’s your impression of the job bank?

OVERARCHING

19. Do you have a sense of how the Commonwealth Seminar has changed over

time?

Interviewer: This is the end of our formal interview, but I just wanted to give you a

chance to tell us anything else that might help us to better understand your

experience with and impressions of the Commonwealth Seminar.

Exhibit B: Survey Questionnaire

[See attached .pdf file]

program evaluation of the commonwealth seminar · program evaluation of the commonwealth seminar...

Documents