evaluating the alaska epscor phase iii: resilience and vulnerability in a rapidly changing north:...

Evaluating the Alaska EPSCoR Phase III: Resilience and Vulnerability in a Rapidly Changing North: The Integration of Physical, Biological and Social Processes

Dr. Julia Melkers, Associate Professor of Public PolicyGeorgia Institute of Technology

Dr. Eric Welch, Associate Professor of Public AdministrationUniversity of Illinois at Chicago

Year 3 Evaluation PresentationAlaska EPSCoR All Hands Meeting

May 2010Fairbanks, Alaska

Stating the Obvious Collaboration is difficult, problems of:

disciplinary language, distances, funding, methodological, conceptual and theoretical approaches lack of clarity about why research questions are

“important”. Requires commitment to learn from other fields Variation in goals and interests of stakeholders and rural

campuses Institutional barriers and facilitators are key to success

Three Year Evaluation of Alaska EPSCoR Phase III A formative and interactive approach.

Can also provide lessons learned for future EPSCoR proposals and initiatives.

An summative evaluation process to serve as an accountability mechanism.

A research driven evaluation, based on studies of science, collaboration, networks, among others.

Year 1 Year 2 Year 3 What are the

foundational capacities of AK EPSCoR faculty and students?

What foundational relationships exist?

What early outcomes can be observed?

How are collaborative interactions and related outputs developing in AK EPSCoR?

How integrative is AK EPSCoR and what are the related barriers?

What benefits are EPSCoR students experiencing?

How are collaborative interactions and related outcomes continuing to develop among students and faculty?

How integrative are these interactions and outcomes?

How are rural campuses and native communities benefiting from EPSCoR? What are the key issues in this?

How are institutional and administrative factors relevant to the ability of AK EPSCoR to meet its goals?

Evaluation Questions

Year 1 Year 2 Year 3Evaluation Focus

Collaboration Familiarity Interdisciplinary

orientation

Collaboration Productivity Factors affecting

productivity Integration Student impacts

Continuation of year 2 issues

Collaboration Administrative issues Rural campus

Data Collection

Baseline Data Collection: Survey of

faculty and students

Interviews with AK EPSCoR leadership

Interim Data Collection: Student

interviews Survey of faculty

and students

Final Data Collection: University

administrator interviews

Native organizations interviews

Rural campus interviews

Survey of faculty and students

Social Networks & Evaluation

Social network analysis can add to the program evaluation methodological toolbox.

Why use SNA in program evaluation? Allows us to capture knowledge development, social and

human capital, and other interim outcomes that cannot be captured in other data.

Adds rigor to attitudinal and self reported data regarding behavior, outputs, outcomes, and relationships.

Adds detailed dyadic data on specific relationships and exchange.

Can be combined with other data to provide a comprehensive picture. Data may be drawn from surveys, interviews and existing documentation.

Some Network Characteristics Network Size

Number of nodes that are connected to others

Number of Ties Number of ties that link

these nodes individuals can have multiple ties)

Network Density measures the percentage

of ties that exist compared the number of possible ties.

This measure provides one way of describing the extent to which potential ties are unexploited.

Number of Ties Density Network centralization index Average Degree Centrality E-I Index (Organization) E-I Index (Specialty)

680 .09 22.36% 16.34% -.43 -.36

Legend: Organization Circle = Org1 Square= Org2 Triangle = Org3

Rank 1=Rank1 2= Rank2 3= Rank3 4= Rank4 5=Other

Specialties Red= Field1 Blue= Field2 Green= Field3

Cross Specialty Ties Green Line = Across Specialty Purple Line = Within Specialty

Some Network Characteristics

Average Degree Centrality measures the average

number of immediate connections that each individual has in the network.

Provides a way to examine the level of participation in network activity by the ‘average’ person in the network.


680 .09 22.36% 16.34% -.43 -.36





Some Network Characteristics

(E-I) Index the extent to which the

network is made up of individuals outside as compared to inside a particular environment or context.

The EI index is calculated as: (external ties – internal ties) / (external ties + internal ties) and ranges from negative one to one.


680 .09 22.36% 16.34% -.43 -.36





Our Final Report

Today’s Presentation Collaborative interactions – Year 3 and Overtime Integration with rural campuses and communities Student Impacts Observations and some lessons learned

Written Report Collaborative Interactions and Outcomes Student benefits and learning Integration with native communities Rural Campus issues Attitudes and feedback regarding EPSCoR issues Lessons Learned Recommendations for future EPSCoR efforts

Overview of Survey Respondents

Year 1 Faculty: 86% response rate (n=59) Students: 85% response rate (n=57)

Year 2: Faculty: 75% response rate (n=79) Students: 72% response rate (n=48)

Year 3 Faculty: 75% response rate (n=111) Students: 57% response rate (n=52)

Year 3 Faculty/Researcher Respondents (n=111)

UAA30%

UAS

6%

UAF64%

biological science

component38%

social science compo-

nent42%

physical science

component20%

Rank:Assistant Professor = 35%Associate Professor = 26%Full Professor = 21%Research faculty & other = 24%

Year 3 Student Respondents (n=52)

84% respondents at UAF

Bache-lors29%

Mas-ters25%

Doc-toral46%

Comparing Developments Across Time Useful to track the EPSCoR community

General observations on community of researchers.

More meaningful to track core group of individuals who have been involved over time. Longitudinal Faculty Comparison Groups Survey Respondents from Years 1 & 3 (n=44) Year 1 respondents and “active” individuals who

responded to Year 3 survey (n=65) (59% of Year 3 respondents)

Changes in Year 2 & 3

COLLABORATION & INTEGRATION

Sole Authoring by Component Y3 (mean number of faculty reporting these products)

.00

.20

.40

.60

.80

1.00

biological component social science componentphysical science component

Sole Authoring by Institution Y3(mean number of faculty reporting these products)

.00

.20

.40

.60

.80

1.00

UAA UAS UAF

What became of Desired Collaboration Year 1?

Respondents // Named desired collaborators 48 // 60Percent collaborators named in same discipline 44/60 (73.3%)Percent collaborators named in other disciplines 16/60 (26.7%)

Same disciplineBio to biological science 25

Phys to physical science 4Soc to social science 15

Different disciplineBio to physical science 3

Bio to social science 10Phys to social science 3

Percent disciplinary collaborations 2/16 (12.5%)

Percent interdisciplinary collaborations 10/44 (22.7%)

Who would you like to collaborate with, but have not done so to date?

Reasons for Successful Collaborations

Sharing common research interests in permafrost related arctic engineering and science issues.

Common interest in phylogenetic methodology, especially methods of DNA sequence alignment.

We are both interested in future trends in moose and caribou populations.

We are both interested in developing programs that link arts with the sciences.

Collaborative style and our complementarity [That person’s] energy and enthusiasm for doing

challenging empirical research.

Production: Journal Articles

Collaborative Outputs 2009-10: Journal Articles 2009-10 Number of Ties Density Average Degree Centrality E-I Index (Campus) E-I Index (Component)

93 .004 .72 -.720 -.462

2008-09 Number of Ties Density Average Degree Centrality E-I Index (Campus) E-I Index (Component)

71 .006 .92 -.52 -.40

Red=Biological ScienceBlue= Social ScienceGreen=Physical ScienceSquare= Year 1 & 3 participant

Impacts on Communication

Changed the way in which you communicate your research 2009-10 Number of Ties Density Average Degree Centrality E-I Index (Campus) E-I Index (Component)

127 .005 1.06 -.591 -.150

2008-09 Number of Ties Density E-I Index (Campus) E-I Index (Component)

113 .009 1.85 -.55 -.42


Impacts on Research

Changed the data collection or analysis methods you use in your research 2009-10 Number of Ties Density Average Degree Centrality E-I Index (Campus) E-I Index (Component)

108 .005 .95 -.685 -.481


82 .006 1.34 -.71 -.77


Growing Collaboration Around Key Research Themes

Disturbance Regimes 2009-10 Number of Ties Density Average Degree Centrality E-I Index (Campus) E-I Index (Component)

120 .005 1.01 -.78 -.33


78 .006 1.2 -.64 -.44


Coalescing of Research Themes

Species response to climate change

Disturbance regimes

Human migration & climate change

Perma-frost

Social Aaspects of

ground engineering

Ecosystem services

2009-10 100 120 86 95 52 95

2008-10 127 78 69 79 57 95

Change in the Number of Collaborative Ties over Time

Do Networks Matter for Continued Activity? Dependent Variable – Indicator of Continuation

Have you developed any new ideas or plans for research as a result of your interaction with EPSCoR Phase III? (1/0)

Independent Variables Familiarity “I understand this person’s knowledge and

research skills. This does not necessarily mean that I have these research skills or am knowledgeable in these domains, but that I understand the skills this person has and domains in which they are knowledgeable.” (Strongly Disagree to Strongly Agree)

Size of Collaboration Network: Since July 2007, with which of the following EPSCoR faculty members did you work with on a journal article. (1/0)

Other controls: Tenured, Biology, UAF

Example Findings

B SE Significance Exp(B)

Familiarity 1.647 0.778 ** 5.191

Collaboration Network Size 1.341 0.687 ** 0.262

Biology 1.962 1.421 0.141

UAF -1.474 1.089 4.366

Tenured -1.223 1.016 3.399

Constant 3.479 1.920 * 0.031

Have you developed any new ideas or plans for research as a result of your interaction with EPSCoR Phase III?

(1/0) *= p<0.10, ** = p<0.05)

INTEGRATION WITH RURAL CAMPUSES AND COMMUNITIES

Stakeholder Interaction

Faculty report limited importance of interaction with stakeholders.

Faculty report few meetings with stakeholders over the course of EPSCoR Phase III (1= never, 2= once, 3= 2 to 3 times, 4= more than three times)

UAA UAS UAF

Private Companies1.86 1.50 1.96

Rural campuses 1.37 1.57 1.50

K-12 education 1.64 2.43 2.05

Native communities 2.22 1.71 2.16

Native Alaskan corporations 1.46 1.00 1.62

State government officials 2.07 1.86 2.26

Local government officials 2.10 1.57 1.95

Federal government officials 2.11 2.00 2.29

Research in Native Communities

Native organizations and local contacts critical. Provide access Experiences with researchers are varied EPSCoR can facilitate the capacity of faculty

to interact with native communities Nature of outreach matters

Input / agreement of community Follow-up is key

Relevant research has implications for attraction and retention in STEM

Rural Campuses

Play key role as liaison to native communities Faculty are important nodes for connection Student are important nodes for connection

Serve multiple roles Skills and training Research for understanding local context and for

ensuring community security Capacity building in community

EPSCoR is an important supporter of research dissemination through Western Alaska Interdisciplinary Science Conference (WAISC).

STUDENT IMPACTS

EPSCoR Faculty and Students

28% of faculty report that they are currently working with an EPSCoR supported student.

50% of those report that the research would not have been conducted without EPSCoR funding.

Students Report Positive Impacts:Research Interests 75% of student respondents are required to

do a thesis. Of these:

65% report that their thesis or dissertation includes some aspects of climate change research.

55% report that their thesis or dissertation topic “has been influenced by their EPSCoR work.”

To what extent has your Alaska EPSCoR experience helped you to develop skills and knowledge in the following areas?(1= not at all, 2= some, 3= a great deal)

coordinating research across institutions

understanding climate change research

analyzing or interpreting data

working on a multi-disciplinary team

communicating technical concepts to others

writing professionally

understanding how to incorporate perspectives from difference disciplines

making academic presentations

designing research

.00 .50 1.00 1.50 2.00 2.50 3.00

PhD

Masters

Bachelors

PROSPECTS FOR THE FUTURE

Faculty Collaboration Interests

UAA UAS UAF

I would like to collaborate more with faculty in different disciplines.

3.19 3.43 3.17

I would like to collaborate more with faculty on other UA campuses.

3.32 3.57 3.00

I would like to collaborate more with students in different disciplines.

3.00 3.17 2.94

I anticipate continuing to collaborate with individuals I have worked with in Alaska Phase III

3.28 3.86 3.31

Generally positive attitudes across institutions regarding collaboration. (4 point scale, 1=strongly disagree)

Faculty Collaboration Interests

Faculty component-based responses show similar variation in interest in interacting with faculty and students in other disciplines.

Biological Social Physical

I would like to collaborate more with faculty in different disciplines.

3.05 3.28 3.30

I would like to collaborate more with faculty on other UA campuses.

3.13 3.14 3.10

I would like to collaborate more with students in different disciplines.

2.84 3.02 3.10

I anticipate continuing to collaborate with individuals I have worked with in Alaska Phase III

3.39 3.28 3.38

Production: Grant Proposals to Extend Phase III Research

Grant Proposals to Extend Phase III Research Number of Ties Density Average Degree Centrality E-I Index (Campus) E-I Index (Component)

93 .004 .75 -.570 -.247


Capacity Development in EPSCoR Phase III Research:

Evidence that collaborative networks have increased in size and diversity over Phase III.

Research theme-based networks are gaining more research ties across components.

Production and grant activities to sustain the capacity developed through Phase III are evident.

Faculty report positive impacts of EPSCoR engagement:

Production Connections Research ideas Changes in research processes

Capacity Development in EPSCoR Phase III Education :

Students report positive impacts of EPSCoR Phase III

Level of student research interests in climate change, as evidenced by thesis and dissertation topics, indicates potential capacity impacts on new generation of climate change researchers.

Strong student attribution of research skill development to EPSCoR Phase III, particularly among undergraduates.

Capacity Development in EPSCoR Phase III

Rural Campuses and Communities: Significant interest in developing stronger ties to bring

research and education opportunities and resources to rural campuses.

Researchers in the larger campuses express concern about “access to” rural communities.

Increasing exposure of climate change research in rural communities may attract and retain students there.

There are key individuals in rural campuses and communities that can help catalyze productive interaction.

Structural and Other Issues Relevant to Future EPSCoR Importance of support of EPSCoR at all levels of

administration and management. In smaller institutions it is not always obvious

how to best benefit from EPSCoR. Administrative cultures constitute barriers to

effective participation in and management of EPSCoR.

Ambitious efforts to cross disciplines and institutions require strong institutional support. Student network development across institutions

can also assist in this.

Questions and Comments

evaluating the alaska epscor phase iii: resilience and vulnerability in a rapidly changing north:...

Documents

epscor students

students interviews

ability of ak epscor

evaluation questions

alaska slide

students final data

chicago year

future epscor proposals