m&e working group and evaluation team geoss and users: user engagement in the evaluation process...

M&E Working Group and Evaluation Team

GEOSS and Users: User Engagement in the Evaluation

Process

Input from User Interface CommitteeLawrence Friedl (USA-NASA)

Evaluation Kick-off MeetingGeneva, Switzerland20-22.January.2010

2

Group on Earth ObservationsUsers & Evaluation Process

Sections of Presentation

I. “Users”

II. UIC work with users & user engagement

III. Mechanisms to reach users

Discussion

Group on Earth Observations I. Users

3

UIC Working Definition of the “GEOSS User”

The user may be any person that uses GEO data or information at various levels of sophistication. They may be considered as entities along a ‘value chain’ that includes, for example, the raw data collector, data archivist, modeler, scenario builder, program manager, policy analyst, and political decision maker.

Users will include the specialist technician, scientist, politician and lay public. The user may be an ‘expert’ in the sense that they have prior knowledge of how web portals may work and intuitively know how to find the data. And, the user may be a ‘novice’, such as an individual of the lay public, who may wish to explore information related to an unusual environmental event that he or she has just witnessed.


4


5

“Novice” User

• Science-to-policy analysts, decision makers, public officials, and the public • May be interested in specific issues that fall within one or more SBAs• Not GEOSS- or SOA-experienced• Looking for any and all observational data relevant to their issues• Want to easily find it and view it


6

GMES Network of Users – User Typology

Users may be classified along various dimensions:Position on value-adding chain (provider, intermediate, end user)Political level they operate in (international, national, regional, local)Field of work (e.g., GEO SBA)Role in projects Activity level

- operational, active, passive, nominal, statistical userReception of products

- how likely to give feedback and be vocal about useFurther and other ones are possible

Approach: User types may be defined in two or more-dimensional matrices

www.gmes-network-of-users.eu

7



I. “Users”



Discussion

8

Group on Earth ObservationsII. UIC and Users

Current UIC User-oriented Approaches & Activities

A. User/Demand-side view of Earth observation priorities (Task US-09-01a)

B. User Types Analysis

C. User-based Call for Proposals

D. Usability Testing of GCI & Portals

E. Communities of Practice

F. GEO Task Sheet Analysis

9

Group on Earth ObservationsTask US-09-01a

GEO Task US-09-01a:

Establish a GEO process for identifying critical Earth observation priorities common to many GEOSS societal benefit areas, involving scientific and technical experts, taking account of socio-economic factors, and building on the results of existing systems’ requirements development processes.

UIC is using an approach that engages representatives of key user groups and geographic regions to identify observation priorities listed in user-oriented documents.

http://sbageotask.larc.nasa.gov/

10


GEO Societal Benefit AreaAdvisory Group

MembersDocuments in Meta-Analysis

Agriculture 11 15

Biodiversity 8 55

Climate 7 35

Disasters 13 40

Energy 14 53

Ecosystems 11 71

Human Health: Aeroallergens 16 117

Human Health: Air Quality 10 35

Human Health: Infectious Disease 17 165

Water 9 56

Weather 5 34

Total 121 676

Earth Observation Priorities (Task US-09-01a)

Documents Reviewed & ad hoc Advisory Groups Members, by SBA (figures as of 4-August-2009)

Analysts and Advisory Groups include people from Australia, Austria, Canada, China, Costa Rica, Denmark, Germany, Finland, France, Ghana, India, Iran, Italy, Kenya, Japan, Mexico, Norway, Paraguay, Russia, USA, Senegal, South Africa, Thailand, Tunisia, CEOS, DIVERSITAS, ECMWF, ESA, FAO, GCOS, IEEE, UNESCO, WMO, and others.

Agriculture SBA User Types

• Forest Protection Officers and Firefighters• Urban Foresters and Arborists• Microbiologists• Watershed Manager• Conservationists• Biologist• Ecologist• Land Use Planners• Health Specialists and Surveyors• Land Use Managers• Land Developers• Natural Resource Extraction (Loggers, Miners, Oilman)• Protected Area Managers• Park Ranger• Plant Pathologist• Recreation and Tourist Specialists• Tourists 11

Biodiversity SBA User Types

• Fish & Wildlife Biologists/Managers• Managers• Protected Area Managers• Botanists • Taxonomists• Conservationists• Ecologists• Land Use Planners• Land Use Managers• Land Developers• Natural Resource Extraction (Loggers, Miners, Oilman)• Policy Makers (National, State, and Local Governments;

NGOs)• Academicians• Researchers• Community Based Monitoring Groups• Concerned Citizens (e.g., bird watchers/groups)• Private Land Owners

12

Climate SBA User Types

PRIMARY USE OF CLIMATE OBSERVATIONSModeling and Scenario Development●Climate modelers●Integrated assessment modelers (e.g., IMAGE, DICE, MINICAM, MERGE)●Meteorologists● Intergovernmental Panel on Climate Change (IPCC) of the WMO and UNEP by way of the international research community contributing to these periodic assessmentsSECONDARY USE OF CLIMATE OBSERVATIONSExamples in Resource Science and ManagementModel and scenario results in turn used by: ●Marine biologists●Hydrologists●Ecologists●Energy researchers●Geographers●Earth system scientists●Environmental scientists●Conservationists●Etc.

POLICY USE OF CLIMATE OBSERVATIONSExamples of Decision Makers Science and management results in turn used by: ● National and sub national governments and international efforts in policy design for GHG stabilization, verification, monitoring, and adaptation● Infrastructure managers, planners and investors (such as the World Bank, Army Corps of Engineers, energy planners)●Etc.

13

Disasters SBA User Types

Research and Forecasting-University Professors-Graduate Students-Researchers at national research centers-Researchers at government research centers-Weather Forecasters and Modelers-Flood Forecasters and Modelers-Landslide Forecasters and Modelers-Earthquake Forecasters and Modelers-Seismologists-Geologists-Hydrologists

Emergency Management and Public Outreach-Risk Mitigation Manager-National and Local Emergency -Planning Managers-Weather Service Providers-Media: – Anchormen – Weathermen – Journalists-Local Law Enforcement/Fire Department Officers

InfrastructureCivil Engineers

Earthquake EngineersEnvironmental EngineersWater Resource ManagerCity Planners and Zoners

Regional PlannersInsurance Assessors

Risk Assessors 14

Ecosystems SBA User Types

• Water resources managers• Natural resources managers• Wildlife managers• Coastal planners and developers• Land use planners• Agricultural managers• Policymakers (local, state, national,

Environmental Protection Departments)• Researchers (climate, ecosystem function,

biological invasion, disease, future change)• Climate impacts analysts

15

Energy SBA User Types

Researchers -Resource Assessment (e.g., at a national lab, studying solar energy potential)-Technology Development (e.g., wind turbine designer)-Energy model developer (e.g., for analysis of likely future biomass scenarios)

Energy Facility Developers and Operators-e.g., Biofuels refinery planner, for plant siting and operations logistics-e.g., Wind farm developer, for plant siting and operations logistics-e.g., State utility, developing plans for how to meet a renewable portfolio standard-e.g., Water resources manager, deciding how to operate a hydropower facility

Energy Policy Developers-e.g., Energy Department official studying renewable portfolio standard options

Energy Regulators-e.g., Energy Commission official responsible for overseeing grid integration

Private Users-e.g., Rural village leader, utilizing renewable energy tools, to optimize village energy system including options such as micro-hydropower-e.g., Citizen determining whether to purchase a solar water heater

Electric Grid Operators-e.g., Independent System Operator (ISO) trying to predict how much wind power they will need to integrate into their regional grid system on a given day

Building Designer or Engineer-e.g., American Society of Heating, Refrigeration, and Air Conditioning Engineer (ASHRAE), to minimize building heating/cooling costs

16

Human Health: Aeroallergens SBA User Types

• Air quality model and systems developers/providers• Air quality scientists• Health care providers• Health outreach and education professionals• Health risk analysts• Interested/aware/concerned members of the public, including sensitive

populations• Public health organizations• Public health policy organizations• Public health managers and decision-makers• Public health policy makers• Non-governmental organizations (NGOs) and advocacy groups• Health scientists (e.g., epidemiologists, immunologists) • Weathercasters

17

Use of Assessments and Forecasts to Promote Health Protection-National and Sub national Policy Makers in Environmental and Public Health Sectors-National and Sub national Air Quality Managers-National and Sub national Public Health Managers and Decision-makers-Health Care Providers-Weathercasters -Health Outreach and Education Professionals-Interested/Aware/Concerned Members of the Public-Businesses Supporting Local Clean Air Programs-NGOs and Advocacy Groups

Direct Use of Earth Observations (EO)-Air Quality Researchers-Environmental ScientistsUse of EO to Develop and Populate Integrative Systems, Analytical Tools, and Models-Air Quality Model and Systems Developers/Providers-Public Health SpecialistsUse of Tools and Models to Assess and Forecast Air Quality -Air Quality Scientists-Environmental Process Modelers and Researchers-Public Health Specialists

18

Human Health: Air Quality SBA User Types

ENVIRONMENTAL CHANGE* HEALTH PROTECTIONDirect Use of Earth Observations (EO)

Climate Scientists* Environmental Scientists (Ecologists, Land

Use/Cover Specialists, etc.)*

International Public Health Organizations (e.g., WHO Global Health Observatory)

National and Sub national Public Health Policy Makers Use of EO to Develop and Populate Integrative Systems, Analytical Tools, and Models

Environmental Process Modelers and Systems Developers*

Public Health Risk Modelers and Systems Developers Environmental Benefits Valuation Specialists

Use of Tools and Models to Assess and Forecast Disease Incidence and Trends Health Risk Analysts Public Health Scientists (e.g., Epidemiologists, Infectious

Disease Specialists)Use of Assessments and Forecasts to Promote Health Protection

National and Sub national Policy Makers in Environmental Sectors*

NGOs and Advocacy Groups* Infrastructure Managers, Planners and Investors*

International Public Health Policy Organizations National and Sub national Public Health Policy Makers National and Sub national Public Health Managers and

Decision-makers Health Care Providers Health Outreach and Education Professionals Interested/Aware/Concerned Members of the Public NGOs and Advocacy Groups

19

Human Health: Infectious DiseaseSBA User Types

Water SBA User Types

• Climate Scientist• Climate Forecaster• Weather Forecaster• Hydrologist• Flood Forecaster• Drought Forecaster• Water Resource Manager• Water Res. Allocator• Urban Water Supplier• Agronomist/Farmer• Hydro Power Engineer• Energy (other) Engineer• Heating/cooling Engineer• Irrigation Scheduler• Manager• Conservationist• Ecologist• Environ. Engineer

• Fire Prevention Planner• Fire Fighter• Environmental Protection

Manager• Nat. Disaster Manager• Drought Monitor• Drought Mitigation Manager• Flood Control Manager• Flood Control Planner• Nat. Hazards Risk Assessor• Insurance purveyor• Land Use Planner• Epidemiologists• Disease Outbreak Forecaster• Pollution Forecaster• Water Quality Manager• Urban Planners• City Development Zoner• Environmental Impact

Assessor 20

Weather SBA User Types

Public organizationsCivil protection agencies and organizations:Warning systems operatorCrisis / disaster / emergency managerRisk managerFire department officerPolice officer-National meteorological and hydrological services (NMHS):Weather forecasterMeteorologist (e.g. scientific applications, systematic climatologic monitoring)Climate modelerFlood modeler, flood forecaster-Miscellaneous:Water resource managerWater resource scientist

Private companies-Aviation:Airport / aerodrome operatorAir traffic controller / control tower operatorPilot-Civil engineering:Building engineer (e.g., optimization of heating technology)-Commercial shipping and offshore industry:CaptainNautical engineerOffshore facility engineerShip owner-Energy industry: Renewable energy consultant (e.g., assessment of available energies)Renewable energy project manager (e.g., planning of photovoltaic or wind installations)-Health care:Health resort managerHealth clinic managerDoctor (e.g. dermatologist)-Media:Weatherman-Meteorological service providers:Weather forecasterMeteorologist-Telecommunication:Telecommunication engineer (e.g., calculation of microwave telecommunication signal losses)-Tourism:Hotel keeperTravel agentEvent manager

Private users-Agriculture:Farmer-Aviation:PilotBalloonist-Shipping:Yachtsman-Miscellaneous private users:Hobby meteorologist

Scientific organizations-Universities and research centers:Scientists from various geoscientific and earth system science disciplines 21

22

Group on Earth ObservationsCFP for Decision Support Projects

Overview

CFP seeks proposals for projects to apply Earth observations to improve policy decisions and management decisions.

CFP seeks to identify and enable practical applications of Earth obs.

Strong focus on Developing Countries and supporting end-users

Showcase the value and benefits of Earth obs. and GEOSS through successful applications examples.

Topics & GEO Tasks

Focus on 4 SBAs:– Agriculture (including Forests)– Energy– Health (including Air Quality)– Water

CFP supports GEO Tasks:• CB-09-01 (Mobilising Resources)• US-09-01b (Communities of Practice)

Project Types:I. New users to Earth obs.II. Some experience using Earth obsIII. Lots of experience with Earth obs

23

Group on Earth ObservationsCFP for Decision Support Projects

Type I & II Proposals (by SBA)

Agriculture 32

Energy 3

Health 18

Water 57

Total: 110

Majority were Type II

Group on Earth ObservationsGEOSS Portal/GCI Usability Testing

GEO Web Portal Usability Testing

GEO UIC set up computer workstations at locations where range of users could get onto GEOSS system and provide feedback

Four rounds: ISRSE Conference (May 2009), electronically (Sept. 2009), GEO Plenary VI (Nov. 2009), ESIP Winter meeting (Jan. 2010)

Two main types of questions:

1. USER INFORMATION QUESTIONSHow would you classify yourself professionally?What best describes your GIS and geospatial experience?Identify the geographic area of interest (AOI) for your search.

2. GEO WEB PORTAL OPERATIONAL QUESTIONS

24

Group on Earth ObservationsCommunity of Practice

Communities of Practice (US-09-01b)

A Community of Practice (CoP) is a user-led community of stakeholders, from providers to the final beneficiaries of Earth observation data and information, with a common interest in specific aspects of societal benefits to be realized by GEOSS implementation.

Each Community of Practice should include representatives from GEO Members, Participating Organizations, and any other stakeholders that have similar interests, goals, and or objectives.

The CoP members should work closely together to create a forum for efficient and effective intelligence and advice to be provided to GEO for the successful implementation of GEOSS. Both developing and developed countries should be represented.

25

Group on Earth Observations Task Sheet Analysis – User Engagement

Task Sheet Analysis

UIC has a responsibility within GEO to address the user element in all of the tasks in the Work Plan. Some tasks have a significant, inherent user-engagement dimension to them, and some tasks do not (though eventually all might).

UIC pursued analysis focused on assessing the quality of the user-engagement component in the task sheets.

Analysis looked especially for tasks that an inherent need for user engagement and a high probability of success in engaging users. Analysis sought to identify key needs/ways to improve Task Leads’ understanding of user engagement

5 major findings and 5 major recommendations

26


Note: Analysis used the task sheets submitted in December 2008 & March 2009 27


29

Task Sheet Analysis

High Low Total

Yes 32 19 51

No 19 44 63

Total 51 63 114

Table 3. Opportunity for Broad User Engagement in the Task vs. Presence of Text on Users

II.A: Text in User

Engagement Section

II.D: Opportunity for Broad User Engagement


User Engagement Aspects of Task Sheets

30

51 task sheets had some text in the UE section (63 had nothing).

Of the 51 task sheets that had text:

- 15 task sheets (29%) had substantive, thoughtful statements

- 25 sheets (49%) had some level of thought

- 11 sheets (22%) had minimal response.

Thus, overall, only 13% of all task sheets (15 of 114) gave substantive attention to user engagement in the UE section.


User Engagement Aspects of Task Sheets

31

Some of the recommendations:

- GEO/UIC should develop materials on user engagement

Document and publish a range of user engagement techniques and examples as a resource to the GEO community.

- GEO/UIC should utilize its User Type Analysis to support task teams to understand, design, and implement user engagement approaches.

UIC should publish user types for GEO’s use and encourage task teams to employ the user types to identify potential user groups they can engage in their tasks.

32



I. “Users”



Discussion

Group on Earth Observations III. Mechanisms to Reach Users

33

Multiple Approaches to Reaching Users

1. Go to where the users are Usability testing

Key conferences where cross-section of user types or specific user-types of particular interest gather .

Surveys and feedback on actual experiences

2. Solicit input and ideas Call for Proposals

Request information from broad to specific user types

Work with key associations that have established networks

3. Work through credible representatives US-09-01a

Representatives able to consolidate and present community-wide views


34

Multiple Approaches to Reaching Users (cont.)

4. Establish on-going fora CoP

Fora for people to gather (physically or virtually) and identify common interests and action to pursue

5. Utilize existing statements Task Sheet Analysis

Gather information from explicit statements of implied opinions and information

In addition: Self-assessment User Types

Review of your approach; perform gap-analyses to identify biases


35

Other Considerations

Protecting from bias: Large sample vs random sampling

Open up broad surveys to get sheer numbers of responses; however, this opens up to bias by not having random sampling

Qualitative vs. Quantitative input

------

UIC has been focused on sheer numbers to begin with, which potentially opens a bias (relies on connecting with most vocal users)

UIC is just moving into more quantitative input

36


Discussion

Lawrence Friedl, UIC Member

NASA Earth Science DivisionDirector (Acting), Applied Sciences Program

1.202.358.7200 ▪ [email protected]

m&e working group and evaluation team geoss and users: user engagement in the evaluation process...

Documents

users user engagement

geoss user

users discussion slide

novice user science

useroriented documents

earth observations task

user types analysis

end user political level