M&E Working Group and Evaluation Team
GEOSS and Users: User Engagement in the Evaluation
Process
Input from User Interface CommitteeLawrence Friedl (USA-NASA)
Evaluation Kick-off MeetingGeneva, Switzerland20-22.January.2010
2
Group on Earth ObservationsUsers & Evaluation Process
Sections of Presentation
I. “Users”
II. UIC work with users & user engagement
III. Mechanisms to reach users
Discussion
Group on Earth Observations I. Users
3
UIC Working Definition of the “GEOSS User”
The user may be any person that uses GEO data or information at various levels of sophistication. They may be considered as entities along a ‘value chain’ that includes, for example, the raw data collector, data archivist, modeler, scenario builder, program manager, policy analyst, and political decision maker.
Users will include the specialist technician, scientist, politician and lay public. The user may be an ‘expert’ in the sense that they have prior knowledge of how web portals may work and intuitively know how to find the data. And, the user may be a ‘novice’, such as an individual of the lay public, who may wish to explore information related to an unusual environmental event that he or she has just witnessed.
Group on Earth Observations I. Users
4
Group on Earth Observations I. Users
5
“Novice” User
• Science-to-policy analysts, decision makers, public officials, and the public • May be interested in specific issues that fall within one or more SBAs• Not GEOSS- or SOA-experienced• Looking for any and all observational data relevant to their issues• Want to easily find it and view it
Group on Earth Observations I. Users
6
GMES Network of Users – User Typology
Users may be classified along various dimensions:Position on value-adding chain (provider, intermediate, end user)Political level they operate in (international, national, regional, local)Field of work (e.g., GEO SBA)Role in projects Activity level
- operational, active, passive, nominal, statistical userReception of products
- how likely to give feedback and be vocal about useFurther and other ones are possible
Approach: User types may be defined in two or more-dimensional matrices
www.gmes-network-of-users.eu
7
Group on Earth ObservationsUsers & Evaluation Process
Sections of Presentation
I. “Users”
II. UIC work with users & user engagement
III. Mechanisms to reach users
Discussion
8
Group on Earth ObservationsII. UIC and Users
Current UIC User-oriented Approaches & Activities
A. User/Demand-side view of Earth observation priorities (Task US-09-01a)
B. User Types Analysis
C. User-based Call for Proposals
D. Usability Testing of GCI & Portals
E. Communities of Practice
F. GEO Task Sheet Analysis
9
Group on Earth ObservationsTask US-09-01a
GEO Task US-09-01a:
Establish a GEO process for identifying critical Earth observation priorities common to many GEOSS societal benefit areas, involving scientific and technical experts, taking account of socio-economic factors, and building on the results of existing systems’ requirements development processes.
UIC is using an approach that engages representatives of key user groups and geographic regions to identify observation priorities listed in user-oriented documents.
http://sbageotask.larc.nasa.gov/
10
Group on Earth ObservationsTask US-09-01a
GEO Societal Benefit AreaAdvisory Group
MembersDocuments in Meta-Analysis
Agriculture 11 15
Biodiversity 8 55
Climate 7 35
Disasters 13 40
Energy 14 53
Ecosystems 11 71
Human Health: Aeroallergens 16 117
Human Health: Air Quality 10 35
Human Health: Infectious Disease 17 165
Water 9 56
Weather 5 34
Total 121 676
Earth Observation Priorities (Task US-09-01a)
Documents Reviewed & ad hoc Advisory Groups Members, by SBA (figures as of 4-August-2009)
Analysts and Advisory Groups include people from Australia, Austria, Canada, China, Costa Rica, Denmark, Germany, Finland, France, Ghana, India, Iran, Italy, Kenya, Japan, Mexico, Norway, Paraguay, Russia, USA, Senegal, South Africa, Thailand, Tunisia, CEOS, DIVERSITAS, ECMWF, ESA, FAO, GCOS, IEEE, UNESCO, WMO, and others.
Agriculture SBA User Types
• Forest Protection Officers and Firefighters• Urban Foresters and Arborists• Microbiologists• Watershed Manager• Conservationists• Biologist• Ecologist• Land Use Planners• Health Specialists and Surveyors• Land Use Managers• Land Developers• Natural Resource Extraction (Loggers, Miners, Oilman)• Protected Area Managers• Park Ranger• Plant Pathologist• Recreation and Tourist Specialists• Tourists 11
Biodiversity SBA User Types
• Fish & Wildlife Biologists/Managers• Managers• Protected Area Managers• Botanists • Taxonomists• Conservationists• Ecologists• Land Use Planners• Land Use Managers• Land Developers• Natural Resource Extraction (Loggers, Miners, Oilman)• Policy Makers (National, State, and Local Governments;
NGOs)• Academicians• Researchers• Community Based Monitoring Groups• Concerned Citizens (e.g., bird watchers/groups)• Private Land Owners
12
Climate SBA User Types
PRIMARY USE OF CLIMATE OBSERVATIONSModeling and Scenario Development●Climate modelers●Integrated assessment modelers (e.g., IMAGE, DICE, MINICAM, MERGE)●Meteorologists● Intergovernmental Panel on Climate Change (IPCC) of the WMO and UNEP by way of the international research community contributing to these periodic assessmentsSECONDARY USE OF CLIMATE OBSERVATIONSExamples in Resource Science and ManagementModel and scenario results in turn used by: ●Marine biologists●Hydrologists●Ecologists●Energy researchers●Geographers●Earth system scientists●Environmental scientists●Conservationists●Etc.
POLICY USE OF CLIMATE OBSERVATIONSExamples of Decision Makers Science and management results in turn used by: ● National and sub national governments and international efforts in policy design for GHG stabilization, verification, monitoring, and adaptation● Infrastructure managers, planners and investors (such as the World Bank, Army Corps of Engineers, energy planners)●Etc.
13
Disasters SBA User Types
Research and Forecasting-University Professors-Graduate Students-Researchers at national research centers-Researchers at government research centers-Weather Forecasters and Modelers-Flood Forecasters and Modelers-Landslide Forecasters and Modelers-Earthquake Forecasters and Modelers-Seismologists-Geologists-Hydrologists
Emergency Management and Public Outreach-Risk Mitigation Manager-National and Local Emergency -Planning Managers-Weather Service Providers-Media: – Anchormen – Weathermen – Journalists-Local Law Enforcement/Fire Department Officers
InfrastructureCivil Engineers
Earthquake EngineersEnvironmental EngineersWater Resource ManagerCity Planners and Zoners
Regional PlannersInsurance Assessors
Risk Assessors 14
Ecosystems SBA User Types
• Water resources managers• Natural resources managers• Wildlife managers• Coastal planners and developers• Land use planners• Agricultural managers• Policymakers (local, state, national,
Environmental Protection Departments)• Researchers (climate, ecosystem function,
biological invasion, disease, future change)• Climate impacts analysts
15
Energy SBA User Types
Researchers -Resource Assessment (e.g., at a national lab, studying solar energy potential)-Technology Development (e.g., wind turbine designer)-Energy model developer (e.g., for analysis of likely future biomass scenarios)
Energy Facility Developers and Operators-e.g., Biofuels refinery planner, for plant siting and operations logistics-e.g., Wind farm developer, for plant siting and operations logistics-e.g., State utility, developing plans for how to meet a renewable portfolio standard-e.g., Water resources manager, deciding how to operate a hydropower facility
Energy Policy Developers-e.g., Energy Department official studying renewable portfolio standard options
Energy Regulators-e.g., Energy Commission official responsible for overseeing grid integration
Private Users-e.g., Rural village leader, utilizing renewable energy tools, to optimize village energy system including options such as micro-hydropower-e.g., Citizen determining whether to purchase a solar water heater
Electric Grid Operators-e.g., Independent System Operator (ISO) trying to predict how much wind power they will need to integrate into their regional grid system on a given day
Building Designer or Engineer-e.g., American Society of Heating, Refrigeration, and Air Conditioning Engineer (ASHRAE), to minimize building heating/cooling costs
16
Human Health: Aeroallergens SBA User Types
• Air quality model and systems developers/providers• Air quality scientists• Health care providers• Health outreach and education professionals• Health risk analysts• Interested/aware/concerned members of the public, including sensitive
populations• Public health organizations• Public health policy organizations• Public health managers and decision-makers• Public health policy makers• Non-governmental organizations (NGOs) and advocacy groups• Health scientists (e.g., epidemiologists, immunologists) • Weathercasters
17
Use of Assessments and Forecasts to Promote Health Protection-National and Sub national Policy Makers in Environmental and Public Health Sectors-National and Sub national Air Quality Managers-National and Sub national Public Health Managers and Decision-makers-Health Care Providers-Weathercasters -Health Outreach and Education Professionals-Interested/Aware/Concerned Members of the Public-Businesses Supporting Local Clean Air Programs-NGOs and Advocacy Groups
Direct Use of Earth Observations (EO)-Air Quality Researchers-Environmental ScientistsUse of EO to Develop and Populate Integrative Systems, Analytical Tools, and Models-Air Quality Model and Systems Developers/Providers-Public Health SpecialistsUse of Tools and Models to Assess and Forecast Air Quality -Air Quality Scientists-Environmental Process Modelers and Researchers-Public Health Specialists
18
Human Health: Air Quality SBA User Types
ENVIRONMENTAL CHANGE* HEALTH PROTECTIONDirect Use of Earth Observations (EO)
Climate Scientists* Environmental Scientists (Ecologists, Land
Use/Cover Specialists, etc.)*
International Public Health Organizations (e.g., WHO Global Health Observatory)
National and Sub national Public Health Policy Makers Use of EO to Develop and Populate Integrative Systems, Analytical Tools, and Models
Environmental Process Modelers and Systems Developers*
Public Health Risk Modelers and Systems Developers Environmental Benefits Valuation Specialists
Use of Tools and Models to Assess and Forecast Disease Incidence and Trends Health Risk Analysts Public Health Scientists (e.g., Epidemiologists, Infectious
Disease Specialists)Use of Assessments and Forecasts to Promote Health Protection
National and Sub national Policy Makers in Environmental Sectors*
NGOs and Advocacy Groups* Infrastructure Managers, Planners and Investors*
International Public Health Policy Organizations National and Sub national Public Health Policy Makers National and Sub national Public Health Managers and
Decision-makers Health Care Providers Health Outreach and Education Professionals Interested/Aware/Concerned Members of the Public NGOs and Advocacy Groups
19
Human Health: Infectious DiseaseSBA User Types
Water SBA User Types
• Climate Scientist• Climate Forecaster• Weather Forecaster• Hydrologist• Flood Forecaster• Drought Forecaster• Water Resource Manager• Water Res. Allocator• Urban Water Supplier• Agronomist/Farmer• Hydro Power Engineer• Energy (other) Engineer• Heating/cooling Engineer• Irrigation Scheduler• Manager• Conservationist• Ecologist• Environ. Engineer
• Fire Prevention Planner• Fire Fighter• Environmental Protection
Manager• Nat. Disaster Manager• Drought Monitor• Drought Mitigation Manager• Flood Control Manager• Flood Control Planner• Nat. Hazards Risk Assessor• Insurance purveyor• Land Use Planner• Epidemiologists• Disease Outbreak Forecaster• Pollution Forecaster• Water Quality Manager• Urban Planners• City Development Zoner• Environmental Impact
Assessor 20
Weather SBA User Types
Public organizationsCivil protection agencies and organizations:Warning systems operatorCrisis / disaster / emergency managerRisk managerFire department officerPolice officer-National meteorological and hydrological services (NMHS):Weather forecasterMeteorologist (e.g. scientific applications, systematic climatologic monitoring)Climate modelerFlood modeler, flood forecaster-Miscellaneous:Water resource managerWater resource scientist
Private companies-Aviation:Airport / aerodrome operatorAir traffic controller / control tower operatorPilot-Civil engineering:Building engineer (e.g., optimization of heating technology)-Commercial shipping and offshore industry:CaptainNautical engineerOffshore facility engineerShip owner-Energy industry: Renewable energy consultant (e.g., assessment of available energies)Renewable energy project manager (e.g., planning of photovoltaic or wind installations)-Health care:Health resort managerHealth clinic managerDoctor (e.g. dermatologist)-Media:Weatherman-Meteorological service providers:Weather forecasterMeteorologist-Telecommunication:Telecommunication engineer (e.g., calculation of microwave telecommunication signal losses)-Tourism:Hotel keeperTravel agentEvent manager
Private users-Agriculture:Farmer-Aviation:PilotBalloonist-Shipping:Yachtsman-Miscellaneous private users:Hobby meteorologist
Scientific organizations-Universities and research centers:Scientists from various geoscientific and earth system science disciplines 21
22
Group on Earth ObservationsCFP for Decision Support Projects
Overview
CFP seeks proposals for projects to apply Earth observations to improve policy decisions and management decisions.
CFP seeks to identify and enable practical applications of Earth obs.
Strong focus on Developing Countries and supporting end-users
Showcase the value and benefits of Earth obs. and GEOSS through successful applications examples.
Topics & GEO Tasks
Focus on 4 SBAs:– Agriculture (including Forests)– Energy– Health (including Air Quality)– Water
CFP supports GEO Tasks:• CB-09-01 (Mobilising Resources)• US-09-01b (Communities of Practice)
Project Types:I. New users to Earth obs.II. Some experience using Earth obsIII. Lots of experience with Earth obs
23
Group on Earth ObservationsCFP for Decision Support Projects
Type I & II Proposals (by SBA)
Agriculture 32
Energy 3
Health 18
Water 57
Total: 110
Majority were Type II
Group on Earth ObservationsGEOSS Portal/GCI Usability Testing
GEO Web Portal Usability Testing
GEO UIC set up computer workstations at locations where range of users could get onto GEOSS system and provide feedback
Four rounds: ISRSE Conference (May 2009), electronically (Sept. 2009), GEO Plenary VI (Nov. 2009), ESIP Winter meeting (Jan. 2010)
Two main types of questions:
1. USER INFORMATION QUESTIONSHow would you classify yourself professionally?What best describes your GIS and geospatial experience?Identify the geographic area of interest (AOI) for your search.
2. GEO WEB PORTAL OPERATIONAL QUESTIONS
24
Group on Earth ObservationsCommunity of Practice
Communities of Practice (US-09-01b)
A Community of Practice (CoP) is a user-led community of stakeholders, from providers to the final beneficiaries of Earth observation data and information, with a common interest in specific aspects of societal benefits to be realized by GEOSS implementation.
Each Community of Practice should include representatives from GEO Members, Participating Organizations, and any other stakeholders that have similar interests, goals, and or objectives.
The CoP members should work closely together to create a forum for efficient and effective intelligence and advice to be provided to GEO for the successful implementation of GEOSS. Both developing and developed countries should be represented.
25
Group on Earth Observations Task Sheet Analysis – User Engagement
Task Sheet Analysis
UIC has a responsibility within GEO to address the user element in all of the tasks in the Work Plan. Some tasks have a significant, inherent user-engagement dimension to them, and some tasks do not (though eventually all might).
UIC pursued analysis focused on assessing the quality of the user-engagement component in the task sheets.
Analysis looked especially for tasks that an inherent need for user engagement and a high probability of success in engaging users. Analysis sought to identify key needs/ways to improve Task Leads’ understanding of user engagement
5 major findings and 5 major recommendations
26
Group on Earth Observations Task Sheet Analysis – User Engagement
Note: Analysis used the task sheets submitted in December 2008 & March 2009 27
Group on Earth Observations Task Sheet Analysis – User Engagement
29
Task Sheet Analysis
High Low Total
Yes 32 19 51
No 19 44 63
Total 51 63 114
Table 3. Opportunity for Broad User Engagement in the Task vs. Presence of Text on Users
II.A: Text in User
Engagement Section
II.D: Opportunity for Broad User Engagement
Group on Earth Observations Task Sheet Analysis – User Engagement
User Engagement Aspects of Task Sheets
30
51 task sheets had some text in the UE section (63 had nothing).
Of the 51 task sheets that had text:
- 15 task sheets (29%) had substantive, thoughtful statements
- 25 sheets (49%) had some level of thought
- 11 sheets (22%) had minimal response.
Thus, overall, only 13% of all task sheets (15 of 114) gave substantive attention to user engagement in the UE section.
Group on Earth Observations Task Sheet Analysis – User Engagement
User Engagement Aspects of Task Sheets
31
Some of the recommendations:
- GEO/UIC should develop materials on user engagement
Document and publish a range of user engagement techniques and examples as a resource to the GEO community.
- GEO/UIC should utilize its User Type Analysis to support task teams to understand, design, and implement user engagement approaches.
UIC should publish user types for GEO’s use and encourage task teams to employ the user types to identify potential user groups they can engage in their tasks.
32
Group on Earth ObservationsUsers & Evaluation Process
Sections of Presentation
I. “Users”
II. UIC work with users & user engagement
III. Mechanisms to reach users
Discussion
Group on Earth Observations III. Mechanisms to Reach Users
33
Multiple Approaches to Reaching Users
1. Go to where the users are Usability testing
Key conferences where cross-section of user types or specific user-types of particular interest gather .
Surveys and feedback on actual experiences
2. Solicit input and ideas Call for Proposals
Request information from broad to specific user types
Work with key associations that have established networks
3. Work through credible representatives US-09-01a
Representatives able to consolidate and present community-wide views
Group on Earth Observations III. Mechanisms to Reach Users
34
Multiple Approaches to Reaching Users (cont.)
4. Establish on-going fora CoP
Fora for people to gather (physically or virtually) and identify common interests and action to pursue
5. Utilize existing statements Task Sheet Analysis
Gather information from explicit statements of implied opinions and information
In addition: Self-assessment User Types
Review of your approach; perform gap-analyses to identify biases
Group on Earth Observations III. Mechanisms to Reach Users
35
Other Considerations
Protecting from bias: Large sample vs random sampling
Open up broad surveys to get sheer numbers of responses; however, this opens up to bias by not having random sampling
Qualitative vs. Quantitative input
------
UIC has been focused on sheer numbers to begin with, which potentially opens a bias (relies on connecting with most vocal users)
UIC is just moving into more quantitative input
36
Group on Earth ObservationsTask US-09-01a
Discussion
Lawrence Friedl, UIC Member
NASA Earth Science DivisionDirector (Acting), Applied Sciences Program
1.202.358.7200 ▪ [email protected]