geob 409 field measurements and data analysisintroduction to data analysis! 7! dimensions.! strictly...
TRANSCRIPT
Koppes / Geography 409 Introduction to data analysis
GEOB 409���Field Measurements and Data Analysis���
1
Koppes / Geography 409 Introduction to data analysis
2
A classical research approach.
Documentation ‘Hands-on’ Creative Part
Hypothesis, motivation, possibilities
Experiment design���Analysis plan
Conclusions, hypothesis verification / falsification
Project Proposal Information
Lab- and field measurements
Experiment documentation
Des
ign
Expe
rim
ent
Ana
lysi
s Data analysis
Data
Analysis documentation
Report
Koppes / Geography 409 Introduction to data analysis
3
Field data in physical geography.
Any natural environment is a complex, multivariate web of interacting variables.
We can never measure ���continuously everything, ���everywhere.
We sample selected data���with an appropriate ���strategy.
Koppes / Geography 409 Introduction to data analysis
4
Error in Scientific Measurement means the Inevitable Uncertainty that attends all measurements���-Fritschen and Gay
Uncertainties are ubiquitous and therefore no reflection on the usefulness of the measurement or the competence of the measurer
If we are to rationally use a measurement the uncertainties must be known quantitatively
Koppes / Geography 409 Introduction to data analysis
5
Data-sets and data analysis – keep it simple!
UCAR Digital Image Library
Koppes / Geography 409 Introduction to data analysis
6
Boolean data Continuous data Classification
-5.24 -3.20 2.32 9.63
14.15 17.83 27.08 16.34 10.22 4.43
-1.32
Cold Cold Cool Cool Warm Warm Hot Warm Cool Cool Cold
Yes Yes No No No No No No No No Yes
What will your data look like? - Data format da
ta d
imen
sion
data
dim
ensi
on
data
dim
ensi
on
Koppes / Geography 409 Introduction to data analysis
7
Dimensions.
Strictly speaking, we sample any physical, chemical or biological variable φ in a four dimensional setting, i.e. as a function of time t and space x (x, y, z).
Luckily, quite often we are not interested in all four dimensions. Likely, we focus on a single or two dimensions due to logistical reasons or because your study object allows this. You might assume that variability in one dimension is much smaller than in another one (homogeneity, stationarity critearia).
→ In your project you will implicitly include certain dimensions / exclude others.
Koppes / Geography 409 Introduction to data analysis
8
Dimensions.
Basic dimension Data-set Resolution Examples of 1-D data
Time Time series Temporal resolution
• One day of 10 min temperature measurements at a climate station • One year of hourly discharge measurements from a stream
Space
Horizontal profile Spatial resolution
• Vegetation classification along a traverse. • A horizontal profile of snow water content along a line.
Vertical profile Spatial resolution
• A tethered balloon run measuring wind with height. • Temperature change in soil with depth.
(Frequency)* Spectrum Spectral resolution • A histogram of different grain sizes in sediments • Irradiance in different wavelengths
*This is strictly speaking not a basic dimension, but a transformation of time or space
Koppes / Geography 409 Introduction to data analysis
9
Examples of data sets - time series.
time dimension
Example: Carbon dioxide in a forest as a function of time of a day
Koppes / Geography 409 Introduction to data analysis
10
Examples of data sets - horizontal profiles.
Example: Horizontal transect through Vegetation
Example: Horizontal transect showing air temperature
horizontal dimension horizontal dimension
Ecosystems of BC / T.R. Oke (1987): 'Boundary Layer Climates' 2nd Edition.
Koppes / Geography 409 Introduction to data analysis
11
Examples of data sets - vertical profiles.
vert
ical
dim
ensi
on
vert
ical
dim
ensi
on
Univ. Stuttgart / T.R. Oke (1987): 'Boundary Layer Climates' 2nd Edition.
Koppes / Geography 409 Introduction to data analysis
12
Examples of two dimensional data sets.
time-space Example: temperatures in a lake as a
function of time of year and water depth
yearly course (time)
space-space (map) Example: land use
hori
zont
al d
imen
sion
horizontal dimension ve
rtic
al d
imen
sion
time dimension
Koppes / Geography 409 Introduction to data analysis
13
Resolution.
• Temporal and spatial resolution: How many data-points per unit of a dimension? Temporal resolution and spatial resolution, i.e 1 measurement a day vs. 1440 measurements a day, or 1 measurement per km vs. 1000 measurements per km.
• Data depth: How accurately can we distinguish between different physical values, i.e. 0.02 vs. 0.0214523.
Illustration: Wikipedia
Koppes / Geography 409 Introduction to data analysis
14
Gridded vs. irregular data
irregular regular
Voronoi tessellation
. Data Points
Nor
th (
spac
e)
East (space)
Nor
th (
spac
e)
East (space)
Koppes / Geography 409 Introduction to data analysis
15
Example of a regular grid in vegetation studies.
Photo: http://www.marine.gov/
Koppes / Geography 409 Introduction to data analysis
16
Integration and interpolation.
Integration refers to the process of combining or accumulating - or more generally to methods of upscaling - data from an existing set of measured data points. Interpolation refers to the process of splitting down or fill-in data to constructing new data points - or generally to methods of downscaling - an existing set of measured data points. Both can be done in time and space domains, and there are various methods.
Koppes / Geography 409 Introduction to data analysis
17
Your data set?
Choose a physical parameter or a classification of interest in your potential project: Data format? Dimensions? Resolution? Regular or irregular? Assumptions?
Koppes / Geography 409 Introduction to data analysis
18
Data processing.
• Correcting sensors with data from lab calibrations or field intercomparisons.
• Plausibility checks - define criteria for errors, experiment disturbances, etc.
• Flag data - remove data that fulfill the above criteria (never delete data forever, just flag it - and backup raw data!).
• Integrate or interpolate data - only if your data are not at the scale required, or if you have to compare two data sets with different resolutions.
• Select data for further analysis if you have made assumptions to fulfill certain criteria.
Koppes / Geography 409 Introduction to data analysis
19
Check your data! Potential approaches?
Koppes / Geography 409 Introduction to data analysis
20
Check your data - potential approaches.
• Global criteria ���(minimum, maximum, ...).
• Local criteria���(rate of change, ...).
• Statistical criteria.
• Manual data flagging.
Standard deviation
CO2-concentration
Koppes / Geography 409 Introduction to data analysis
21
Analysis tools
• Describe data distribution - statistical probability of occurrence, histograms, statistical moments, ...
• Find events - peak detection, integration, ...
• Find and quantify correlations (same variable at different locations, same variable at compared different times, between two variables, correlation between model and measured values) - correlation, regressions, curve-fitting, statistical tests
• Find groups and dominating dimensions - Clustering, principal component analysis,
• Find process dominating scales - Spectral analysis finds process dominating time and length scales, wavelets.
Koppes / Geography 409 Introduction to data analysis
22
Data analysis tools. Method / System Advantage Limitations Example of a system
Manual Analysis very simple, fast up to a few 10s of data-points, no large data-sets, no modelling
Calculator, paper, pen...
Spread-sheet software
simple analysis and graph tools
Limited # of data points, limited statistics, modelling, and automation & slow.
Microsoft Excel
GIS system complex spatial analysis and modelling
Expert knowledge. Expensive.
Workstation with ArcGIS
Statistical software and programming languages
complex and fast time series analysis, automation, modelling
Programming skills. Expensive.
Workstation with Matlab, R, IDL ...
Koppes / Geography 409 Introduction to data analysis
23
Your data analysis?
Think about your potential project: Instrumentation, calibrations? Data checks? Analysis concept? Software needs? Hardware needs? again: Keep it simple!
Koppes / Geography 409 Introduction to data analysis
24
Documentation : Meta-data
• Document your ideas and plans → proposal.
• Document your instrumentation (specifications, manufactures, serial numbers).
• Document your sampling strategy (how, where and when...)
• Document your data files (parameter, units, time-zone, location)
• Document your analysis (data filtering, data selection criteria, statistical methods)
• Document and prove your conclusions → report
log- or field note book