dr lisa wise 20/09/2002 designing web surveys dr lisa wise
TRANSCRIPT
Dr Lisa Wise 20/09/2002
Designing Web Surveys Dr Lisa Wise
Dr Lisa Wise 20/09/2002
Overview
• web surveys• overview of survey design • when to use surveys• how to design survey questions• data analysis and reporting
– data be collection, storage and analysis
– ethical issues to do with handling sensitive or personal data
Dr Lisa Wise 20/09/2002
Web-based Surveys
• can do from anywhere with web access – but samples are biased against those with no
internet access (> 50% of Aust households are connected - AC Nielson Mar 2002)
• data can be “validated” before submission• note that validation is in terms of correct data
type rather than meaningfulness• issue of multiple submissions vs anonymity
• quick data collection phase for researcher
Dr Lisa Wise 20/09/2002
Web Survey
web server
databasesurvey form processing scripts
emailSurvey data might go to adatabase or might be storedin a file in the file systemor could be emailed to the researcher
filesystem
Dr Lisa Wise 20/09/2002
Web survey design
• use standard web design principles • especially ensure that
– survey is not too long (or can be saved)
– information travels correctly between screens
– error messages, if they occur, are meaningful
– all fields are validated and can handle appropriate input and reject inappropriate input
– check field length, data type, illegal characters
Dr Lisa Wise 20/09/2002
Making a web survey
• put the questions into an HTML form• get the data formatted for analysis using
the scripting tools of your choice• set up access controls if required• test across a range of browsers with a
range of good and bad input– ensure security / integrity of your data, but also
of the system collecting your data
Dr Lisa Wise 20/09/2002
Data collection and storage
• use cgi-mailer or FormMail to email formatted form variables to researcher– http://www.its.monash.edu.au/web/resources/cgi-mailer.html
– http://www.its.monash.edu.au/web/resources/formmail.html
• use a script (eg Perl, PHP) to write to a file on the filesystem
• use a script to put data into a database– don’t put into a database if you are then going
to take it out into SPSS / excel to analyse it !!
Dr Lisa Wise 20/09/2002
Privacy
• “Privacy laws impose quite specific requirements regarding data storage and security, access and correction rights, ensuring data is accurate before use, used only for the purpose for which it was collected or for which consent has been given, and disclosure only in limited circumstances.”
• “Legal requirements may be more detailed than ethical requirements …”
• http://www.monash.edu.au/resgrant/human-ethics/privacy/index.html
Dr Lisa Wise 20/09/2002
Handling data
• shared responsibility of technical staff and researchers to ensure that privacy and confidentiality requirements are met– “information is only de-identified only if all
identifying information has been irreversibly removed from the record”
– “retention of codes which allow recovery of identifiable information means record is not de-identified”
Dr Lisa Wise 20/09/2002
Cost effectiveness
• even a simple web survey takes a couple of days to implement and test
• cost-effectiveness of solution needs to include real costs– eg email vs database solution might involve 3
days of coding for programmer versus 1 day of data entry (cutting and pasting from email to analysis program) for researcher
– cost saving or cost shifting?
Dr Lisa Wise 20/09/2002
Making a survey ...
• technical “design” requirements covered in previous slides but what about designing the actual survey questions ?
• there is a whole discipline area focused on designing surveys and questionnaires and analysing data collected using these tools
• it is not a trivial exercise and committees / managers are not ideal survey designers
Dr Lisa Wise 20/09/2002
Overview of survey design
• survey design requires that you have clear research questions
• survey questions need to be focused on answering your research questions
• survey design includes generating the questions and planning the data analysis
• planning the data analysis happens before the survey is released !!
Dr Lisa Wise 20/09/2002
When to use surveys
• surveys allow you collect lots of data relatively quickly and cheaply
• lots of poor quality data is never better than smaller amounts of high quality data
• lack of time and money are not excuses for collecting poor quality data - if you can’t afford to collect reasonable quality data, don’t do the research.
Dr Lisa Wise 20/09/2002
Personal data
• most surveys request demographic information about respondents
• usually ask for opinions about something• collecting either type of information has
ethical and privacy implications• survey designers should be familiar with:
– http://www.monash.edu.au/resgrant/human-ethics/
– above page has link to Use of Personal Information
Dr Lisa Wise 20/09/2002
… from Privacy statement on web
“On-line SurveysAll research surveys conducted on-line by
University staff and /or students which involve the collection of personal information, will have received approval from the University's Committee for Human Ethics in Research.
A survey might ask visitors for unique identifiers (such as login information).”
Dr Lisa Wise 20/09/2002
Sensitive information
• “Do the records or information you are collecting or using include any sensitive information (such as political opinion or memberships, religious beliefs or affiliation, philosophical beliefs … )”– note that peoples’ opinions are considered to
be personal information
– be aware of perceived power relationships and potential access to confidential information
Dr Lisa Wise 20/09/2002
Classroom and workplace surveys
• work / class surveys have ethical issues related to perceived power relationships between respondents and researchers
• even if survey does not require specific ethics committee approval, the ethical and privacy principles should be considered
• go through all the ethics forms whether or not they need to be submitted
Dr Lisa Wise 20/09/2002
SCERH ethics forms
• based on National Statement on Ethical Conduct in Research Involving Humans (NH&MRC, 1999)
• apply to “anyone who is gathering information about human beings and organisations through interviewing, surveying, administering questionnaires, observing human behaviour, taking human tissue / fluids …”
• “ … there are no exceptions, exclusions or blanket permissions”
Dr Lisa Wise 20/09/2002
Sampling
• Sampling design and survey design must be tightly coupled
• Who or what are you planning to draw conclusions about ?
• Are they a homogenous group or are there sub-groupings in the population ?– do you need a stratified sample?
– do you want to compare between groups?
Dr Lisa Wise 20/09/2002
Sampling
• Surveys usually use convenience samples• Demographic information is collected to
allow comparisons between target groups • If targeting specific groups is critical to
your research, consider interviewing participants, or delivering and collecting surveys from selected participants
• convenience sample =/= random sample
Dr Lisa Wise 20/09/2002
Sampling
• for statistical techniques, calculate the sample size required for valid conclusions
• consider whether you want lots of general data or whether you are actually interested in very specific focussed data
• a large survey doesn’t give more objective data than eg a focus group unless it follow rigorous methodology
Dr Lisa Wise 20/09/2002
Types of questions
• Open ended– more difficult to answer and to code or to
analyse objectively
• Closed questions– forced choice (one of two mutually exclusive)
– multiple choice (one of several)
– checklist (one or more of several)
– partially closed (alternatives including “Other”)
Dr Lisa Wise 20/09/2002
Questions
• avoid double-barrelled questions• avoid leading questions• avoid motherhood statements• avoid undefined terms• ensure that your questions lead to
responses that interest you and conversely that responses that interest you are elicited by your questions
Dr Lisa Wise 20/09/2002
Order effects
• funnel questions from general to specific• can use general filter questions to
determine whether respondent should be asked detailed questions
• can have some practice questions• counterbalance order of presentation• prevent response sets• can use alternate forms of questions
Dr Lisa Wise 20/09/2002
Content analysis
• talk to expert about content analysis• consider whether you are happier with the
assumptions underlying content analysis or your research team’s ability to interpret and code responses
• if you code responses, take measures of inter- and intra-rater reliability – do raters make similar / consistent judgements
Dr Lisa Wise 20/09/2002
Response scales
• response scales should allow people to communicate what they want– eg there are differences between neutral,
undecided, don’t know, don’t care, n.a.
• anchor points should be bipolar– need to test this on pilot sample
boring interesting
boring fun
Dr Lisa Wise 20/09/2002
Measurement scales
• Ratio scales (true zero - eg age, height)• Equal Interval scales
– Thurstone scales, Likert scales, Guttman scales, semantic differentials
• Ordered scale (eg <18, 18-30, 31-50, >50)• Categories (ITS, Med, Sci, Arts …)• Different types of data are amenable to
different analyses: consult a statistician!
Albrecht et al, Social Psychology, pp 190-198
Dr Lisa Wise 20/09/2002
Rating scales
stronglydisagree
stronglyagree
undecided
Likert scales should have 5 marked values not 7
The following multiple choice format is still an interval scale1. Strongly disagree2. Disagree3. Undecided4. Agree5. Strongly agree
disagree agree
Dr Lisa Wise 20/09/2002
Rating scales• constructing rating scales to have specific
psychometric properties is labour-intensive and requires expertise– multiple forms or minimal number of questions?
– group similar questions or intermix?
– alter format to avoid response bias or keep consistent to avoid response errors?
– normalise data or trust participants responses?
• consult a statistician or social scientist
Dr Lisa Wise 20/09/2002
Measurement
• rating scales should be equal-interval scales to use parametric statistics
• the fact that you have numerical data does not mean it is accurate or reliable– male (1) - female (0) … obvious that 0 and 1
are codes not numbers
– rarely (1) - sometimes (2) - often (3) … things that can be ranked are not necessarily equal interval scales - can you take the mean?
Dr Lisa Wise 20/09/2002
Statistics
• Descriptive– describe a sample distribution in terms of
shape, central tendency and variance
• Inferential– draw inferences about population parameters
based on sample statistics
– hypothesis testing - test whether a statement is true or false based on sample statistics
Dr Lisa Wise 20/09/2002
Reliability and validity
• Reliability– test - retest
– surveys are only a snapshot at particular time
• Validity– face validity
– criterion-based validity
– construct validity
– internal / external validity
Dr Lisa Wise 20/09/2002
Hypothesis testing
• null hypothesis H0 (no effect of experimental manipulation)
• alternative hypothesis H1 (experimental manipulation has an effect on this test statistic)
• type 1 error (accept H1 when H0 true)
• type 2 error (accept H0 when H1 true)
Dr Lisa Wise 20/09/2002
Inferential statistics
• use statistics to make inferences about populations from your samples
• need to be aware of assumptions underlying statistical tests– eg t-tests and anovas assume continuous
underlying variable, normal distribution and homogeneity of variance
– what happens when assumptions are violated?
Dr Lisa Wise 20/09/2002
Statistics
• most survey data is not ratio and may not be interval data (depending on your perspective on this)
• non-parametric statistical tests don’t make assumptions about underlying distribution
• don’t use statistics to show things you can’t see by eye - statistics help decide if what you can see is “significant”
Dr Lisa Wise 20/09/2002
Correlational studies
• surveys usually correlate opinions with demographic information
• correlations show relationships between variables but don’t address causality
• correlation coefficients indicate strength of relationship (between 0 and 1)– r of .8 explains 64% of variance, or the degree
to which knowing X can predict the value of Y
Dr Lisa Wise 20/09/2002
But I’m not doing real research ...
• I just want to run a quick survey• This stuff doesn’t apply to me cos I’m not
doing real research
• . . . so what are you doing?– “anyone who is gathering information about human
beings and organisations through interviewing, surveying, administering questionnaires, observing human behaviour, … no exceptions ...”
Dr Lisa Wise 20/09/2002
What constitutes research?
– If you are going to summarise responses on your survey and you are going to act on them in any way, you are doing research
– for that research to be meaningful, you need be aware of proper research methodology and the limitations of what you are doing
– if you can’t do it properly, don’t do it at all !!
– informed professional opinion can far more valuable than poor quality survey data
Dr Lisa Wise 20/09/2002
When to use web surveys
• small amount of non-confidential data from wide range of people– not good for sensitive or confidential data
– not good where loss of data would be a major problem
– not good for long surveys (usability issues)
• can use web to download a survey which is printed out and submitted