the questionnaire - web viewthe questionnaire will often form part of a larger piece of work. this...

05 May 2023Final V1.

Birmingham Public Health

Analysis for Consultations and SurveysRough Guide

Contents1 The Questionnaire.......................................................................................................................4

2 Timeframes.................................................................................................................................4

3 Responses....................................................................................................................................5

3.1 Third Party Agencies..........................................................................................................5

3.2 Late Responses..................................................................................................................6

3.3 BeHeard............................................................................................................................6

4 The Data......................................................................................................................................7

4.1 Data cleaning.....................................................................................................................7

4.2 Validation Rules.................................................................................................................7

4.3 Missing data......................................................................................................................7

4.4 Weighting..........................................................................................................................7

5 Analysing the data.......................................................................................................................8

5.1 Data from open questions.................................................................................................8

5.2 Coding open questions......................................................................................................8

6 Data Reporting............................................................................................................................9

6.1 Report Template...............................................................................................................9

6.2 Formatting.........................................................................................................................9

6.3 Reporting quantitative data / statistics.............................................................................9

6.4 Reporting qualitative data...............................................................................................10

6.5 Summary.........................................................................................................................10

6.6 Appendices......................................................................................................................10

A Basic Guide to Analysing Consultations using BeHeard....................................................................12

Coding Framework..............................................................................................................................21

Analyst Guide to ConsultationsJanuary 2015 Birmingham Public Health Information & Intelligence Team

Consultation Analysis

A Basic Guide

Birmingham Public Health carries out a series of surveys, questionnaires and consultation exercises throughout the year, for a variety of purposes - evidence health issues, map current services or assets, consult the public on service changes, etc.

The questionnaire will often form part of a larger piece of work. This is especially true for consultations which may involve user engagement, workshops, public forums, mailings and more.

1 The Questionnaire

All engagement and consultation questionnaires must:

follow the Consultation Good Practice and Data Protocols final approved by Deborah Towle & Charles Ashton Grey be published on BeHeard collate all responses on BeHeard (i.e. all completed hard copy questionnaires must be

manually inputted into BeHeard)

It is recommended that the person responsible for writing the Questionnaire speaks to the Data Analyst to ensure the questionnaire can be analysed appropriately.

Sometimes an external agency(ies) may be commissioned to gather additional questionnaire responses. In these instances, the agency must enter each completed questionnaire onto BeHeard.

1.1 Questions

Be mindful of the adding too many open ended questions

2 Timeframes

Timeframes are set by the Project Lead as part of the project plan. However, as a guide the main milestone dates should be as follows (set/mandatory timeframes are in red).

Stage Days Allowed Comment

Survey Development 30 days – discussions / design10 days – engagement tender advert

& shortlisting10 days – approved by DT & CAG10 days – develop easy read

questionnaire

Many elements to consider and not all are covered in this list e.g. consultation documents and model need to be written.All questionnaires also need to be approved by information Governance before they can proceed. An easy read version of the questionnaire and

Analyst Guide to Consultations Page 4 of 23January 2015 Birmingham Public Health Information & Intelligence Team

http://inline.birmingham.gov.uk/cs/Satellite?c=Page&childpagename=Policy%2FInlinePageLayout&cid=1223092538810&pagename=BCC%2FCommon%2FWrapper%2FInlineWrapper

Stage Days Allowed Comment

consultation papers also need to be developed

Survey Run Time 60 Days There isn’t a minimum but it should be sufficiently long enough to allow people to respond to it. The BCC recommended time for a survey is 60 days. However, a case can be made for longer or shorter.

Postal entries 3 – 5 days Postal entries will be sent during the consultation and will need to be entered onto Beheard. Once survey closes, allow three days for postal entries to arrive and be entered onto the system (by admin).

Analysis & Report 5 - 10 days After the postal entry lay by (above), the data can be downloaded and formally analysed. The report takes a minimum of 5 days for a final draft but may take longer depending on complexity and volume of responses.

3 Responses

Submitted Questionnaires

Submitted questionnaires are made via BeHeard online survey, by freepost and collated through workshops or focus groups

Hard Copy Questionnaires

All hard copy questionnaires arriving by post or collated through workshops/focus groups must be entered onto BeHeard by the admin team, as written.

Third Party Agents All questionnaires gathered by third party agents (where they have been commissioned to do so) must enter the submitted questionnaires onto BeHeard, verbatim.The third party agent must enter their own email address when entering onto BeHeard so these questionnaires can be identified and credited to them (an email field must be added to the questionnaire).Once entered onto BeHeard, any hard copy questionnaires must be forwarded to the Project Lead

Workshops / Focus Groups

A combined response from one group maybe entered onto Beheard. However this may impact on the response numbers and demographics as the submission will show as one response.It may be more appropriate to pass this response to the Data Analyst who will summarise the key points and show in the report under a separate heading.In both cases, the number of people consulted and demographics for the group should be included in the response.

Organisations or groups Large organisations or groups may also submit a formal response. These cannot always be entered into BeHeard as they may not follow the questionnaire format. In these cases, the response should be given to the Data Analyst who will summarise the key points from the response and include in the report as a separate heading.


3.1 Third Party Agencies

Sometimes it is appropriate to commission a third party agency(ies) to engage with specific target groups e.g. substance misusers, homeless etc.

The Third Party Agency(ies) is usually permitted to use any method they feel appropriate to engage the target groups (e.g. money incentive, survey monkey, focus groups, cold marketing). However, all agencies must use all the questions from the survey questionnaire (exactly as written) and input the responses into BeHeard. Paper questionnaires or anecdotal responses are not permitted.

Agencies should include their own email address when adding submissions onto BeHeard (as well as the respondents email if appropriate) to identify questionnaires submitted by the Agency.

Where focus groups are being held by the Agency, it may be appropriate for the Agency to provide a separate report on any comments raised that are not covered in the survey. This report must be submitted by survey close so it can be summarised and incorporated into the main Report if appropriate.

It is not expected for a third party agency to provide any analysis on questionnaire responses, other than general feedback from any events.

3.2 Late Responses

The online survey closes at midnight.

Postal entries sent on the last day of the survey duration must be included in the report. Time must also be allowed for any additional questionnaire inputting required by the Admin Team.

It is recommended at least 3 working days are set aside after the survey close to allow time for postal entries to be received and any backlog of hard copy questionnaires to be entered onto BeHeard by the Admin Team.

Any questionnaires received after this time cannot be entered onto BeHeard. Instead, they should be passed to the Data Analyst who will:

Count the number of late submissions and include a line to this effect in the Report o e.g. 10 responses were received after the deadline and have not been included within

the response totals or demographic information. However, any comments raised have been reviewed and incorporated where appropriate.

Review any comments made and incorporate into the Summary response where appropriate

3.3 BeHeard

For instructions on how to use BeHeard, see Appendix.

It is important that each questionnaire is analysed on BeHeard whilst the survey is open on-line. This will allow the Analyst to identify any gaps in responses (e.g. low number of responses from men) and for the Project Team to then target their engagement.


Every submission is analysed question by question, and notes made in the appropriate box for that question (see appendix)

Do not leave this analysis to the survey close.

4 The Data

The Data Analyst will analyse each questionnaire via BeHeard – paper questionnaires will not be included in any analysis. All questionnaires must be entered onto BeHeard by the administration team.

The Data Analyst will be responsible for:

4.1 Data cleaning

Data cleaning is the process of dealing with (or eliminating) invalid data. It can happen either before data entry (where questionnaires with obvious errors are dealt with) or after data entry by manipulating the data in the database. In order to clean data, we need to set validation rules.

DATA SHOULD NOT BE CLEANED WITHOUT CONSULTATION WITH THE ANALYST

4.2 Validation Rules

There will always be respondents who misunderstand the question, answer in the wrong box, or give you a response you were not expecting.

Validation rules are put in place to deal with these inconsistencies and must be applied to all questionnaires. For example, you ask “what day is your rubbish to collected on?” and give Monday to Friday as options. The majority may tick one box, but some respondents may have two collections – e.g. one for recycling, one for general – and tick two boxes.

To deal with this double-ticking, you can:

Omit the response completely Re-code the question so that double ticks are allowed and assume that those who didn’t

double tick did so because their rubbish was collected on the same day.

Neither is perfect, but whatever one you choose, you must apply consistently to every questionnaire response.

4.3 Missing data

We cannot assume anything from missing data (e.g. where a box has not been ticked) and therefore the field is left blank and not recorded. Where providing statistical analysis on a question, always annotate the number of responses.


4.4 Weighting

We do not apply weighting to question responses.

5 Analysing the data

Once the survey is closed, and all postal submissions have been entered, a data report will be downloaded from BeHeard and the data analysed.

The final report including all findings and conclusions will be made available working 5 days after the postal close date, unless agreed otherwise. This is a suggested timeframe and will depend on the number of responses received, their complexity and any other consultations taking place.

5.1 Data from open questions

Open questions may be treated in a variety of ways, depending on the type of open question, and what you wish to use the data for. Numerical or short answer open text questions can be treated as follows:

(a) Open numerical questions (quantitative)

These will include demographics or frequency, etc. This is simple numerical data and can be analysed statistically to produce means, modes and medians, etc. This data should be converted to a consistent form before analysing (e.g. age and months converted into months). Quantitative data is presented in pie or table format (see 6.3)

(b) Short open data from “other” categories (qualitative)

These need to be coded into common themes/categories and reported. This is normally reported as an addendum to the question (e.g. the responses to the question are reported in a pie chart, and the responses to the “other” categories are expanded below it, as text or an additional chart).

5.2 Coding open questions

This can be a long process so you should ensure you check questionnaires daily throughout the course of survey, and code them as you go. Essentially the aim is to aggregate responses to open questions under a series of themes or “codes”. There may be several codes applied to a single response. For example:

Response Coding applied

I was waiting for the bus. It was raining pretty hard and I slipped. I fell over a broken paving slab and had to call an ambulance. My arm hurt pretty bad. I waited for over 20 mins.

FallWet pavementPavement in disrepairAmbulance attendedAmbulance 20 mins response


(a) Create a coding frame

Read through the first 50 or so responses and get an idea of what issues are cropping up in open questions. Write them down (I use Excel) under the question headings and give them some sort of structure or hierarchy. For large coding frames or high number of responses, it may be appropriate to add a number to each of the codes.

(b) Code the remaining questionnaires

Read through the remaining questionnaires and apply the relative code to each response. Refine your frame as you go, adding new codes when necessary.

For more about Coding read my second guide GUIDE Qualitative Analysis Coding.

6 Data Reporting

6.1 Report Template

Use the Consultation Report Template.

In each section include:

Executive Summary – This is added after the report is completed and will include key information from the Introduction, Overview and Summary.

Introduction – this is taken from the overview page of the Consultation on BeHeard Data Sources and Scope – include number of responses received, any additional services

commissioned, details of how audiences were engaged, etc. Key findings - list the questions and report the data under each question Summary - overall conclusion and summary of findings Appendix – include detailed data and copy of the questionnaire

6.2 Formatting

Contents page is automatically updated Use heading styles appropriately All charts, tables and maps are to have caption label Pie charts are used or % data, bar graphs for number data Include number of responses for each question

6.3 Reporting quantitative data / statistics

Graphs and pie charts are used to present all quantitative data (i.e. from closed questions). All graphs and charts must have a clear title, data labels and a caption.

Consultations tend to stand alone, so it is unlikely we can compare statistics between reports (e.g. last year X% people stated …, this year Y% …). However, if you can or need to, consider using a significance test to make sure there is a real difference.


Be consistent when reporting figures (e.g. in the number of decimal places). Numbers and percentage should be rounded to nearest whole number wherever possible.

We do not weight figures. Always include the number of responses. All percentages are against total number of responses. Where reporting on sub-

categories, always use numbers rather than percentages (e.g. when reporting on “other”) and state number of responses in total.

Detailed statistical data should be put into the Appendices.

6.4 Reporting qualitative data

Report as numbers rather than percentages. Beware of bias Always use in context Highlight any issues with the data Provide explanation of any coding or comments Put detailed comments in the Appendices

6.5 Summary

Summarise the key statistics raised in the report. The purpose of this report is to present the data, not make recommendations or draw

conclusions. Comments should clarify the data (e.g. put it in context), highlight any bias or draw attention to any gaps in data/areas for further investigation.

6.6 Appendices

This report may be published. Do not include identifiable data (e.g. full email addresses or postcodes).

Include maps, additional charts to expand on question data and full comments. Include a copy of the questionnaire


Analysing Consultations using BeHeard

A Basic Guide to Analysing Consultations using BeHeard

BeHeard is the on-line platform used for all Birmingham City Council consultations. This is accessed by the Public via the BCC website.

You must undergo training with Delib (the company who designed and manage the site) before you get a log-on.

Getting Started

Once you have training, you will get a log-on.

Go to www.birminghambeheard.co.uk/login

Enter your user name (usually your email address) . Type exactly as given, paying attention to capital letters.

Enter password

Select Log In

Page 12 of 23 Birmingham Public Health Information & Intelligence

http://www.birminghambeheard.co.uk/login

Find Consultation

This is the consultation screen

The most recently added consultations (for the whole of BCC) will appear here

To find a consultation, click on Find Consultations

Type the consultation title (or part of) into the Search for Consultation box .

Click on Search

Note: If you have already been added as an analyst, the consultation will appear in the Analyse Consultation list .


A list of results will appear.

Please note, be specific. The search will look at all consultations across BCC, both open and closed. If you have used a common word (as in the example ) you will get back a long list of results.

Find the one you want and click on the link.

It will open the consultation in a new window.


Consultation Page

This Overview page is what the Public sees.

To analyse the responses, click on Consultation Dashboard


Consultation Dashboard

Status- open (viewable by the public), closed (submissions can’t be made) or not published (not posted online). You can also publish or retract the survey here . Please note, retracting or altering questions will affect data already submitted.

Add yourself as an Analyst for the survey to appear in your Analyse Consultation list

Number of responses .

Request a summary report from BeHeard. This shows number of responses for each question. Please note this will only show tick boxes responses, not individual comments.

Number of responses analysed by you on BeHeard

View each submitted questionnaire response and analyse responses

Download the responses into an excel document here to create charts for the final repot – always include the analyst fields so any comments you have made on the questionnaires are included


Click on Interpret the Responses (on the previous screen shot ).

Analysing the Responses

Only one submitted questionnaire is opened at a time.

Unique identifier number for this submission . You can use this to find this particular submission again.

Number of responses and number left to be analysed

This is a mandatory field. You must tick this box when you have analysed the responses


Read through the responses one by one.

Where the respondent has made comments, summarise the comment in the analyst field.

Be consistent when summarising comments (called “coding”) and keep a note of the phrase used (called a “coding frame”). See example at end.


You can also use the Analyst Field to correct mistakes make in the response. For example, in the response shown, the respondent has not ticked a box but did state they had fallen under “other” .

The correct response has been noted in the Analyst Field .

When completed, all the questionnaire data will be downloaded into excel and the Analyst can then amend the data to show the correct response.


When the questionnaire response has been analysed, click on save

Click on Next Unanalysed for the next questionnaire submission


Coding Framework

Below are some examples of a coding framework. Each text comment is summarised in a short phrase or theme (called a code) and recorded here under the relevant question. When analysing the questionnaire responses, the analyst will refer to this frame and use the appropriate phrase or code to summarise the comment. This allows the analyst to put the comments under headings and provide some analysis of common themes. The coding frame can be added to as needed. See document “Coding Surveys” for more.


of 23 Birmingham Public Health Information & Intelligence

the questionnaire - web viewthe questionnaire will often form part of a larger piece of work. this...

Documents