request for proposal intermediate-level professional ... · 3a. create a course outline using...

Request for Proposal

Intermediate-Level

Professional Learning Program

(Stage Two)

Course Development

November 20, 2015

Proposals are due to the CES by 5 PM EST on Fri. Dec. 11, 2015. Bidders are asked to submit

an email indicating their intent to submit a proposal by 12 Noon EST on Mon. Dec. 7, 2015 to

the CES Executive Director. See Section 3.1 for instructions.

Questions on this RFP are welcome before 12 Noon EST on Mon. Dec. 7, 2015. Please see

Section 6.0 for further detail.

.

1

Request for Proposal: Intermediate Learning Course Development Canadian Evaluation Society ~ November 2015

Table of Contents

1.0 Background 2

2.0 Project Overview 5

3.0 Proposals 8

4.0 Selection Criteria 10

5.0 General Conditions 12

6.0 Questions and Inquiries 14

7.0 Appendices 15

2


1.0 Background

The delivery of quality professional development training opportunities is a main service offering

of the Canadian Evaluation Society1 (CES) to its members and represents a core revenue

stream for the national Society and its 11 Chapters across Canada. The purpose of expanding

the range of courses at this time is to provide better service to CES members across Canada by

increasing access to intermediate learning at affordable prices and reducing participant and

instructor travel costs, as identified in a member survey on training needs2. To that end, CES

wishes to expand its range of professional learning courses to include new content for learning

at the intermediate level.

The primary audience for the new intermediate courses is the CES Membership composed of

roughly 1,800 professional evaluators from all parts of Canada and around the world, working

exclusively or in part on evaluation and monitoring in all sectors and levels of government, non-

governmental organizations, and private sector. It is envisioned that additional non-CES

learners will also be able to access the online courses.

In February 2015, the CES awarded an Intermediate Professional Learning - Stage One

contract for the design of a plan of seven new intermediate-level3 courses4 to supplement the

existing trio of National courses.5 Under the terms of the Stage One contract, the contractor

1 The CES is a not-for-profit organization, which is structured as a virtual model overseen by a national Board of

Directors, supported by an administrative Secretariat in Ontario and managed by a part-time Executive Director in

British Columbia. A network of Professional Development representatives from each CES Chapter collaborates with

the CES Professional Learning Committee for planning and delivery of courses. http://www.evaluationcanada.ca/our-

structure.

2 Roy, S.N., Kishchuk, N., Gauthier, B., Borys, S. (2013) Survey on Evaluation Training Needs and Preferences.

Paper presented at the CES Conference, Toronto, June 10, 2013.

http://evaluationcanada.ca/distribution/20130610_roy_simon_kishchuk_natalie_gauthier_benoit_borys_shelley.pdf

3 For the purposes of this project, the term “intermediate-level training” has been defined as (a) providing in-depth

information and hands-on learning and (b) requiring some basic knowledge of the specific content area as a pre-

requisite but not in-depth knowledge or skills. Intermediate-level training should focus on knowledge or skills that

build on basics.

4 The term “courses” is used to indicate learning opportunities equivalent to half a day to two days in duration (e.g.,

face-to-face or online workshops) that do not carry university or college credit. These courses are designed to

support evaluators in developing competencies at the intermediate level. Course material may be original or

reworked according to CES requirements and Canadian evaluation context. An expectation of evidence of learning

will be included within each course design.

3


conducted an environmental scan on existent evaluation courses in focus areas

identified by the CES 2013 survey on professional learning;

reviewed the results of the 2013 CES membership survey on professional learning;

consulted subject matter experts (SMEs);

examined methods of e-learning and online learning;

prepared a report outlining the content of seven courses;

made recommendations for seven new online6 or face-to-face courses; and

presented a two-year timeline for implementation (2015 – 2017).

An abridged version of the Stage One report is provided as Appendix A.

The present Request for Proposal (RFP) provides the terms of reference for the contract(s) to

develop four of the new courses by May 30, 2016:

Evaluability Assessment – approx. 3 hours, online learning7

Data Quality Assurance – approx. 3 hours, online learning

Interpreting and Using Quantitative Results for Evaluation – approx. 2 days/12 hours, in

person

Evaluation Theories and Models – approx. 6 hours, online learning

Course development notes from the Stage One project are provided as Appendix B. These

include descriptions of the target audience, prerequisites, competency domains, and learning

objectives for each course.

5 The introductory Essential Skills Series (2014) and the Intermediate Logic Models workshop (v. 2012) and the

Survey Methods course (2012). At present, the CES-sanctioned courses are coordinated and delivered face-to-face

by CES chapters across Canada. The chapters engage qualified instructors who are familiar with the materials to

provide training for registered participants. CES receives a fee per workshop, collected by the local chapter, in

exchange for the provision of the course materials in electronic form and certificates of completion following the

delivery of the course(s).

6 The term “online” is used to indicate that the course can be delivered, completely or in part, using an

online or electronic medium such as Citrix Webinar, Moodle, Blackboard, Desire to Learn, Velsoft, or

other platform.

7 The length of the course and the mode of delivery have been suggested as a guideline. Bidders may

submit bids that vary from these suggestions but maximum bid levels are firm.

4


As per recommendations in the Stage One report, the CES has contracted an Instructional

Designer/Project Manager (IDPM) to provide project management services, instructional design

training and support, course design templates, technical assistance and quality control for

course development to each of the contractors engaged to develop the individual courses.

5


2.0 Project Overview

2.1 Purpose and Objectives

The purpose of this RFP is to solicit proposals for contracted assistance with Stage Two, the

design of four new intermediate-level courses on behalf of the CES. Bidders are not required to

bid on all four of the courses as a package. Bidders may submit proposals for one, two, three, or

four of the courses. Requests for proposal for the remaining courses are to be released in the

summer of 2016 and are not included in this RFP.

The courses will be based on the guidelines included in the Phase One consultant’s report

(Appendix A) using a combination of face-to-face instruction and the online learning platform

selected by the CES.

2.2 Scope of Work and Deliverables

Figure 1 presents a relationship model illustrating the context for the role of ID/PM within the

Stage Two course development project. Table 1 presents the tasks and deliverables required

for each course development contract.

The designer of each course will be expected to complete a pilot test of the newly designed

course, with instructional plan and associated resources pre-approved by the ID/PM, and make

final revisions to the course based on pilot test feedback. The CES Professional Learning

Committee, in consultation with individual course developers, will decide the location of each

pilot test. Since the designer of each course will not subsequently be the exclusive presenter of

the course, the designer must ensure that, once finalized, the course materials are specific

enough for effective presentation by other facilitators.

6


Figure 1. CES Intermediate-Level Professional Learning Program (Stage Two) Concept Plan

2015 – 2017.

The project requires the contractor to perform the tasks and submit deliverables as shown in

Table 1.

Table 1. Tasks and Deliverables.

Item Tasks Deliverables Dates

1a. Participate in introductory

teleconference training

provided by the ID/PM

Completed orientation and

training. Jan. 11- 22, 2016.

1b. Prepare a detailed work plan

for the development of the

course including design,

resource production, online

Detailed work plan and names of

the two SME advisors due within

three weeks of the award of the

contract with any subsequent

Four (4) courses in 2015/16 and three (3) courses in 2016/2017.

7


formatting, pilot test, and

revision.

revisions subject to the approval

of the designated ID/PM or CES

liaison. 1c. In consultation with the ID/PM

and CES liaison, establish a

two-person advisory panel

with other SMEs to provide

feedback on the course

development process.

2. Engage in Skype, Citrix,

telephone and/or email

communication with the

ID/PM and SME advisors at

least once a month per

course during the term of the

contract to give and receive

feedback on the course

development.

Record of dates and purpose of

communication.

3a. Create a course outline using

guidelines provided by the

ID/PM and Module 1 lesson

plan including interactive

learning activities, specific

CES competencies

addressed, associated

resources, and evidence of

learning for review by the

ID/PM.

A complete course outline and

Module 1 submitted to the ID/PM.

by Feb. 8

Feb. 8, 2016

3b. Create lesson plan(s)

including interactive learning

activities, specific CES

competencies addressed,

associated resources, and

evidence of learning for the

delivery of a pilot test. Format

materials in online format for

courses so indicated in

Section 1.0.

One complete set of course

materials for the pilot test due with

the endorsement of the ID/PM

and the two SME course advisors. April 11, 2016

4. Conduct pilot test. Instructor and learner feedback. Apr.12 – 30, 2016

5. Prepare a final version of the

course materials based on the

instructor’s perceptions and

feedback from the pilot test

participants, the CES

Professional Learning

Committee via the designated

liaison, and the ID/PM.

Final set of course delivery

materials in pdf or online form

depending on the requirement

specified in the RFP. May 30, 2016

8


3.0 Proposals

3.1 Submission

Bidders are asked to submit an email indicating their intent to submit a proposal by 12 Noon

EST on Mon. Dec. 7, 2015 to the CES Executive Director via

[email protected]

The bidder must submit an electronic copy of the proposal to the CES Executive Director via

[email protected] by 5 PM EST on Fri. Dec. 11, 2015. Acceptable

formats for the electronic version are *pdf, *doc or *docx. The name of the course or courses

for which the bid is submitted must be clearly identified in the file name. Short acronyms have

been suggested for the bidder’s convenience:

Evaluability Assessment (EA)

Data Quality Assurance (DQA)

Interpreting and Using Quantitative Results for Evaluation (QRforE)

Evaluation Theories and Models (ET&M)

3.2 Length

Proposals should not exceed 10 pages, single-spaced with 1-inch margins and 12-point font

(approximately 3,000 words excluding any references and appendices).

3.3 Identification of Bidders

In the event that a team submits a bid, one individual must be identified as the principal bidder

on behalf of the bid. All further communication with CES regarding the proposal and the

contract, if awarded, will be via this individual. Further, if a proposal is submitted by more than

one person, all participating individuals must be clearly identified in the bid proposal and a short

Curriculum Vitae (CV) must be included for each person named. Bids must identify the

academic and/or practical qualifications, competencies, and experience of the bidder(s) as they

relate to the identified scope of the project.

The names and contact information of three references able to comment on the abilities of the

bidder or team with respect to subject matter knowledge, course development, and teaching

experience must be provided. Please note that CES may contact some, none, or all references

as it determines appropriate.

mailto:[email protected]

9


3.4 Eligibility of Bidders

This RFP is open to all bidders who meet the bid requirements indicated in this document with the following exception affecting a company in which a Board member or Board member’s spouse, child, or grandchild is an employee or has an ownership interest. In this case, an employee of the company is eligible to respond to contracting opportunities provided that the Board member is not involved in any phase of the contracting process, does not use his/her position to influence a decision in the awarding of the contract, and declares the conflict of interest to the Board prior to the submission of a proposal.

3.5 Proposal Content

Each proposal must address the bidder’s

Subject matter knowledge for the course(s) on which the proposal(s) are based;

Understanding of the role of the course developer in this project;

Understanding of the scope of this project;

Willingness to work with an ID/PM on the instructional design of the course(s);

Willingness to obtain training in the online learning program to be chosen for this project

(if it is indicated for delivery entirely or partly online);

Training and/or experience in instructional design;

Training and/or experience in delivering face-to-face learning modules for professional

practitioners at the intermediate level or training and/or experience in delivering online

training, blended, and/or e-learning courses;

Sensitivity to the Canadian evaluation context;

Proposed work plan (including proposed course development process, timelines,

services, associated level of effort, and costs) to complete this project; and.

Insurance coverage, if any, for general liability and/or errors and omissions.

10


4.0 Selection Criteria

4.1 Mandatory Requirements

Each proposal will be assessed based on the criteria indicated in Table 2 provided the following

criteria are met:

Proposal is received by 5 PM EST on Fri. Dec. 11, 2015.

Price is at or below the amount(s) shown in Section 5.5 (exclusive of taxes but inclusive

of all expenses). Inclusion of additional items as separately priced bids are acceptable

should the bidder wish to propose one or more ideas beyond the scope of this RFP for

consideration. No travel is expected as part of this contract.

English is the primary language for this contract. The CES Board of Directors anticipates

undertaking the translation into French of each course, following its development, pilot

test, and completion in English. However, the translation of the courses is not included in

the scope of this project.

The length and format of proposal as per Section 3.2 of this RFP.

Each team member is identified if the proposal is submitted on behalf of more than one

person.

A team leader is identified if the proposal is submitted on behalf of more than one

person.

4.2 Evaluation of Proposals

A minimum of three members of the CES PLC will form a proposal review committee to

evaluate all complete proposals. The proposals will be scored and ranked, independent of the

lump sum price.

The CES reserves the right to make an award based directly on the proposals submitted or to

negotiate further with one or more bidders.

4.3 Proposal Scoring

Each proposal will be assessed based on the criteria indicated in Table 2 provided that the

mandatory requirements in Section 4.1 are met:

11


Table 2. Proposal Scoring.

Criteria Points

Evidence of subject matter knowledge for the content of the

proposed course. 25

Evidence of training and/or experience in delivering face-to-face,

online, blended, and/or e-learning instruction as per the format of

the course named in this RFP.

5

Evidence of training and/or experience in developing

professional learning modules with adult learning strategies. 5

Evidence of past experience working with a coach and/or SME

advisors in virtual settings and willingness to do so in this project. 10

Project work plan showing detailed evidence of the bidder’s

understanding of the terms of reference for this work including

clear and feasible timelines related to training, consultation,

coaching, and communication with ID/PM and two SME advisors.

10

Clarity, organization, completeness, and professional

appearance of the written proposal and any appendices. 5

Credentialed Evaluator (CE) designation held by the bidder

and/or team member(s). 5

Evidence of knowledge of the Canadian evaluation context. 5

Evidence of ability to draw on relevant resource materials or

case examples in French language. 5

Bidder insurance for general liability and/or errors and omissions. 5

Total Possible Points 80

Proposals must meet a minimum scoring of 56 points in order to be screened in for

assessment and selection.

12


5.0 General Conditions

5.1 Right to Amend RFP

The CES reserves the right to amend or supplement the RFP, giving equal information and

cooperation by way of issued addendum to all potential bidders through the CES website

(www.evaluationcanada.ca). Bidders who have indicated their intent to bid will receive email

notification of any amendment or supplement. Please see Section 6.0 for instruction on how to

indicate intent to bid.

5.2 Bidder Incurred Costs

All costs incurred in the preparation and presentation of proposals in any way whatsoever shall

by wholly absorbed by the bidder(s). The location of the pilot test will be determined in

consultation with the course developer. In the case of the face-to-face courses, the CES liaison

with consult with the course development contractor regarding the location for the delivery of the

pilot test. Approved venue and travel costs for the pilot test will be covered separately by the

CES.

5.3 Indemnity

The bidder(s) will indemnify and save harmless the CES from and against all claims, demands,

losses, damages, costs, and expenses made against or incurred, suffered, sustained, done or

omitted by the CES at any time before or following termination of the agreement.

5.4 Acceptance of Proposals

The CES is not bound to accept the lowest price or any proposal of those submitted. Proposals

will be assessed based on the criteria specified in Section 4.0 above.

5.5 Maximum Funding

The funding for this project is limited to the following amounts CA, exclusive of applicable taxes

but inclusive of all expenses:

Evaluability Assessment $5,000

Data Quality Assurance $5,000

Interpreting and Using Quantitative Results for Evaluation $15,000

Evaluation Theories and Models $10,000

http://www.evaluationcanada.ca/

13


Bids in excess of this amount will be considered non-compliant. The payment schedule is

shown in Table 3.

Table 3. Payment Schedule.

Invoice

Amount

Deliverables, in relation to Table 1

25% Completion of initial work plan and SME advisor confirmation (1a,1b,1c and

2)

50% Completion of draft course materials in consultation with the ID/PM and SME

advisors in readiness for the pilot test.(2, 3a, 3b, and 4)

25% Completion of course materials incorporating instructor and learner feedback

from pilot test with final approval by ID/PM and CES Professional Learning

Committee. (2 and 5).

5.6 Ownership

All materials created through this contract shall be the property of the CES and shall not be

published, released, or used for training others without the written consent of the CES. The

course designer will be credited with the course design in the course materials.

5.7 Termination of Contract

A contract awarded on the basis of a response to this RFP may be terminated by either party

with a 30-day notice, with outstanding payments to be negotiated between the contractor and

the CES.

5.8 Insurance

It is recommended that the contractor shall maintain

Comprehensive General Liability Insurance throughout the term of the Agreement in the

amount of not less than $2,000,000 and

Errors and Omissions Insurance throughout the term of the Agreement in the amount of

not less than $1,000,000.

14


6.0 Questions and Inquiries

Should prospective bidders have any questions regarding this RFP, they may contact the CES

Executive Director in writing prior to 12 Noon EST on Mon. Dec. 7, 2015 using the following

email address: [email protected].

In the interest of equality of access to information, responses to questions will be posted on the

CES website at the same link as the RFP notice; questions may be paraphrased and will appear

as anonymous.

[email protected].

15


7.0 Appendices

Appendix A

Abridged Version of the CES Intermediate-level Professional Learning Program: Stage One

Report (May 2015)

(PDF file posted with this RFP)

Appendix B

Draft Notes on Course Development: Stage One Report Appendix (May 2015)

(PDF file posted with the RFP)

CES Intermediate-level Professional Learning Program:

Stage One Report – May 16, 2015 Abridged Version

Full Report Prepared by Kylie Hutchinson

Community Solutions Planning & Evaluation (604) 243-9458

[email protected] www.communitysolutions.ca

mailto:[email protected]

2

Background CES members have indicated a desire for more intermediate-level evaluation training. In 2013 CES National acted in response to this need by supporting a survey of members to identify their level of interest in 23 potential course topics. Following this survey CES National decided to implement a two-stage plan to initiate development of additional professional learning at this level:

Stage One - Broadly define at least six intermediate courses1 to be sanctioned by the CES

Stage Two - Engage contactors to design the curriculum for each The purpose of Stage One was to bring the conceptualization of the courses to a point where development of individual courses can be commenced through the development of:

a short-list of six to seven course areas

high-level course outlines including learning objectives, major content areas, and delivery options

an implementation plan for Stage Two

a proposed budget

a draft Request for Proposal (RFP) for soliciting course designers

recommended template for curriculum design

a list of possible subject matter experts (SMEs) and instructional designers. The courses are intended to range from three hours to two days maximum and would ideally be delivered via a variety of formats to meet the varying needs and geographical locations of members. CES Professional Learning Committee member and Credentialed Evaluator, Sandra Sellick, served as the CES contact for this the project. To provide input into the process an Advisory Committee was struck consisting of the following CES members:

Keiko Kuji-Shikatani

Nancy Carter

Russ Graham

Simon Roy

Steve Montague Two ad-hoc members also participated in meeting discussions: Benoît Gauthier, CES President, and Kathy Gerber, Chair, Professional Learning Committee.

1 For the purposes of this report, the term “courses” has been used as a generic term to refer to workshops,

webinars, eLearning, blended learning, and other forms of non-university professional development lasting a maximum of two days or less.

3

Methodology As the recipient of the Stage One contract I conducted the following activities:

facilitated the determination of a short-list of six to seven topic areas conducted an environmental scan of existing courses drafted course outlines for each topic area sent the course outlines out for review by selected subject matter experts and the

Advisory Committee researched typical course development costs.

Initial Course Selection Although the initial survey of members provided some direction in terms of course topic areas, there were few obvious choices arising from the findings. Therefore, I triangulated information from three sources to assist the Advisory Committee in making a rational and transparent decision:

the 2013 member survey

Advisory Committee discussions

ad-hoc key informant interviews with nine leaders of the evaluation community: Shelley Borys, Director General, Evaluation Directorate, Public Health Agency

of Canada (PHAC) Anne Patenaude, Director, Evaluation, formerly Dept. of Fisheries and

Oceans (DFO), now Internal Audit Brian Moosang, Senior Advisor, Treasury Board Centre of Excellence for

Evaluation Yves Gingras, Director General, Employment and Social Development Canada

(ESDC) Evaluation Directorate Wendy Rowe, Professor, Royal Roads/Consortium of Universities for

Evaluation Education (CUEE) Isabelle Bourgeois, Professor, École nationale d'administration publique

(ENAP) Gail Vallance Barrington, Principal, Barrington Consulting Susan Kistler, Past Executive Director, American Evaluation Association

(AEA) Stephanie Evergreen, Past Director, eLearning Initiatives, AEA

The Advisory Committee agreed to the final short-list of topic areas below:

Evaluation Theories & Models (including Systems)

Economic Evaluation/Cost Analysis

Data Quality Assessment

4

Quantitative Analysis

Qualitative Analysis

Evaluability Assessment

Evaluator Ethics Courses not making the short-list, but recognized as priorities for future development included:

more detailed and advanced Quantitative Approaches for Evaluation

Triangulation Techniques

Rapid Assessment Methodology

Communicating Findings Environmental Scan Once a short-list of seven courses was determined, I conducted a targeted environmental scan of existing intermediate-level workshops and webinars in the evaluation field. The search methodology included:

CES professional development webinars

CES pre- and post conference workshops (2008 – 2015)

AEA pre-and post conference workshops (2008 – 2015)

AEA eStudies (2014 – 2015)

AEA Summer Evaluation Institute (2014-15)

Claremont Professional Development Workshop Series (2014)

Statistics Canada

Marketing Research & Intelligence Association (MRIA)

The Evaluator’s Institute (2015)

International Institute for Qualitative Methodology (University of Alberta)

IPDET

EvalPartners

Consortium of Universities for Evaluation Education (partial search)

general Google search on “evaluation training” and other keywords

Coursera & Ed/X The results indicated gaps in certain areas and provided detailed information on the scope, length, and cost of typical training offered which was useful for later development of the course outlines. Course Outlines Next I developed high-level course outlines using information from the environmental scan and my own evaluation knowledge. I then sent these draft outlines to selected subject matter experts for review and comment as listed in each individual outline.

5

a) Intermediate-level Target

The distinction between Beginner, Intermediate, and Advanced can be blurry and subjective and assigning a level of “intermediate” is challenging. However, the Advisory Committee was unanimous in their belief that these courses would:

not be a substitute for university-level courses (approx. 36-39 hours)

emphasize relevancy and practical application for practicing evaluators (vs. theory only)

focus on Canadian evaluation contexts and examples

require participants to have familiarity with basic evaluation concepts plus direct practical experience conducting or commissioning an evaluation.

Existing Course Level Definitions CES Professional Learning Committee Guideline

Introductory level focuses on basic concepts and generic techniques

Intermediate level provides in-depth information and hands-on learning

Advanced level covers challenging topics and focuses on discussion among informed parties AEA Pre-Conference Workshops

Beginner: Attendees need no prior knowledge of the specific content area in order to participate fully and effectively in the workshop. The information or skills will be new for those who enroll.

Intermediate: Attendees need some basic knowledge of the specific content area, but need not have in-depth knowledge or skills. The workshop will focus on knowledge or skills that build on the basics.

Advanced: To participate fully, attendees must have a substantial working knowledge or skill level in the specific content area. Generally attendees currently use the knowledge or skills in their jobs. At this level, knowledge or advanced techniques are offered to refine and expand current expertise.

b) Continuum of Learning

Using a model similar to the Marketing Research and Intelligence Associationcourses have been classified along a continuum of learning as either “core” or “professional development” according to the following definitions and figure.

Core courses provide a continuum of learning for members with intermediate content that

builds directly on the Essential Skills Series range in topic and complexity, but, when viewed as a whole, represent a diverse

and critical wealth of knowledge for any individual who wishes to be recognized as a Credentialed Evaluator

Professional Development courses

6

are geared to more experienced practitioners are shorter, in-depth, and nimble offerings on more specialized and/or timely

topics that are difficult to classify as either intermediate or advanced provide a future option to address topics identified on the 2013 member

survey but not included in this phase of the course development may be able to take advantage of other innovative delivery formats, e.g.

podcasts with thought leaders

c) Draft Outlines

Draft course outlines are found in the appendix of the draft Request for Proposals. Note that these outlines should be edited prior to circulating the actual RFP and will likely change slightly to better reflect what the course developers believe is feasible within the time frame and is a priority for inclusion.

d) Sanctioning External Courses While the idea of sanctioning other courses sounds good in theory, it would not meet the Advisory Committee’s desire to develop courses that are specific to the Canadian evaluation

7

context. The CES also does not currently have a process in place for sanctioning external courses. What I believe is a more likely scenario is that individuals who already deliver evaluation courses might apply to the RFP and adapt their existing course material for the CES in a cost-efficient manner. The only course area where sanctioning appears to make sense is for Evaluator Ethics. To this end, I recommend that the CES conduct a formal review of both the ARECCI and TCPS 2 CORE ethics courses by selected members of both the CES Professional Designations and Professional Learning committees to determine the appropriateness of each for CES’ needs, plus any necessity for customization for an evaluation context.

e) Course Development Coordinator

Evaluation experience plus instructional design is a rare skill combination, and I suspect many RFP proponents will only be SMEs. Therefore, I recommend pairing all SMEs with an instructional designer (ID) for course development, either one provided by the CES or one of their own choosing. This individual must have significant experience with distance online training. Two IDs that I spoke with believed that most of the budget would be spent on the SME’s time, with the ID acting more like a coach. Another option is to hire an individual with instructional design to not only work with individual SMEs but also manage the entire course development and piloting process. While this will have budget implications, it will greatly serve to improve the curriculum quality and overall learner experience and satisfaction. Instructional designers charge between $50 - $100 per hour.

8

Proposed Implementation Plan

Mass Implementation Option Staged Implementation Option

Activity 2014-15 2015-16 2016-17

Ma Ju Ju Au Se Oc No De Ja Fe Ma Ap Ma Ju Ju Au Oc No De Ja Fe Ma Ap Ma

1. Advisory Committee sign-off on course outlines, budget, RFP, and work plan

A

2. Professional Learning Committee review and sign-off of course outlines, budget, RFP, and work plan

3. CES Board review and sign-off of course outlines, budget, RFP, and work plan

4. Recruit and hire Project Manager

5. Issue RFP for individual course developers

6. Select and award course development contracts

7. Strike individual course Reference Groups (2-3 members max)

8. Course development period

9. Reference Groups review and comment on courses

10. Professional Learning Committee reviews and signs-off on courses

11. Pilot test initial offerings

12. Course developers incorporate feedback and make revisions

13. Formal distribution of courses

9

Topic Evaluation Theories & Models

Economic Evaluation/Cost Analysis

Interpreting and Using Quantitative Results

Data Quality Assessment for Evaluators

Systematic Qualitative Analysis for Evaluators

Target Audience Evaluators or end-users/commissioners of evaluation projects

Evaluators or end-users/commissioners of evaluation projects




Pre-requisites Familiarity with basic evaluation concepts plus direct experience conducting or commissioning an evaluation. Prior completion of the CES Essential Skills Series is advised.

Familiarity with basic evaluation concepts plus direct experience conducting or commissioning an evaluation. Prior completion of the CES Essential Skills Series is advised.

Participants should have experience either conducting or commissioning evaluation projects plus familiarity with basic quantitative analyses. Prior completion of the CES Essential Skills Series plus university-level statistics is strongly advised.

Familiarity with basic evaluation concepts plus direct experience conducting or commissioning an evaluation. Prior completion of the CES Essential Skills Series is advised.

Familiarity with basic evaluation concepts plus direct experience conducting evaluation and working with qualitative data. Prior completion of the CES Essential Skills Series is advised.

Competencies Technical Situational

Technical Technical Technical Technical

Learning Objectives (At the end of the course, participants will be able to:)

explain how evaluation theory can be used to improve evaluation practice

distinguish between major theoretical perspectives and different evaluation approaches using defined criteria

assess their own practice critically and select an appropriate evaluation approach for a given context

identify criteria for selecting an evaluation approach that matches the evaluation context

describe the application and use of different evaluation approaches

identify situations where economic evaluation is warranted and feasible

distinguish between the most common types of economic evaluation used in evaluation

define common terms used in economic evaluation related to costs and outcomes

define summary measures used for different economic evaluation methods

describe sensitivity analysis techniques often used in economic analyses

describe typical challenges and limitations encountered when conducting and using economic evaluation

design a basic cost analysis, as the first step in an economic evaluation

interpret and present findings from a basic cost analyses

identify common statistical tests used to address typical evaluation questions

explain when to use which tests and why and under what conditions

accurately interpret statistical output produced by others and derive conclusions

describe common pitfalls when interpreting quantitative analysis

explain points in the evaluation process where data quality issues are addressed

describe typical errors and quality issues that evaluators face with primary and secondary quantitative and qualitative data

identify DQA techniques to improve the quality of both primary and secondary data

state the advantages and limitations of qualitative data

distinguish between different types of qualitative analysis

choose an appropriate type of qualitative analysis for a given evaluation context

describe practical and ethical issues when analyzing, interpreting, and reporting qualitative data as well as common pitfalls

Delivery Options webinar (6 hours)

in-person (2 days) – to allow for significant hands-on practice and opportunity for questions

using a co-instructor model where a Subject Matter Expert is teamed with an experienced trainer to ensure the content remains at a practical and relevant level

in-person (2 days)

webinar (3 hours)

In-person (2 days)

Courses at a Glance

10

Topic Evaluability Assessment Ethics for Evaluators

Target Audience Evaluators or end-users/commissioners of evaluation projects

This course is designed for evaluators or end-users/commissioners of evaluation wishing to expand their knowledge of ethical issues for evaluators.

Pre-requisites Familiarity with basic evaluation concepts plus direct experience conducting or commissioning an evaluation. Prior completion of the CES Essential Skills Series is advised.

Participants should have a familiarity with basic evaluation concepts plus direct experience conducting evaluation. Participants must provide evidence of completion of [Phase 1 course] at the time of registration. Prior completion of the CES Essential Skills Series is advised.

Competencies Technical Situational

Reflective Interpersonal

Learning Objectives (At the end of the course,

participants will be able to:)

distinguish between evaluability assessment and full-scale evaluation

state the benefits of an evaluability assessment

describe different evaluability assessment models and summarize common components across models

describe tools and techniques for conducting an evaluability assessment

give examples of common challenges and strategies for success

CES Portion:

summarize the CES Guidelines for Ethical Conduct

describe common ethical issues faced by practicing evaluators

explain the role of cultural competence in ethical evaluation practice

identify potential ethical challenges in an evaluation

articulate/access alternative or optional approaches to resolving ethical issues

promote open discussion of ethical challenges and options for resolution in their work environment

Delivery Options webinar (3 hours)

I suggest addressing the topic area of ethics through a two-phase process:

Phase 1: Members complete either the ARECCI or TCPS 2 CORE course

ARECCI – eLearning plus 1 day in-person

TCPS 2 CORE – self-paced eLearning

Phase 2: Members complete a one day in-person workshop to address other CES-specific ethical issues not covered in the above

1 | P a g e

Design of Intermediate-level Professional Learning Courses Canadian Evaluation Society May 2015

Appendix B

Draft Notes on Course Development

from the Stage One Report for the

Design of CES Intermediate-level Professional

Learning Courses for Evaluators

Author: Kylie Hutchinson, Community Solutions Planning & Evaluation

for the Canadian Evaluation Society

Please note that the DACUM chart and lesson plan following p. 12 are

provided only as examples of planning tools. The Instructional

Designer/Project Manager for the development of these courses may present

a modified or different design during the orientation for course developers in

January 2016.

2 | P a g e


CES Intermediate-level Professional Learning Project Topic: Evaluability Assessment Evaluator Competencies: Technical, Situational Environmental Scan Highlights:

majority of courses 1 day in length

no Canadian courses specifically on EA Rationale for Inclusion:

Pros Cons

scored medium-high on needs assessment for intermediate (34%) and lower for combined (59%)

this is one area where it might be more important to have a Canadian perspective, e.g. federal evaluation requirements?

Type: Core Target Audience: This course is designed for evaluators and commissioners of evaluation projects wishing to broaden their knowledge and use of evaluability assessments in their regular evaluation practice. Pre-requisites Participants should have a familiarity with basic evaluation concepts plus direct practical experience conducting or commissioning an evaluation. Prior completion of the CES Essential Skills Series is advised. Description When evaluation resources are scarce it is critical they are used in the most cost-effective manner. Evaluability assessment is an under-used (but rapidly becoming more popular) evaluation planning tool that is used for determining the feasibility of, and best approaches for further evaluation. When done well a good evaluability assessment can significantly inform the choice of an appropriate evaluation design, build stakeholder support, save time and resources, as well as inform program design and improvements in the process. Participants will leave with greater appreciation of how an evaluability assessment can be applied in their own practice.

3 | P a g e


Learning Goal To increase the use of evaluability assessment in evaluation practice. Learning Objectives At the end of this course, participants will be able to:

distinguish between evaluability assessment and full-scale evaluation

state the benefits of an evaluability assessment

describe different evaluability assessment modelsand summarize common components across models

describe tools and techniques for conducting an evaluability assessment

give examples of common challenges and strategies for success Content Ideas

purpose of evaluability assessment

how it differs from full-scale evaluation

benefits of evaluability assessment

common components/steps across models

engaging stakeholders

developing a program theory/logic model

assessing program design, alignment with program theory, and consistency of implementation

assessing program capacity and plausibility of goal attainment

assessing measurement and data capacity

availability, appropriateness, and quality

choosing an appropriate and feasible evaluation design

facilitating use of evaluability assessment results

stakeholder uses

specific tools and techniques for application

data collection methods and tools

how to apply in different evaluation contexts

common challenges and strategies for success Summative Assessment: Yes

4 | P a g e


CES Intermediate-level Professional Learning Project

Topic: Data Quality Assessment for Evaluators Evaluator Competencies: Technical Environmental Scan Highlights:

tends to be touched on only briefly as part of standard data collection/analysis courses

Rationale:

Pros Cons

rated high in needs assessment (35%)

ESS only provides minimal content on “evidence quality” (Module 11) and “assessing data quality” (Module 12)

nothing really offered in this area in one comprehensive workshop

Type: Professional Development Target Audience: This course is designed for evaluators wishing to expand their knowledge of Data Quality Assessment to ensure the quality of the data they collect or use. Pre-requisites Participants should have a familiarity with basic evaluation concepts plus direct practical experience conducting evaluation. Prior completion of the CES Essential Skills Series is advised. Description Evaluators collect data, and use even more. But how does the average evaluator ensure the data they collect or use is trustworthy and of the highest quality? In this course learn what problems evaluators typically face with their quantitative and qualitative data and techniques for carrying out Data Quality Assessment (DQA).

5 | P a g e


Learning Goal To increase awareness of techniques to assess data quality. Learning Objectives At the end of this course participants will be able to:

explain points in the evaluation process where data quality issues are addressed

describe typical errors and quality issues that evaluators face with primary and secondary quantitative and qualitative data

identify DQA techniques to improve the quality of both primary and secondary data Content Ideas

what does “data quality” mean?

points in the evaluation process to consider/address data quality

dimensions of data quality

accuracy

completeness

validity

reliability

timeliness

confidentiality

precision

integrity

rigour, neutrality, balance, and transparency.

typical errors and problems with data that evaluators face by phase of evaluation: e.g., question design, sampling, data collection, data processing/analysis, generalization, reporting

quantitative and qualitative

primary and secondary

e.g. missing data

difficult to reach populations

uncooperative subjects

low survey response rates

data entry/coding errors

poor quality administrative data/program records

other

tips and techniques for assessing/ensuring data quality

quantitative data

data management systems

components needed to ensure data quality

assessing data management systems and administrative/program data

data verification

6 | P a g e


verification techniques

assessing survey data

12 steps of data cleaning

qualitative

validity, trustworthiness and authenticity of qualitative data

assessing literature/systematic reviews

other?

resources

Benoît Gauthier’s Survey Research Assessment Framework

USAID Data Quality Assurance Checklist

MEASURE Evaluation Data Quality Assessment Methodology and Tools

MEASURE Evaluation Data Quality Assurance Tools

Jörg Blasius and Victor Thiessen, Assessing the Quality of Survey Data, Sage, 2012.

Summative Assessment: Yes

http://circum.com/index.cgi?en:doc:T028

http://usaidlearninglab.org/sites/default/files/resource/files/Data%20Quality%20Assessment%20Checklist.pdf

http://www.slideshare.net/measureevaluation/measure-evaluation-data-quality-assessment-methodology-and-tools

http://www.cpc.unc.edu/measure/tools/monitoring-evaluation-systems/data-quality-assurance-tools

7 | P a g e


CES Intermediate-level Professional Learning Project Topic: Interpreting and Using Quantitative Results Evaluator Competencies: Technical Environmental Scan Highlights

existing courses range from three hours to eleven weeks, general overview to specific tests and techniques

university-level courses not included Rationale

Pros Cons

although some introductory evaluation workshops might cover the basics of descriptive stats, most (if not all) would not cover analysis

federal evaluators are either “really competent in stats, or not at all”

lack of interpretation skills mentioned in the TBS Health of the Evaluation Function report

target audience:

people entering evaluation from other professions?

evaluators who want a quick refresher?

other providers are not reluctant to cover this topic in 1-2 days

although many other opportunities, none are tied to where the ESS leaves off

could work if clear about target group and scope

Is CES in the business of teaching stats?

can’t begin to compete with in-depth university-level training

requires two days in-person to provide opportunities for practice and feedback

one federal key informant said that their inexperienced evaluators are more likely to learn on the job from more experienced ones, but of course this doesn’t cover everyone

already many opportunities for short courses in this area

Advisory Committee considered three options:

o Option #1 - basic refresher for evaluators that picks up where ESS leaves off o Option #2 - focus on interpretation and use of statistical output for evaluators

plus end-users/commissioners of evaluation

8 | P a g e


o Option #3 – more advanced webinars on specific quantitative techniques used in evaluation

ultimately decided on Option #2 Type: Professional Development Target Audience This course is designed for evaluators and/or commissioners and end-users of evaluation who wish to expand their understanding of, and ability to interpret, quantitative analyses specific to evaluation. Pre-requisite Participants should have experience either conducting or commissioning evaluation projects plus familiarity with basic quantitative analyses. Prior completion of the CES Essential Skills Series plus university-level statistics is strongly advised. Description Whether or not you actually conduct evaluations yourself, or work with the findings of others, you are likely exposed to quantitative analyses. Being able to understand and accurately interpret quantitative results is a critical skill for all who do and use evaluation. This course is designed specifically for individuals who have taken a university-level statistics course at some point in their career but would like to increase their competence and confidence in interpreting statistical output as end-users of evaluation. Learning Goal To increase the competence and confidence of evaluation end-users in interpreting quantitative analyses and statistical output. Learning Objectives At the end of this course, participants will be able to:

identify common statistical tests used to address typical evaluation questions

explain when to use which tests and why and under what conditions

accurately interpret statistical output produced by others and derive conclusions

describe common pitfalls when interpreting quantitative analysis Content Ideas

picks up where the ESS leaves off

9 | P a g e


focus less on ability to run actual tests and more about understanding the rationale for using which tests and when, under what conditions, limitations, being able to understand statistical output and correctly derive conclusions

what type of evaluation question can be solved by each technique

what these techniques actually do

what results you can expect when you use these techniques

what pitfalls await the unwary user

comparisons to assess incrementality: e.g., comparisons in time, in space, between variables

Summative Assessment: Yes

10 | P a g e


CES Intermediate-level Professional Learning Project

Topic: Evaluation Theories & Models Evaluator Competencies: Technical, Situational Environmental Scan Highlights:

looked at overviews instead of one specific theory or model/approach

range from 1 hour to 3 days

all with a strong focus on relating theory to practice Rationale for Inclusion:

Pros Cons

rated very highly on needs assessment

several federal key informants see it as a real need

mentioned as an issue in the TBS Health of the Evaluation Function report

there is new federal interest in new ways of doing evaluation

many come to evaluation from other disciplines and lack the underlying theory

picks up where the ESS leaves off on “Underlying theories and their implications” (Module 1)

Type: Core Target Audience: This course is designed for evaluators and commissioners of evaluation projects wishing to broaden their knowledge and use of different evaluation theories and approaches in their practice. Pre-requisites Participants should have a familiarity with basic evaluation concepts plus direct practical experience conducting or commissioning an evaluation. Prior completion of the CES Essential Skills Series is advised.

11 | P a g e


Description: Too often, evaluation is conceptualized as the application of research methods associated with social sciences, in the absence of an overall theoretical framework. Evaluation theories have emerged to fill that gap, allowing for a clear distinction between the general field of applied research and the specific field of evaluation research. Over time the evaluation field has embraced a growing number of different approaches stemming from different evaluation theories and models. But what are these theories, models, and approaches, and what do these they mean for the practicing evaluator? A greater appreciation of them can improve one’s evaluation practice by making an evaluator’s values more transparent and directing the selection of appropriate evaluation approaches. This course will bridge the gap between evaluation theory and evaluation practice with a strong focus on practical application. Learning Goal To increase the practical awareness and application of evaluation theory in day-to-day evaluation practice. Learning Objectives At the end of this course, participants will be able to:

explain how evaluation theory can be used to improve evaluation practice.

distinguish between major theoretical perspectives and different evaluation approaches using defined criteria.

critically assess their own practice and select an appropriate evaluation approach for a given context.

identify criteria for selecting an evaluation approach that matches the evaluation context.

describe the application and use of different evaluation approaches. Content Ideas

what are evaluation “theories”

difference between evaluation “theories”, “models”, and “approaches”

overview of major theoretical streams in evaluation, use as an underlying framework:

Marv Alkin & Tina Christie’s Evaluation Theory Tree Shadish, Cook & Leviton’s Foundations of Program Evaluation Mertens & Wilson’s Program Evaluation Theory & Practice

what distinguishes evaluation theory from other social science and

program theory

emphasis on Canadian theories and models of evaluation

https://books.google.ca/books?id=swDMkRzVCRIC&pg=PA381&lpg=PA381&dq=evaluation+theory+tree+revisited&source=bl&ots=jaNAcgNajD&sig=t50cdQfawVN3JHe0trbXBtndG4c&hl=en&sa=X&ei=FbMVVeT5JorUoASr1oKwBQ&ved=0CEIQ6AEwBA#v=onepage&q=evaluation%20theory%20tree%20revisited&f=false

http://www.sagepub.com/books/Book3133

http://books.google.ca/books?id=DUAmbbuWC1oC&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false

12 | P a g e


supplemented with Canadian materials that reflect the Canadian approaches to evaluation theory/models, Canadian contexts (values, political system, evaluation planning/process, credentialing, as examples), and feature Canadian evaluation theorists, researchers, and leaders

why be cognizant of evaluation theory

points where evaluation theory can impact the evaluation process

examination of 4-6 major models/approaches

utilization-focused

developmental

participatory

empowerment

systems/complexity

theory-driven evaluation contribution analysis

distinguishing factors

purpose of evaluation

program context

role of evaluator

evaluator values

types of stakeholders

other

key techniques and strategies

advantages and limitations Summative Assessment: Yes

35 | P a g e


SAMPLE COURSE PROFILE (DACUM CHART)

COURSE TITLE: CES Essential Skills for Evaluation (4 days) DATE: December 11, 2008

A1. Explain major uses and/or benefits of evaluation 1 hr C

A2. Diagram relationship between planning, management, and evaluation 30 min C

A3. Identify ways to improve evaluation practice 30 min C

A4. Cite formal standards for making evaluations ethical and fair 1 hr C

A. Explain the role of evaluation in effective program delivery

. A

3 hr

B1. Describe major types of evaluation 1 hr C

B2. Describe different approaches for conducting any type of evaluation 30 min C

B3. Select the most appropriate type and approach to use 30 min C

B. Choose an appropriate type of evaluation and approach B

2 hr

C1. Explain purpose of a needs assessment 15 min C C

C2. List steps in conducting a needs assessment 15 min C C

C3. Identify methods for collecting needs assessment data 30 min C

C.Describe how to conduct a needs assessment C 1 hr

D1. Explain purpose of an evaluability assessment 30 min C

D2.Explain steps in conducting an evaluability assessment 30 min C

D. Describe how to conduct an evaluability assessment D

1 hr

E1. Explain purpose of logic models 15 min C

E2. Diagram a complete logic model 2 hr C

E3. Clarify role of logic models in evaluation 45 min C

E. Prepare a logic model E

3 hr

36 | P a g e


F1. Explain purpose of an evaluation framework 30 min C

F2.Describe common evaluation framework formats 30 min C

F3. Develop indicators for measuring outcomes 1 hr C

F. Develop an evaluation framework F

2 hr

G1. Explain purpose of a process evaluation 2 hr C

G2. Identify process evaluation questions 1 hr C

G3. Distinguish between process evaluation and performance monitoring 1 hr C

G. Design a process evaluation G

3 hr

H1. Explain purpose of an outcome evaluation 30 min C

H2. Select appropriate research design 3 hr C

H3. Identify possible threats to validity 1 hr C

H4. State steps in developing an outcome monitoring system 30 min C

H. Design an outcome evaluation H 5 hr

I1. Summarize advantages & disadvantages of various data collection methods 1 hr 15 min C

I2. Explain how to conduct common data collection methods 1 hr C

I3. Select data collection methods for evaluation framework 45 min C

I. Choose a data collection method I 3 hr

J1. Specify principles for effectively communicating evaluation findings 15 min C

J2. Identify innovative methods for communicating results 30 min C

J3. Develop a communication plan 15 min C

J. Communicate evaluation results effectively J

1 hr

37 | P a g e


1.0 Sample Lesson Plan

38 | P a g e


Sample Lesson Plan

Opening/Hook So now our evaluation framework is almost complete! But we’re just missing the last step, and a very crucial step, to finally having a complete plan for your evaluation. Can anyone guess what it is?

Session #4 – 3 hr

2

Objectives By the end of this unit you will be able to summarize at least two advantages and disadvantages of six types of data collection methods, explain how to conduct three common types of data collection, and select appropriate ones for each evaluation question listed in your own evaluation framework

1

Pre-Assessment Raise your hand if you already do some form of data collection in your day to day work. What types of data collection do you tend to do?

2

Learning Tasks Instructor Activities Learner

Activities

Resources Time

1. List common data collection methods

a) Ask learners to brainstorm all the possible ways to collect data; record on flipchart; highlight the six most common

listen and respond flipchart 10

2. Classify methods as quantitative or qualitative

a) If we had to classify all these methods into one of two kinds, what would they be?

b) Define qualitative (“deep”) and quantitative (“wide”)

listen and respond 5

3. Describe advantages and disadvantages of each

a) Arrange learners into small groups; instruct each group to spend 5 minutes at one of six flipchart stations around the room and list the advantages and disadvantages of the method listed at each station; after 5 minutes move to the next station and add to what the previous group has done; keep moving until each group has been to each station

b) Debrief each flipchart/method in the large group; add specific tips/highlights about each method

c) Tell story of:

program records - AMSSA and financial data

observation – Allan Best’s smoking study/Hawthorne Effect

case studies – MQP’s mantra, McGill McConnell program, VCH Storytelling Conference

d) Note more detailed pages in workbook e) What questions do you have at this point?

think in groups and provide ideas

listen and respond

six flipchart stations

workbook

60

39 | P a g e


Learning Tasks Instructor Activities Learner Activities Resources Time

4. Explain how to conduct an interview

a) Model a “good” and “bad” interview with a volunteer

poor listening

finishing their sentences

no pregnant pause

not explaining the purpose and confidentiality

b) Debrief what was wrong c) Review “Interview Tips” in workbook pg. 32

observe and respond

read and listen

chairs workbook

10

5. Explain how to conduct a focus group

a) Review workbook “When to Use Focus Groups” and “Steps in Conducting”

b) Tell story of BCCDC focus group with Dr. Rekart

c) Recruit five volunteers to model a “good” and “bad” focus group

secretely assign a shy and dominant talker

assign a note taker and show two recorders

d) Debrief what was wrong and suggest how it can be corrected

e) Review workbook “When to Use Interviews vs. Focus Groups”

f) Point out focus group resources on display table

observe and respond

read and listen

workbook chairs hats

15

6. Explain how to design a simple questionnaire

a) Tell story of my thesis and 82% mail response rate

b) Arrange learners in small groups and hand out sample questionnaire; ask learners to identify all design errors

c) Debrief in large group; hand out answers and corrected version

d) What questions do you have at this point?

listen

think in groups

questionnaire activity handouts

35

40 | P a g e


Learning Tasks Instructor Activities Learner Activities Resources Time

7. Determine source of data for each evaluation question (or outputs/outcome) in evaluation framework

a) Review Civic Leadership program slide example

listen CL slide 5

8. Identify corresponding data collection method


b) What might be some other ways of collecting this data?

listen and respond CL slide 5

9. Assign a time frame and responsibility for completion


b) Ask learners to individually complete the data collection columns in their own evaluation framework they have been working on throughout the course; circulate and offer one-on-one assistance and feedback; ask learners to debrief their complete evaluation framework in partners; ask for general comments or observations on the activity in a large group

listen

work individually

CL slide 20

Post-Assessment Ask learners to pretend that I’m a new evaluator who has never done any data collection. In a large group, tell me the advantages and disadvantages of using interviews, focus groups, and questionnaires. Go in round robin fashion.

Next ask learners to tell me as a new evaluator how to conduct an interview, focus group, and properly design a questionnaire.

10

Closure So let’s just recap the main points.

What are the two main types of data collection?

What is one advantage of each of the six types of data collection we’ve looked at?

What is one disadvantage of each?

So know we know how we’re going to collect our data, let’s turn our attention now to how we can effectively communicate our findings.

5

request for proposal intermediate-level professional ... · 3a. create a course outline using...

Documents