competency 8 edci577project4_jbecker (1)

7/29/2019 Competency 8 EDCI577Project4_JBecker (1)

1/26

Evaluation of University Extension HTML/XHTML Training for Website Development

Project IV: Evaluation of University Extension

HTML/XHTML Training for Website Development

James Becker

Purdue University

Strategic Assessment and Evaluation

577

Evangelia van Barneveld

October 12, 2012

Identification of evaluation goals and scope (5) 5

Description of process (includes Gantt chart) (5) 5

Data collection (instruments description) (20) 15

Analysis process (20) 15

Report (10) 10

Requirements (5) 5

TOTAL 55

Comments:

Jim a good start. Still lots of opportunity to evolve the plan.

Comment [A1]: Ah yeaI know her.lot!


2/26

Evaluation of University Extension HTML/XHTML Training for Website Development 1

Table of Contents

Executive Summary ...................................................................................................................................... 2

Key Findings and Recommendations.....3

Introduction .................................................................................................................................................. 4

Data Collection and Analysis........................................................................................................................ 6

Level 1Reaction to Regional Trainings...6

Level 2Learning.7

Level 3Behavior Change..7

Level 4Results8

Conclusion..9

References10

Appendices...11


3/26


To: The Extension Administrative Team

From: IDEA ConsultingDate: 10/9/2012

EXECUTIVE SUMMARY

University Extension as an organization provides each county Extension office with a

basic website. It is up to each county to develop the content of the website and populate it with

local activity and program information. Many University Extension county websites have done

an inadequate job of website development. The purpose of this training program is to improve

the quality of University Extension County websites. With 92 counties in the state of Indiana,

University Extension is faced with a large task to bring up the quality of so many websites.

IDEA Consulting began a needs analysis process by (see Appendix B) meeting with University

Extensions administrative team in June. Regional and County Directors said Extension

Educators do not presently possess the knowledge and skills for creating quality web pages on

their county websites. Improved county websites would help meet the main objective to give the

public a better web interface with University Extension. Several Extension Educators were

interviewed and confirmed the issue of lack of training and heavy workloads, as factors

preventing them from doing a better job with the websites.

Extension technology trainers will utilize MindLeaders, Web Publishing and Design

with HTML 4.01 and XHTML: Basics of HTML and XHTML, online curriculum. They will

train Extension Educators statewide, to create web pages on their websites during the formal

training and provide support beyond the training.

The objective of the training process will be to create capable Extension Educators to act

as resources and web page developers in county offices, statewide, to improve the overall quality

Comment [A2]: Gotcha


4/26


of county Extension web pages. Approximately 120 Extension Educators will attend one of six

regional trainings, 3 Days each. They will be held around the state from September through

October. An additional 5 months will allow for collecting additional outcomes and holding focus

groups with end users of the many Extension websites around the state.

Using Kirkpatricks four levels for evaluating training programs we will collect and

analyze data from different parts of the training. Data will include information about reactions to

the training, knowledge and skills received and behavior change and results.

Key Findings

The results from different parts of the evaluation may include:

Satisfaction or dissatisfaction with the training facilities Satisfaction or dissatisfaction with trainers Satisfaction or dissatisfaction with online training modules Increased employee knowledge of Web design and HTML/XHTML Increased employee knowledge of interactive multi-media web features Increased employee knowledge of urls, cascading style sheets and links Employee creation of new county web pages Types of new county web pages created How new knowledge and skills are being utilized Additional training that is needed Clientele reaction to new web pages and information on web pages Clientele expectations and desire for future web pages

Recommendations

Possible recommendations may include:

Follow-up with Extension Educators regarding additional training they feel is needed,once they have had time to use knowledge and skills from the training.

County Directors and Regional Directors should meet with Extension Educatorsregarding the overall development of web pages to review, progress, quality and to

address problems. Extension Educators should continue to be supported by University Trainers and by on-

line training resources.

Comment [A3]: Sounds good.


5/26


University Extension as a whole could consider adopting a Balanced Scorecard programto encourage performance improvement as it relates to training of its employees. Thiswould help the organization monitor how it is doing with professional development.

Introduction

University Extension has a long tradition of providing research based information to

people who would not have access to the University otherwise. The importance of county

websites has increased tremendously in recent years. Still the potential for using websites for

more than mere information is not being reached. Many competitors have better websites with

increased abilities to provide e-learning, educational videos, and interaction with clientele.

Evaluating training and website development will aid University Extensions goal to create high

tech avenues that continue to do as their slogan suggests, Bringing Knowledge to the People.

The training includes the following courses:

Web Publishing and Design with HTML 4.01 and XHTML: Getting Organized Web Publishing and Design with HTML 4.01 and XHTML: Basics of HTML and XHTML Web Publishing and Design with HTML 4.01 and XHTML: HTML, XHTML, and Cascading Style Sheets

(CSS)

Web Publishing and Design with HTML 4.01 and XHTML: Adding Interactivity and Multimedia withHTML and XHTML

The four courses are broken into smaller modules; each course will take 2-3 hours to complete.

With breaks, lunch and time for discussions, evaluation surveys, pre and post tests, the regional

trainings will fill 3 days each. Each of the 6 trainings will have approximately 20 Extension

Educators. Combining the on-line courses with live instructors, allows Extension Educators to

have live support. Later when the training is over they will continue to have access to the courses

online and can return to them to refresh information or check on something they need help with.

Instructors at the regional trainings will also work with Extension Educators to set-up a peer

organized Facebook group, which will serve as their own learning community to support each


6/26


other. The Extension employed trainers will also continue to be available by phone from their

campus location.

Specific reasons for evaluating this training and subsequent results, is to clearly identify

how well the problem of upgrading county websites can be addressed with this kind of training.

The objective of this evaluation process will be to measure and quantify how well Extension

Educators learn the information in the training. Then it will look at how well they use it in their

work to create better web pages, transferring the skills to their jobs (Kirkpatrick & Kirkpatrick,

2006).

Target Performers

Extension Educators are on the ground where clientele live, it is easy to see they have

very busy schedules. Doing one more thing is a major concern to them. All Extension

Educators have Masters degrees, yet much of the new cloud technologies have come along

since many of them finished their degrees. Many have never participated in an online course

before, so using online courses with live instructors will be a new experience. In order to use the

web and other online technologies to communicate better with clientele, they must be willing to

learn and commit time and effort. Success will depend upon the trainings ability to help

Extension Educators see what advantages their new knowledge and skills give them.

Comment [A4]: Good


7/26


Data Collection and Analysis

As mentioned earlier in the Executive Summary, this evaluation will utilize Kirkpatricks

evaluation framework, which is divided into 4 levels. There will be 6 regional trainings held in

different parts of the state. This will keep the size of each training group relatively small, close to

20 people in each. Level 1 and Level 2 data will be collected at all 6 regional trainings.

Level 1Reaction to Regional Trainings

An online survey will be administered as part of the training to 100% of the attendees.

The survey will be a mixture of multiple choice and Likert scale questions (see Appendix F).

There will be some open ended questions to invite specific comments. This will result in a

mixture of quantitative and qualitative data to analyze.

The questions pertain to the appropriateness of the training content, effectiveness of the

trainers, the training location, facilities, the online training quality, ease of understanding and

ability to get questions answered. The survey results will be tabulated and the averages will be

used to apply ranges as guidelines. For example if a Likert scale such as Very Good, Good,

Average Not Good and Very Bad were scored 5 = Very Good, 4 = Good, 3 = Average, 2 = Not

Good and 1= Very Bad, you could tabulate all the answers to that question. Lets say 30 people

take the survey, 10 people answer Very Good, 10 answer Good, 7 answer Average, 2 Not Good

and 1 Very Bad. You would add 50+40+21+4+1=116/30=3.8 average for that question (see

Appendix H). The scores then can be used to rate the satisfaction of the different groups taking

the training at different locations around the state. The can see how well they are doing in regard

to the different questions. Answers to open ended questions are considered qualitative. These

answers will be sorted into categories and quantified to the extent they are clearly saying the

similar things. For example if the question asked you what you would change about the training

Comment [A5]: Appendix H does notenlightenment on this topic.


8/26


and five people wrote they would prefer more face to face instruction, these answers would be

grouped together.

Level 2 - Learning

A Pre-Test/Post-Test (see Appendix G) will be administered at the beginning and ending

of the regional trainings. Extension Educators will take a Pre-Test at the beginning of the training

about HTML/XHTML and other related web design topics that will come up in the training. An

identical Post-Test will be given at the end of the training. The scores for both tests can then be

compared for each Extension Educator. Areas that need improvement can be identified for the

group as a whole and for individuals. Additional training options or repeating certain online

training segments will be identified for individuals.

Level 3 Behavior Change

Extension Educators will have a number of things to encourage their usage of their new

knowledge and skills. They will have each other, via a Facebook group dedicated to

HTML/XHTML and other web design topics they choose to bring to the learning community.

In this way they can become their own support group in addition to on-campus trainers who will

be available via phone.

Approximately 60 days or more after their training, Extension Educators will be sent an

online survey to be completed. The survey (see Appendix C) will ask for examples of how they

have used their training to work on their websites. They will rate their confidence to utilize what

they have learned to work on their websites, as well as tell how often they are working on their

websites. They will provide specific examples of what they have done and how they have done

it.

Comment [A6]: Youd probably al socode those open-ended answers in order tthe potential groupings might be.

Comment [A7]: Good

Comment [A8]: Thats not your role aevaluator, right? You need to figure out Wparticipants were successful or not in thei

Comment [A9]: Again, this would bethe instructional designer, not the evaluatneeds to focus on the application of the kand skills learned in training on the job.

Comment [A10]: And how will you a


9/26


Level 4 Results

University Extensions websites are where many of their clientele and public get

information about programs. Timely, reliable information helps them to keep up on news for

their farms, homes, children, and family members. This goal to improve service to clientele via

websites cannot be proven without feedback from the people who use them. Six regional focus

groups will be held. Pre-training focus groups from six different areas of the state will be held in

July, 2012 and post-training focus groups will be held mid-February through March, 2013.

County Extension offices will provide clientele mailing lists and assist in sending out requests

for participation in the focus groups.

Focus groups in July, 2012 will set a benchmark of where people think the websites are

before the training of Extension Educators. Each focus group member will be asked to fill out a

brief website evaluation (see Appendix D), for 2 University Extension County Websites, before

participating in the focus groups via Adobe Connect. They will be asked to turn in these forms

after the focus groups are held. Though the focus groups will be held by Adobe Connect, local

offices may offer to host so that people may attend and utilize a speaker phone system, if they

prefer that to calling in from home.

In mid-February through March, a second round of focus groups/website evaluation

groups will be assembled from 6 different regions of the state. Adobe Connects will be utilized

for the focus groups with, volunteers, 4-H parents, 4-H members, Master Gardeners, teachers,

farmers, etc.

The focus group members will be 6-10 people each from each of 6 regions. On the Adobe

Connect sessions they will then be asked questions to help define their concerns about specific

websites (see Appendix E).

Comment [A11]: What does that meathese web sites are at?

You knowthere are a TON of metrics ygather from log files that have nothing topeoples opinions. For example, has traff

to the web site? How long are people stayweb site? What sort of content is downloany)? What sort of questions are fielded bCONTACT US people? Basically, you wwhat the impact is on the organization of

these improved web pages (and the organgoal is NOT nicer web pages). What doeorganization envision happening once theare improved? THAT is your level 4.

Comment [A12]: What people? Howselect them?


10/26


Conclusion

Technology training is never an easy task to take on. There are many things to consider,

because people must be ready to utilize the newly learned knowledge. In this case Extension

Educators are to receive significant training in web page design, but this not an easy subject.

There could be a certain amount of push-back from the educators receiving the training. This

could impact Level 3 evaluation results for Educators who do not follow through to use their new

knowledge and skills. However, if there is success at improving County Websites, Extension

Educator will benefit from increased web traffic on their sites and better communication with

clientele.

The training program itself is relatively inexpensive (see Appendix A). The online

curriculum being used is a trial course offered free by MindLeader. The main expenses for the

training are travel, food and facilities. The trainers are from University Extensions own IT

training staff, already under salary. The evaluation report will hopefully be able to show

successful training results at level 2 and usage of the Extension Educators new skills at level 3.

The best result of course would be many exciting new web pages on County Extension websites,

enjoyed by clientele.

Comment [A13]: Here is your L4 stuf


11/26


References

Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006).Evaluating training programs (3 ed.). San

Francisco CA: Berrett-Koehler Publishers, Inc.

Lemay, L. (2006). Web publishing and design with html 4.01 and xhtml: basics of html and

xhtml. Retrieved from http://central.mindleaders.com/lms/tile.do?def=topFrame
http://central.mindleaders.com/lms/tile.do?def=topFramehttp://central.mindleaders.com/lms/tile.do?def=topFramehttp://central.mindleaders.com/lms/tile.do?def=topFrame


12/26


APPENDICES


13/26


Appendices

Appendix A Program Chart

Appendix B Project Gantt Charts

Appendix C Level 3 Educator Survey

Appendix D Website User Evaluation

Appendix E Website Focus Group Questions

Appendix F. Level 1 Survey

Appendix G......................... Pre-Test/Post-Test

Appendix H................................ .Sample Tabulation


14/26


Appendix A

Evaluation Program Chart

Project

Extension HTML/XHTMLTraining for WebsiteDevelopment

Scope

Evaluation of the course andLevel 1 through Level 4 datacollected.

Identification

Version 25.6N1

Creator

Jim Becker

Purpose Levels Objective to Measure

To evaluate the success oftechnology training forExtension Educators toimprove county websites.

Level 1ReactionLevel 2 LearningLevel 3Behavior ChangeLevel 4 - Results

Measure the increased abilities of Extension Educators to work oncounty websites and the results based on clientele reaction to websitesthe Educators have worked on.

Step

1.Needs Assessment2.3 Day Regional Training3.Collect Level 1 & 2 Data4.Pre-Test and Post Test5.Employee Level 3 Survey6.Clientele Focus

Groups/Collect Level 4Data

7.Final Data AnalysisComplete

Who

Extension Training StaffExtension Training StaffExtension Training StaffExtension Training StaffExtension Training StaffExtension Training Staff

Extension Training Staff

Date

June 4July 5Sept. 4Oct. 18Sept .4Oct. 18Sept. 4- Oct. 18Jan. 24Feb. 8July 9July 30 & Feb 13

Mar 28

Budget

Travel$1200Food - $2500Printing $ 950Trainers $8000Total $12650.00

Data to be Collected

Level 4 Clientele reaction towebsites

Level 3 Application ofKnowledgePre-Test & PostTestLevel 1 SurveyLevel 2 Survey

When

Before & after ExtensionEducator trainingAt least 60 days after training3 Day Reg. Ext. Educ.Training3 Day Reg. Ext. Educ.Training3 Day Reg. Ext. Educ.Training3 Day Reg. Ext. Educ.Training3 Day Reg. Ext. Educ.Training

How Collected

Focus Groups & Writtenwebsite evaluationSummer/SpringOnline SurveyOnline SurveysOnline SurveyOnline SurveyOnline SurveyOnline Survey

From Whom By Whom

Clientele Ext. Training Staff

Ext. Educators Ext Training StaffExt. Educators Ext Training StaffExt. Educators Ext Training StaffExt. Educators Ext Training StaffExt. Educators Ext Training StaffExt. Educators Ext Training Staff

Target Measures Methods of Analysis Analysis

Analysis will be conducted by University Extension Trainers. Some ofthe questions on the different surveys will be appropriate forquantitative data, but other open ended questions will be suited only

for qualitative data. The different results will be used to create a reportthat will analyze the training process and its objectives.

Reaction to 3 Day Regionaltraining by ExtensionEducators

Online survey completed byExtension Educators.

Knowledge of web design,HTML and XHTML, etc.

Increase in knowledge basedupon comparison of Pre-Test andPost Test Scores

Comment [A14]: Sounds like the WHalso include clienteleas per your bullet

Comment [A15]: Im not sure L4 predata needs to be specifically collected by evaluator per seit seems to me that the ISSUES that have been brought up aboutpages would serve as good post-training metrics.


15/26


Application of knowledge byusing it on County ExtensionWebsites

Survey completed by ExtensionEducators at least 60 days after 3Day Regional Training.

Clientele reaction to countyExtension websites prior totraining of ExtensionEducators. Clientele will eachuse a paper evaluation toevaluate 3 websites, thenparticipate in a Focus groupvia Adobe Connect

The paper evaluation of websiteswill be collected from theUniversity Extension Clientele.Comments and answers toquestions during the focusgroups will be recorded andtranscribed. Similar answers tospecific questions will be codedand categorized.


16/26


Appendix B -GANTT Charts

Planning and Needs Analysis

Regional Extension Educator Trainings

Pre-Test/Post-Test and Level 1 Data Analysis


17/26


Appendix B Contd

Level 3 and Level 4 Data Collection


18/26


Appendix C

Level 3 Extension Educator Survey


19/26


Appendix C Contd


20/26


Appendix D

Website User Evaluation Sheet

Poor Below Average Average Above Average Excellent

Ease of Navigation

Home page link always

visible

Website looks good in any

browser.

Text and images load

quickly.

Information and topics are

easy to find.

Pages are well organized

and easy to read.

Information for different

Extension Programs, ie. 4-

H, Agriculture, Nutrition,

etc, are current.

Pages for different

Extension programs are

uniform and attractive.

Page content is not

disorganized.

Links are not broken

and go to correct, up to

date information.

Comment [A16]: Insert instructions fimplementation.

Comment [A17]: You should avoid cquestions (AND)

Comment [A18]: This is a double negdisorganized.


21/26


Appendix E

Focus Group Questions for Website Evaluation

Navigation

Is the website easy to navigate or get around to information you want? Are you able to find the information that you came to the site for? Are the pages well organized to make finding information easier? If not, what are they

like?

Is there a structure to the pages that make the information seem logical? Are the links all connected to the information like they say they are?

Content

Do the different site areas, ie, 4-H, Agronomy, Health and Human Sciences, have thekind of information you think they should have? If not why not? What could they do

better?

Do you find the information you need when you come to the site? If not is the issuecontent or organization?

Is the content up to date? Is there old information that needs to be removed? Does the site make use of video and/or sound?

Aesthetics

Is the site pleasant to look at and visit? Do the colors and images add or detract from the navigation of the site? Do the pages use a common color theme? Does the layout of the site vary a lot from page to page?

Comment [A19]: What are the instrucimplementation? How long does this take

participants need access to the web sites tindividually?

Comment [A20]: Are you having themusability testing of the web sites?

Comment [A21]: Shouldnt the answquestions be known before hand? Isnt th

the analysis that the IDs do?

Comment [A22]: How can a user tell content is not present?


22/26


Appendix F

Level 1 Survey


23/26


Appendix F Continued

Level 1 Survey Continued:


24/26


Appendix G - Pre-Test/Post-Test Comment [A23]: Instructions.


25/26


Appendix G Contd


26/26


Appendix H

Sample Tabulation Ratings for a 10 Point Scale,

Excellent to Poor (Kirkpatrick & Kirkpatrick, p. 62)

_10_Excellent

5_Very Good

_3_Good

1_Fair

1 Poor (10x5=50)+(5x4=20)+(3x3=9)+(1x2=2)+(1x1=1)=82

The rating is 82/20 or 4.1

Comment [A24]: This looks like a 5-panswer options.

competency 8 edci577project4_jbecker (1)

Documents