KENTUCKY VICTIM ASSISTANCE ACADEMY: 2013 EVALUATION REPORT

Submitted by:

Patricia Desrosiers, Ph.D., LCSWPrincipal Investigator/ Project Manager

Christopher James, MSWCo-Evaluator/Assistant Project Manger

Western Kentucky UniversityDepartment of Social Work

1

2013Kentucky Victim Assistance Academy

Western Kentucky University

Department of Social Work

KENTUCKY VICTIM ASSISTANCE ACADEMY: 2013 EVALUATION REPORT

Introduction

Last month, the inaugural Kentucky Victim Assistance Academy, KVAA, took place at Barren River State Park in Lucas, Kentucky. This academy has been developed over the past two years by professionals who work with victims from different areas across the state. Victim Assistance Academies started at the national level in 1994 and have since branched out into individual state academies. There were approximately 50 people who attended this year’s academy in Barren County.

This week-long training provided victim advocates with tools and skills they could apply in their work with victims of crime. Before arriving at the academy, each student was required to complete approximately eight hours of pre-work that covered introductory material and the history of victim advocacy in the United States. The week began by covering the basics of victim advocacy; including topics such as the Victim Bill of Rights, confidentiality, mandatory reporting, and trauma informed care were covered. The academy then focused on the complex ins and outs of the criminal justice system and how a working knowledge of this system can be beneficial for victim advocates. On day 3 attendees learned about victim compensation, the parole board, restorative justice, and the many impacts of crime. The next main section of the academy gave attendees the opportunity to learn about how to effectively work with victims who have disabilities, immigration or language access challenges, or cultural barriers that reduce service utilization. The academy concluded with information on crisis management, vicarious trauma, and provider self-care.

Throughout the week there were several activities which deserve highlighting. These include a question and answer session with a panel of veteran victim advocates, a mock crime scene which led to a mock trial, and even a visit from a nationally recognized victim advocate who helped start the first National Victim Assistance Academy, Anne Seymour. We received a great deal of positive feedback from participants and have begun work on preparing for next year’s academy.

2

KENTUCKY VICTIM ASSISTANCE ACADEMY: 2013 EVALUATION REPORT

Presentation of Results

The evaluative results from the 2013 academy are displayed in four sections. These sections are divided into the pre-test and post-test comparison, evaluation data submitted by participants, and recommendations for next year’s Academy.

I. Pre-test and Post-test Comparison

At the beginning of the academy, each participant submitted a completed pre-test. This test was composed of 27 multiple choice and true or false questions. Participants were instructed to complete this test before they began work on their pre-work assignments. The graph below shows the average score for each question on the pre-test in blue. On the last day of the academy, participants were required to complete a post-test. Again, this test was 27 multiple choice and true or false questions, but was different that the pre-test. Although the specific questions varied from the pre-test, they did come from the same content areas which allowed for a direct comparison between the test questions. The average scores for each section on the post-test are displayed in the graph below in red. Investigators chose to use two different examinations to safeguard against threats to validity, specifically the testing effect.

Investigators utilized a t-test to determine if the differences between the mean scores on the pre-test and post-test were significant. The pre-test mean with 48 tests (N=48) was 67.28 %; the post-test mean with 45 (N= 45) was 73.09%. The pre-test scores ranged from 41 to 85%, and the post-test scores ranged from 48 to 100%. An independent samples t-test showed that the two groups significantly differed in their average change of test score, t (1, 91) = -2.607, p = .011.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 270

0.10.20.30.40.50.60.70.80.9

1

Pre/Post Test Question Comparison

PreAvgPostAvg

Test Question Number

Per

cen

t Co

rrec

t

3

It should be noted that 6 participants left on Thursday afternoon skipping the Friday morning training. Those participants did complete the post-test, but some of them did so in less than 5 minutes.

There was one comment on the pre-test:

“Pre test was insightful and helpful.”

II. Participant Evaluation Forms

In this section, participant evaluations of each module will be described. A graph of the ratings for each module will be offered. The average ratings for module content and module presenters will be separated. Additionally, these ratings will be discussed, and select participant comments will be offered to clarify the ratings further. Lastly, a discussion of the overall academy ratings will be provided.

A. Pre-Work Module

The table above depicts the pre-work modules 1-6. The average module scores ranged from 4.43 to 4.66 with 5 being the most commonly given score for these modules. The pre-work modules were self-study, so there were no presenters. The median score for the pre-work modules was also 5. Comments about the pre-work predominantly related to the amount of work being an unrealistic expectation. Many stated that the pre-work lacked organization. One commenter summed these views up well. They wrote:

1 2 3 4 5 63

3.54

4.55

5.56

Pre-Work Module

Module Number

Rat

ing

4

“Unrealistic to expect to have 8 hours to do this. Plus, very difficult to follow and decipher pre-work sections. Numbers did not correlate. One page was missing entirely from my handout.”

Four participants reported a dislike of the Intimate Partner Abuse online training. In addition, several participants found the pre-work to be “informative” and “insightful.”

B. Module #1

The table represents the average scores for the content that was presented in module 1. As you can see, the averages range from 4.5 to 4.7. The most common score for content sections 1.1 to 1.4 was 5, section 1.5 was 4, and section 1.6 was 6. The median for all content sections in module 1 was also 5. Content section 1.1 had a range of 2, sections 1.2 through 1.5 produced a range of 3, and section 1.6 had a range of 4.

1 2 3 4 5 63

3.5

4

4.5

5

5.5

6

Module #1 - Presenters

Presenter

Rat

ings

1 2 3 4 5 63

3.5

4

4.5

5

5.5

6Module #1 - Content

Section

Rat

ings

5

This chart represents the average scores for each of the presenters in module 1. The average ratings ranged from a minimum of 4.52 to a maximum of 4.81. Presenter 1 received the highest rating, while presenter 2 received the lowest rating of this group of presenters. The median and mode scores for all presenters in module 1 were both 5. Ratings for presenter 1 had a range of 2; presenters 2 through four had a range of 3, while presenters 5 and 6 had a range of 4.

Written comments focused on needing more direct explanation of the week’s organization, content being dry and lengthy, and suggestions for change. Participants suggested “more interactive activities”, “moving the panel to the end of the week because knowledge stems questions,” and “have someone give an overview of the topics to be covered over the five days to begin.” The instructors for this day were characterized as “very warm and engaging” and “good speakers!!”, and the content was called “a great deal of good information.”

C. Module #2

As evident by the chart above, content areas in module 2 had more variation in ratings as compared to the previous two modules. The average content ratings for module 2 ranged from 3.87 to 5.4. The median score for content sections 2.1 to 2.7 was 5, section 2.8 was 4, and section 2.9 was 5.5. The mode for sections 2.1 through 2.6 was also 5; while the mode for section 2.8 was 4 and sections 2.7 and 2.9 was 6. Content sections 2.1, 2.3, 2.4, 2.6, and 2.7 all produced a range of 3, while sections 2.2 and 2.4 both produced a range of 4. Section 2.8 had the highest range at 5.

1 2 3 4 5 6 7 8 93

3.5

4

4.5

5

5.5

6

Module #2 - Content

Section

Rat

ing

6

1 2 3 4 5 6 7 8 9 10 11 12 133

3.5

4

4.5

5

5.5

6

Module #2 - Presenters

Presenter

Rat

ings

This chart represents the average ratings each of the presenters in module 2 received from participants. The average scores ranged from a low of 3.87 to a high of 5.41. The median rating for the majority of presenters in module 2 was 5. Exceptions include NM who received a median rating of 3.5 and KS and TLW who both received median ratings of 6. The mode of the ratings for the most of the presenters in this module was also 5. However, the mode of ratings for NM was a 4, while the mode of ratings for VW, KS, MLW, JH, and MB was 6. The range of ratings for presenters in module 2 was primarily (75%) 3 and 4, with there being slightly more presenters with a range of 3.

Comments centered around the varying presentation styles of this module. The material when coupled with interactive activities, such as the mock grand jury, the mock trial, and the jeopardy game, seemed to better clarify material for participants and keep their attention. Participants stated:

“Grand jury info was a little confusing. The mock grand jury did help. The mock trial was very good, kept everyone’s attention.”

“The mock trial was not only entertaining, but also very informative. Speakers on this day were very interesting and energetic.”

Participants appreciated having hard copies of the PowerPoint presentations that were being used. Suggestions from participants included having the Kentucky collegiate mock trial team perform the mock trial, have a Kentucky parole board representative speak on their section, and they wanted more information from the law enforcement and community advocacy sides of working with victims.

7

D. Module #3

Module 3 contained only two different sections of content. The average ratings for the sections in this module were the highest among training modules and were both calculated at approximately 5.7. In addition, the median and mode scores for content in module three were also highest and were both 6. The ranges of ratings in this section were both 3.

The two different sections of content in module 3 were both presented by the same presenter. Similar to the content ratings, the presenter in module 3 received the highest ratings among all the presenters at the conference. The average ratings for each section the presenter covered were 5.76 and 5.79. The median and mode of the ratings was calculated for both sections and were found to be 6. In addition, the range of scores for this presenter was 2 for both sections that were presented.

1 23

3.5

4

4.5

5

5.5

6

Module #3 - Presenter

Presenter

Rat

ing

1 23

3.54

4.55

5.56

Module #3 - Content

Section

Rat

ing

8

Participants seemed to thoroughly enjoy hearing from the national victim advocate expert, Anne Seymour, and not one negative comment received. They particularly enjoyed the “interactive” and “energetic” presentation style and several commented that they hope she is able to return in the future. To quote one participant, they found this module to be “awesome, informative, and holistic.”

E. Module #4

The content areas in module 4 received average ratings ranging from 4.5 to 5.19. The median score of the content ratings for each section is this module was 5. The mode of the ratings was 5 for sections 4.1, 4.2, and 4.4, while section 4.3 had a mode of 4. The range of ratings for section 4.1 was 2, section 4.2 was 5, and ratings for sections 4.3 and 4.5 had a range of 4.

There were a total of five different presenters in module 4. The average ratings for presenters in this section ranged from approximately 4.7 to 5.3. The median and mode of

1 2 3 4 53

3.5

44.5

55.5

6

Module #4 - Presenters

Presenter

Rat

ing

1 2 3 43

3.54

4.55

5.56

Module #4 - Content

Section

Rat

ing

9

the ratings was 5 for presenters 1, 2, 3, and 5, while presenter 4 received a median and mode rating of 6. The ratings for presenters 1 and 2 had a range of 2, presenter 3 had a range of 3, and scores for presenters 4 and 5 had a range of 3.

Again, the participants reported an overall appreciation for the interactive activities, having PowerPoints available to follow along, and felt the information was applicable. There were a few participants who reported the immigration and disabilities sections of this module were too lengthy. A good representative comment given by a participant was:

“I enjoyed the activities that the disabilities presenters had us do. I enjoyed the video clip and information on when/how to use an interpreter.”

F. Module #5

Module 5 had a total of 3 content sections. The average ratings for these sections ranged from 4.9 to 5.1. Similar to some of the previous sections, the median and mode scores of the ratings for each of these sections were all the same; the median of the scores was 5 and the mode was 6. In addition, the range of the ratings for each of these sections was 3.

1 2 33

3.54

4.55

5.56

Module #5 - Content

Section

Rat

ing

10

There were 2 different presenters for module 5. Presenter 1, 2, and 4 in the chart is the same person. The average ratings for these two presenters ranged from 5.22 to 5.34. The median score of the ratings for the first presenter was 5, while the ratings for the second presenter had a median score of 5.5. The mode of the ratings for both presenters was 6. The rating scores for both presenters primarily had a range of 3; however, presenter one’s scores for the first section had a range of only 2.

Comments about module five were a good mixture of both positives and suggestions for improvement. Several participants reported they enjoyed the small group activities and found the videos helpful to apply the information. There were also several comments about providing more emphasis on vicarious trauma and self care when working as a victim advocate. One participant had particularly strong feelings about this module, stating:

“It’s not okay to push vicarious trauma and self care out for more information on other subjects. That’s too often what we do anyway; push that important stuff out for other work stuff. Maybe keep Friday all about these topics. Don’t set a bad example!”

G. Overall

1 2 3 43

3.54

4.55

5.56

Module #5 - Presenters

Presenter

Rat

ing

11

At the end of the evaluation form, there were three questions that addressed the academy as a whole. The average score of all three of these questions was 4.66. The first question

asked participants to rate the organization of the academy. The median score of these ratings was 5, while the mode was 6. The second question in this section asked participants to rate whether or not the academy was a good use of their time. The median and mode of these ratings were both 5. The third question in this section asked the participants to rate whether or not the content presented at the academy was appropriate to their need. The median and mode scores of these ratings were both 4. The range of scores for all three of these questions was 4.

The overall average of the content from all modules, including the pre-work, was 4.95. The overall average of the ratings for the presenters was slightly higher at 5.05.

At the end of the academy, participants provided a great deal of suggestions and comments on how to improve for future years. One comment that seemed to reoccur was that the advocates would have liked more information on EPOs and DVOs. Another suggestion that was given numerous times was to provide an opportunity for break-out sessions so the advocates can choose to learn more on topics they are interested in.

1 2 33

3.54

4.55

5.56

Overall Evaluation

Section

Rat

ing

Content Presenters3

4

5

6

Overall Averages

Averages

Rat

ings

12

Several participants also reported that the quality of the audio in the conference room was very poor. A few participants felt it would have been helpful to have actual victim’s stories that applied to each particular section throughout the week. Many of the participants also noted in their comments that they felt like the conference was a great networking opportunity. Comments that summarized the group’s feedback about the academy included:

“Improve the audio. I absolutely enjoyed this academy. Being such a new advocate, this material was very needed.”

“The sessions were good, however, it would have been nice to have modules to choose from, as much of the material was previously known while other parts were completely new. Small breakouts would have been nice. Overall, good first effort.”

“…EPOs, DVOs, more education on domestic violence. Overall, great and very informative. Great Job!”

“Valuable networking. Good, basic information on basic information. Excited about ability to get provisional national certification (NOVA). Great handouts. Appreciate speakers who have included emails and business cards for future questions.”

IV. Recommendations for Next YearA. From Anne Seymour

Anne had the following recommendations for next year’s academy:

1. The general rule for adult learning is to offer at least 10 minutes of interactive learning activities for every 50 minutes/hour of training (this equates to one-or-two learning activities per session).

2. It helps faculty if students have name plates, so they can call on participants by their actual names. You can prepare these in advance with the student’s first name in LARGE print, and have them “decorate their name plates” when they first arrive in the training room.

3. At the NVAA and SVAAs I’ve participated in, we move students EVERY day, and explain that the purpose is to ensure that they meet the entire student body and improve their network of colleagues (AND shhhh….get out of their “comfort zone” with friends!).

4. RE: seating of students: You can determine up front from student applications, those who have a few more years in the field. Throughout the week, make sure one of the more

13

“seasoned” students is at each table (“Teacher’s pets” a bit, but helpful in mentoring the newbies !)

5. It’s helpful to schedule a five-minute “review/preview” each morning to review what was learned on the previous day; and highlight what will be learned today.

6. It’s helpful to have a “parking lot” for students to put up sticky notes with questions/issues they have (which can be reviewed each morning during review/preview). A parking lot is also helpful for faculty members, who can say “we don’t have time to address that now, but let’s put it on the parking lot and we’ll make sure to cover it in review/preview.”

7. Music is essential! In advance, students can be asked to bring their Ipods and play their music. It is the best way to send folks on breaks, and they know to return when the music stops.

8. It’s helpful for faculty/staff to sit at each student table, instead of in the back of the room, to the degree possible. They are there to help facilitate any individual/small group activities, and constantly “check the pulse” of students (to make sure they still have one )!

9. For a large room, one standard microphone and two lavaliers are needed to make sure all lectures/comments/questions are heard.

10. I have a large cadre of decorative posters and inspirational quotations that can be hung high on all walls in the training room – they add a pop of inspiration and color!

11. It’s a good idea to invest in three-to-four boxes of Mr. Sketch markers – have four markers on each table, and a bunch up front for presenters.

12. AND if faculty/staff are willing, it’s really nice to have a morning or evening walk, led by a faculty or staff member. 30 minutes and slow, good bonding activity!

I personally agree with all of Anne Seymour’s recommendations, and they would cost less than $500 to implement. I recommend we implement all 12.

B. From Participants

14

At the end of the academy, participants provided a great deal of suggestions and comments on how to improve for future years. These comments were summarized in a previous section but will be repeated here. 1. More use of the Participant Manual/PreWork concepts and materials during the

training2. Less, no, or improved PreWork3. Break-out sessions so the advocates can learn about choice topics4. Improved audio quality5. Actual victim’s stories or a victim panel that applied to each particular section

throughout the week 6. More networking opportunities, have participants tell their name each time they

speak7. Resource handout for advocates8. More information on EPOs and DVOs, trauma informed care, vicarious trauma &

self-care, law enforcement role, crisis management, KCVC, and the resort, but less content in general.

9. Less information on immigration, death notification, psychiatric specifics —too detailed

10. Less antagonism towards offenders—felt the mother of 6 activity helped11. More interactive activities, games, movement (yoga/meditation)12. PowerPoint handouts13. More time with advocate panel and at the end of the week14. Shorter days, more frequent and longer breaks, free coffee15. KY Parole Board there in person16. Anne Seymour to come back17. Graduation at the end of the Academy with certificate presentation

These recommendations are reasonable, but somewhat conflicting. It is a challenge to provide more information on 7 topics (less on 2) in less time with more breaks. It is difficult to integrate PreWork concepts when they don’t have time to do PreWork. It is hard to network more when you are divided into breakout sessions.

C. From Dr. Desrosiers

Taking into account all of the feedback given from Anne Seymour, the participants, and the steering committee members my recommendations are as follows:1. Curriculum

A. Shorten the Immigration Visa section from 2 hours to 1 hour making it more basicB. Add that 1 hour onto the Vicarious Trauma and Self Care ModuleC. Change the Victim Scenario into a DV situation with an EPO required to pull in

that content.

15

D. Ensure that each 50-60 minutes of class includes at least 10 minutes of movement/art/interaction. It is best if that occurs in the middle of the hour.

E. Hold a Train the Trainer full-day event in February. We have wonderfully passionate and knowledgeable experts, and it would be best for them to have the skills to develop their own activities. Train the Trainer would benefit all. Anne Seymour has agreed to do this if we schedule her immediately. Alternatively, that is a service that OVC TTAC can provide for free. It is referred to as the “Ultimate Trainer”, and we can request Anne with no guarantee of getting her. We may get a different trainer, and that service also requires 5-6 months advance notice.

F. Revise the instructor manual following the Train the Trainer event in February. At the event the Trainers will add their activities and complete their PowerPoints providing them to KVAA staff at that time.

G. Add 2 Breakout sessions (a 1 hour on Wednesday and a 2 hour on Friday) to give participants choices about what information they need.

2. Academy Training ScheduleA. Format of the Academy could be 5 full days of 9-noon and 1-4 with 4-5pm

reserved at the end of the day for completing quiet work. Participants may choose to take this work home for completion each day. 8-9am daily will be reserved for networking activities, sign in, and review of learning each morning.

B. Add a first training day and mid training day 2.5 hour required evening activity. The first training night will be a great Keynote Speaker following a victim panel with either a variety of victims or a specific type of victim (other Academies have had great luck with that). Mid training will be a fun activity like the history jeopardy. Mandatory lunch on the second training day will include a special speaker on a select topic. It would be wonderful if we could get a sponsor for that second training day lunch and/or the evening activities. This equals 40 hours face to face time if you count the 4 hours of quiet work and the Graduation Ceremony (and I do).

C. Move the advocate panel to the last morning. D. Rework the sequencing of the curriculum placing lighter topics next to heavier

ones to improve the flow of the training and reduce participant fatigue. See the suggested Agenda for specifics.

E. Add a graduation ceremony at the end of the Academy where certificates are given out and the week is celebrated.

3. Venue and InstructorsA. Keep the venue the same to reduce the time and effort involved in building a

relationship and learning the processes of a new vendor.B. Provide participants with specific instructions related to comfort at the venue

(such as bring a sweater or jacket and snacks) as well as information about local restaurants and points of interest.

16

C. Implement morning walks Tuesday to Friday to increase participant energy levels and networking.

D. Have all Instructors go through one central person for instructions on booking/payment information/etc. to reduce confusion.

E. Have a different type of victim advocate as the resident expert each day to make sure that both community and legal advocacy get equal time.

F. Purchase or rent a second lavalier, standing microphone, and some good speakers to use at this yearly training. We need better sound for sure. We should also offer music between sessions to up the energy level and signal session is starting.

G. Get topic specific posters for the room, and bring Tanya’s Training Kit.4. Participant Manual, handouts, and PreWork

A. Reduce PreWork due prior to the Academy from 8 to 3 hours requiring only the Kentucky Crime Victim Bill of Rights, Federal Court System, AND PreTest sections prior to the Academy. Place the other 4 PreWork Modules on the end of each day as work they will do in class during a quiet study time.

B. Improve the quality of the Participant Manual by clarifying the instructions, explicitly stating the purpose of the information/PreWork, and adding a resource guide with names, addresses, and phone numbers of advocate resources across the state.

C. Provide PowerPoint slide handouts OR Note Taking Guides for participants to facilitate note taking and reduce anxiety of participants. These can be provided online as we did this year with the PreWork.

These recommendations are lengthy; however, I think it is reasonable to implement them all over the course of the coming year. The large number of suggestions will certainly facilitate a fruitful steering committee discussion as we determine the most appropriate improvements to implement for the 2014 Kentucky Victim Assistance Academy curriculum.

With the success of the first year academy, it is tempting to leave the well enough alone. It is likely that these revisions and refinements will make this Academy one of the best in the country. Thanks to everyone involved for your hard work in making the 2013 KVAA a huge success.

17

This document was produced by the Justice and Public Safety Cabinet and Western Kentucky University under 2011VFGXK004, awarded by the Office for Victims of Crime, Office of Justice Programs, U.S. Department of Justice. The

opinions, findings, and conclusions or recommendations expressed in this document are those of the contributors and do not necessarily represent the

official position or policies of the U.S. Department of Justice.

18