1. Evaluation Report of Teacher Technology Training at BP School Submitted by: Ashley Hungerford Boise State University Graduate Student Submitted to: Boca Raton Preparatory School Administration

2. 2 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 Summary The administration of BP school initiated a mandatory technology training program for all teachers Pre-K to 12th grade. The administration observed that recently purchased technology was not being properly utilized in the classrooms and felt that teachers needed to be trained in how to properly operate and integrate the new technology. The program took place over a 6 week period and encompassed teachers of all grade levels. The program objectives were to ensure that: Teachers are comfortable with the technology tools. Teachers create lessons that integrate technology into their classrooms. Teachers have positive feelings toward technology. Over one year ago, the BP school received money from the schools owners to purchase a substantial amount of technology for classroom use, up to this point it has gone largely unused. The teachers have only received one hour of training on the Smartboard, and many have never taken a technology or computer class before. The administration believes that the teachers are willing to learn how to use the technology, but the training needs to be provided in such a way that it doesnt take up their personal time. The administrators of the BP school have requested this evaluation to determine if the new technology training program is the best use of the schools very limited resources. In order to determine if the goals above were met, the process of data collection began by providing a pre-training technology skills test that assessed the level of ability, and comfort, the teachers had with various technology tools. Those results were then used to assign higher ability level teachers a mentee with a lower ability level. Surveys were also provided that asked teachers to rank, using a scale, the feelings that they had towards various technology tools, prior to, and then again after, training sessions. After completing the training sessions, teachers were given a follow-up activity that would allow them to practice using the new technology skills in their classrooms until the next training session. Throughout the training, mentees met regularly with mentors, who also filled out observation checklists on the progress of their mentees. Administrators provided paperwork of prior technology integration in pre-training lesson plans for each teacher, and post training lesson plans were also evaluated for incorporation of technology integration. A formative assessment of the program was completed at the mid-way point; the formative assessment took place in the form of a meeting; the meeting was informal and all participants were present to suggest improvements for the final weeks of the program. The results of the program have been processed and all of the data suggests that the program is meeting its current goals. Teacher surveys indicate an increased level of comfort with technology and an overall improvement in the attitude of the teachers towards technology. The mentors observation checklists also indicate an increase in the skill level of the teachers. Incorporation of technology skills in teacher lesson plans attained 100% teacher participation, and it was reported in the observation, that all mentee teachers utilized at least two forms of technology in their classrooms each day after the first training session. http://www.fotopedia.com/items/flickr-5146213309 3. 3 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 Description BP school administrators asked the question, Can 3-week technology training sessions for teachers improve the efficiency, effectiveness, and impact of technology at BP school? The school identified the following objectives for the program: 1. Teachers are comfortable with the technology tools. 2. Teachers create lessons that integrate technology into their classrooms. 3. Teachers have positive feelings toward technology. The technology training program at BP school took place in two 3-week sessions over a 6 week period. Each 3-week session of the training was composed of a formal two hour technology training in the early part of the first week, which provided instruction and practice in the areas of hardware, software, and internet technologies. The remainder of each 3-week training session involved two formal meetings with an assigned mentor. The first formal meeting was to provide assistance with technology integration into existing lesson plans; this meeting was followed-up by a mentor observation of the lesson. The purpose of the second formal meeting between mentor and mentee was to work together to write new lesson plans for the following week using the technology skills recently acquired. Informal meetings between mentors and mentees were encouraged in order to increase the rapport and comfort level in the partnership. Aside from the mentor observation, teachers, including mentors, were observed by a member of the administrative team. The third week of the training session involved a meeting between all involved parties in order to complete end of session surveys and discuss the progress of the program. The training program was attended by 35 teachers, the teachers were given a skills test to determine which teachers were skilled enough to act as mentors. Upon reviewing the results 15 teachers acted as mentors, and the remaining 20 teachers were mentored by a teacher or an administrator. Throughout the program, both the mentors and mentees were surveyed pre and post-training in order to determine their comfort level with technology, as well as their attitude towards technology. Various means of data collection were utilized in order to properly assess the success of the training program in meeting its objectives. Data collection took the form of surveys, observation checklists, skills testing, lesson plan data, and informal meetings and observations. Evaluation Method The participants of the technology program consisted of 35 teachers and 5 administrators, this is the faculty of BP in its entirety. The week prior to the programs initiation, the teachers and administrators were given a technology skills test in order to break them into smaller groups. 20 teachers were identified as being low ability level technology users, and were therefore assigned the role of mentee. 15 teachers and the 5 administrators were identified as being of a more advanced ability level, and were therefore assigned as mentors. All classroom teachers and administrators attended the training sessions, and all classroom teachers were required to complete the various aspects of the training program; the administrators were only required to complete the mentor observation forms and attend meetings, they were not required to do their own lesson plans or use technology in a classroom. The training program was led by the schools 4. 4 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 technology director, who also acted as an administrative mentor, and evaluator. All of the stakeholders for this program were directly involved in each step of the training and evaluation. The initial evaluation process began with a meeting; the administrative staff at BP school had decided to initiate a technology training program and they wanted to ensure that the program was worth the time and money that would be required for long-term implementation. The evaluator met with key stakeholders to identify the needs of the school, and the goals and objectives of the training program. These meetings allowed the evaluator to determine the intended outcomes, and the objectives to be measured. The program evaluation began by providing a technology skills test to all participating faculty, including administration. All faculty were also provided with a rank survey that requested information regarding the teachers comfort level with technology. The teachers skill levels were analyzed, and all participants that scored below a 70% were identified as having low technology ability levels. All participants that scored above a 70% were identified as having higher ability levels. These test results were then compared against the teachers surveys to compare their skill level with their comfort level and attitude towards technology. The bottom 20 scores in both skill level and comfort and attitude were identified and placed into a mentee group. The top 15 teachers and top 5 administrators were assigned as mentors. The mentors then met with the technology trainer/evaluator to discuss mentor roles in data collection, and to encourage mentor input and ideas for the program. Mentors then met with their mentees to discuss their role in the process; the meeting took place in an informal location with food provided to encourage participation and build rapport. The following day mentors completed a pre-training observation in their mentees classroom. The observation checklist included questions related to current technology usage by the mentee teachers. Monday, the following week, all participants in the program (mentors and mentees) attended the first technology training program, which took place during school hours. The two hour program included instruction on computer hardware basics including peripherals, but the majority of the time was specifically geared towards learning how to properly use the learning management system, Managebac, and how to use the Smartboard with the Smartboard Notebook program. Teachers were given the last 20 minutes of the training time to go to their rooms with their mentors and practice the skills they had just learned. The following day the mentors met with the mentees to help them add technology integration into their lesson plans. The formal meeting took place during teacher conference time and lasted approximately 35-40 minutes. On Thursday, of the same week, all classroom teachers were observed by their mentors and an administrator to see if technology integration was happening in the classrooms; formal observation checklists were completed. The following day, mentors and mentees met again to prepare the following weeks lesson plans to integrate technology. During week two, mentors made a surprise visit to mentee classrooms to complete a formal observation sheet. At the end of week two all participants once again filled out the comfort level and attitudes surveys. During week three all participants met to discuss the findings and recommend changes for the second session. 5. 5 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 The second session followed the same format, but the lessons addressed email, web 2.0 sites, and Microsoft Office. At the end of the second 3-week session another final survey, skills test, and informal meeting was completed. Results Technology Skills Test The table below shows the results of the technology skills test. The scores show that 20 teachers scored below a 70%, and were placed in the low ability level group. 15 teachers and 5 administrators scored above a 70%, and were placed in the high ability level group. These numbers will be used in conjunction with other data to determine mentor and mentee groups. The table also shows the skills test results for teachers after the training program. There is a marked increase in the skill level of the teachers. 8 teachers remained in the under 70% category, while 32 teachers and administrators scored above a 70%. Table 1: Technology Ability Skills Test Under 70% Over 70% ability 10% - 40% 41% - 70% 71% - 80% 81% - 100% Pre- Training 14- teachers 6- teachers 8- teachers 1- Administrator 7- teachers 4- Administrators Post-Training 3- teachers 5- teachers 14- teachers 13- teachers 5- Administrators Comfort Level With Technology Teacher Survey The table below shows the pre-training (before any sessions) results, the midway (after one training session) results, and final (after both sessions) training results from the teachers surveys regarding their comfort level with technology. The results indicate that prior to the training the number of teachers that felt uncomfortable to somewhat uncomfortable with almost all aspects of technology was significantly higher than those that were somewhat comfortable to comfortable with almost all aspects of technology. The results also indicate a steady change in these results as each training session took place. The numbers began to progressively increase in the comfortable range as they decreased in the uncomfortable range. This would indicate a strong positive correlation between the training program and increased comfort level of the teachers when using technology. 6. 6 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 Table 2: Comfort Level with Technology Survey Results Very Uncomfortable Somewhat Uncomfortable Somewhat Comfortable Very Comfortable Training Time Line Pre Mid Final Pre Mid Final Pre Mid Final Pre Mid Final Computer Hardware 13 8 3 10 9 6 9 15 23 8 8 8 Computer Software 15 15 3 9 9 4 9 9 20 7 7 13 Smartboard Hardware 7 4 0 7 8 1 16 18 20 10 10 19 Smartboard Software 15 5 3 15 10 8 8 15 17 2 10 12 Email 2 2 0 8 8 0 20 20 15 10 10 25 LMS 17 8 4 15 10 8 2 12 18 6 10 10 Peripherals 10 6 1 10 10 1 10 13 18 10 11 20 Web 2.0 sites 25 25 5 10 10 10 4 4 15 1 1 10 Total 104 73 19 84 74 38 78 106 146 54 67 117 Teachers Attitudes Towards Technology Survey The chart below shows the attitudes of the teachers and administrators towards technology while participating in the technology training program. The data indicates that the teachers have varying ranges of attitudes towards technology prior to the training sessions; no one attitude is predominant over another. After the training sessions the numbers progressively change showing an increase in the positive attitudes of teachers towards technology. This would indicate a positive correlation between the training program and positive attitudes towards technology. Chart 1: 0 2 4 6 8 10 12 14 16 18 Very Negative Somewhat Negative Neutral Somewhat Positive Positive Teacher's Attitudes Towards Technology Pre-Training Mid-Training Final 7. 7 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 Lesson Plan Integration Results The chart below shows the number of times that teachers at BP school integrated technology into their lesson plans, per week, prior to training and then measured again throughout training. The results indicate that pre-training more than one-half of the teachers at BP school were not including technology in their lesson plans. After the training 100% of teachers integrated technology into their lesson plans, and the number of times technology is mentioned in the lesson plans per week has steadily increased since the training program began. Chart 2: Classroom Observation Checklist Results The chart below shows the results of the classroom observations. The mentors and administrators were provided with a simple checklist to indicate whether the teacher they were observing used the various technology resources available to them appropriately during the time the observer was in the classroom. The results indicate an increase in the proper usage of technology resources. It should be noted that the observers planned to visit the classroom when technology was scheduled to be used, and the teachers were aware that an observation would be taking place at some point during the week. Chart 3: Pre-Training week Week 1 Week 3 Week 6 0 5 10 15 20 25 Number of Times Per Week Technology was Integrated into Lesson Plans Used 0 Used 1-5 Used 5-10 Used 11+ 0 20 40 60 NumberofTeachers Technology Skills Observed Proper Technology Use Observations Final Mid-Training Pre-Training 8. 8 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 Discussion The purpose of the evaluation was to determine if the technology training program at BP school would be a wise investment in order to improve the technology usage and overall technology friendly culture that the school desires. In order to make that determination the school administrators set objectives that it wanted the program to meet. It is the determination of the evaluator that the training program currently being implemented at BP school did in fact meet the objectives set by administrators. The data that was collected throughout the technology program would indicate that the program has been successful in accomplishing each of the 3 objectives. As with any method of data collection, there is room for error, and consideration must be given to the possibility of flawed outcomes due to the presence of the evaluator, the mentors, and the administration in the classrooms throughout the evaluation process. Limitations of the evaluation that should be taken into consideration are as follows: The program was only observed for a period of six weeks, the equivalent of two training sessions. The program was designed, implemented, and evaluated by the same person, and while the evaluator made every attempt to remain unbiased, for the sake of full transparency it must be mentioned. The administrators of the school took an active role in the observation of the teachers, which forced teachers to put a higher value on the implementation of the new technology, which might not have been the case otherwise. Finally, the teachers were all required to fully participate in the program, forced participation often makes full buy-in difficult. Taking into consideration the above limitations, the program did meet its objectives. The first objective was to ensure that teachers are comfortable with the technology tools. By analyzing the data from table 1, table 2, and chart 3 it was determined that all teachers showed some increase in their comfort with, and ability to operate, the technology in their classrooms. The second objective was to ensure that teachers create lessons that integrate technology into their classrooms. This objective was met by having the mentors work with their individual mentees to create integrated lesson plans. The success of this objective can be seen in chart 2, which shows an increase in the number of times teachers used technology integration in their weekly lesson plans. The third objective was to ensure that teachers have positive feelings towards technology. This objective is more difficult to assess due the subjective nature of a feeling or attitude, and the fact that the accuracy of the data relies upon the teachers honesty in their attitudes towards technology survey. By examining chart 1, the evaluator was able to distinguish that the overall attitudes of the teachers towards technology became increasingly more positive as the training sessions progressed. It has been mentioned several times that after the first training session all program participants met together, as an informal formative assessment of the program, in order to discuss ideas for changes to the program in the following session. Discussions took on a surprisingly positive turn, and suggestions for the program centered on the subject matter being taught. Many teachers suggested allowing faculty members to have some input about the things that they wanted to 9. 9 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 learn about with regards to technology. Other suggestions were to somehow provide a reward or stipend to the mentors for the additional work that the program requires. One of the biggest concerns that teachers expressed was the short timeline for the training process. Many mentors wanted to spend more time with the mentees in between training sessions. The final recommendation was to consider making some of the training sessions optional. It is the recommendation of the evaluator that the program be continued with a few adjustments being made to improve teacher and program success. Recommendations are as follows: 1. The training timeline should encompass a one month period. The teachers indicated in the formative assessment discussion that they felt as though the meetings were too close together and it was particularly difficult for mentors to find the time to assist their mentors in the way they had hoped. In considering the cost benefit analysis of the program, an increased session timeline would benefit the school financially. Currently the school uses substitutes to provide coverage for mentors when they need to meet with mentees. By increasing the timeline, teachers would be able to meet with mentees during conference periods due to the flexibility of the scheduling. 2. If the progress continues, the mentor/mentee relationship aspect of the program would need to be altered. Some suggestions for future alterations to the program would be to assign all new teachers, and those still struggling with technology integration, a mentor. The remainder of the teachers would be assigned peer reviewers for each technology session. It would also be recommended, that teachers be encouraged to lead technology training sessions. The school can increase teacher buy-in, and provide an opportunity for teachers to take ownership of their learning. 3. Allow teachers to suggest technologies that they would like to learn more about. In the formative discussion, teachers expressed the desire to learn many new technologies and were hoping to be able to choose what lessons they would learn. It is the determination of the evaluator that by allowing teachers the freedom to suggest the subject of the lessons the BP school allows teachers to take part in their own learning, thereby increasing the program buy-in. 4. Teachers and administrators should work together to agree upon mandatory and voluntary training courses. The nature of the program requires teachers to buy-in to the program in order for it to be successful. By creating training programs that provide options for teachers BP school increases teacher buy-in. Teachers expressed their willingness to continue the program and attend programs that the school deems essential, but they would also like to have some trainings that are optional and perhaps even more specified to the skills they would most often use in their classrooms. 10. 10 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 Invoice Date: 12/12/2013 Invoice # 100 Ashley Hungerford 1111 My Street My City, Florida 33333 954-555-5555 Fax 954-555-5455 To BP school 1111 My school School City, Florida 33334 561-555-5555 Customer ID BP1984 E v a l u a t o r F e e s $ 5 0 . 0 0 p / h N u m b e r o f H o u r s L i n e T o t a l Initial meeting 2 $100.00 Creating tests and surveys 3 $150.00 Training time 4 $200.00 Teacher Observations 6 $300.00 Additional Data Collection 3 $150.00 Data Analysis 48 $2400.00 Final Evaluation Report 96 $4800.00 Total $8100.00 Travel Fees 50 cents per mile Number of Miles Line Total 15 days 30 per day $225.00 Total $225.00 Supplies Line Total Paper, ink, copies $60.00 Total $60.00 Miscellaneous Line Total Daily expenses (food etc) 15 days on site $150.00 Total $150.00 T o t a l $8535.00 Make all checks payable to Ashley Hungerford Thank you for your business! 11. 11 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 Appendices Technology Skills Test (based on questions from the Florida Gulf Coast Technology survey) Hardware Do you understand the basic function of computer hardware components such as the CPU, monitor, keyboard, and file storage? Can you find and start a program? Can you navigate between programs? Can you save files to the hard drive or removable storage, such as a CD or flash drive? Can you exit or quit an application? Can you log off a computer? Can you shut down a computer properly? Software Can you create a new document? Can you save a document? Can you cut, copy and paste text? Can you change font style and size? Can you utilize spell-check? Can you center text? Can you change the line spacing in a document? 12. 12 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 Can you set margins? Can you change the page orientation from portrait to landscape? Can you include page numbers? Can you use headers and footers? Can you create a numbered or bulleted list? Can you create a table? Can you insert graphics, images, or clip art? Email Do you know how to access your school email account? Can you read email messages? Can you compose and send email messages? Can you reply to an email message? Can you manage your email by moving messages between folders, forwarding messages and deleting messages? Can you send attachments through email? Can you create contacts in your address book? Can you create class contacts lists? LMS (Managebac) Do you know your Managebac login? Can you read, respond to, and post messages on the discussion board in Managebac? Can you view your units within Managebac? Can you create, alter and restore your units in Managebac? 13. 13 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 Can you post assignments on Managebac? Can you upload/download files to and from Managebac? Can you create a drop box for students to submit an assignment? Can you take attendance on Managebac? Peripherals Can you send a document to the 500ci printer? Can you send a document to the 5050 printer? Can you hold a print job as a private print? Can you access your account on the copier? Can you scan and email yourself a document on the copier? Can you use the document camera to take an image? Can you use the document camera to show live images? Smartboard Can you turn on the Smartboard? Can you use the pens/eraser? Can you turn on the projector? Do you know how to calibrate the screen? Do you know how to access the onscreen keyboard? Do you know how to freeze your screen? Smart Software Do you know how to open Notebook? Do you know how to create an interactive document? Do you know how to access images on the gallery? 14. 14 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 Do you know how to find pre-made interactive lessons? Web 2.0 Do you know how to login to your Voicethread account? Can you create a Voicethread? Can you manage the voicethread by allowing others to view and edit? Can you login to your Glogster account? Do you know how to create a Glog? Can you add video, text, sound, and images? Do you know how to login to your Pixton account? Can you create a comic strip? Do you know how to login to Powtoon? Can you create a Powtoon? Do you know how to login to Prezi? Can you create a Prez? Can you add additional bubbles? Do you know how to change your path? Can you use any other Web 2.0 programs 15. 15 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 Comfort Level Survey Attitudes Towards Technology Survey 16. 16 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13 Observation Checklist 17. 17 Evaluation Report of Teacher Technology Training at Boca Raton Preparatory School 12/13/13