evaluation plan and report mizzou k-12 co-teach course · 2019. 11. 26. · evaluation plan and...

Evaluation Plan and Report Mizzou K-12 Co-Teach Course

Authors: Shann Bossaller, Abigail Fidler, Glorimar Rodriguez and Susan Detrie

IS LT 9455 | Spring 2016 | 05/09/2016

Team 3: Bossaller, Detrie, Fidler & Rodriguez IS LT 9455 | Final Evaluation Plan and Report

05/09/2016

2

Contents Executive Summary ....................................................................................................................................... 4

Introduction .................................................................................................................................................. 5

Background ................................................................................................................................................... 5

Overview ................................................................................................................................................... 5

HSE ............................................................................................................................................................ 6

Mizzou K-12 Online ................................................................................................................................... 6

Co-Teach Course ....................................................................................................................................... 7

Purposes ........................................................................................................................................................ 8

Stakeholders .................................................................................................................................................. 9

Decisions ..................................................................................................................................................... 10

Questions .................................................................................................................................................... 10

Participant Sample ...................................................................................................................................... 11

Interview Protocol ................................................................................................................................... 11

Survey Protocol ....................................................................................................................................... 12

Usability Testing ...................................................................................................................................... 12

Subject Matter Experts ............................................................................................................................ 12

Formative Evaluation Methods ................................................................................................................... 13

Summative Evaluation Methods ................................................................................................................. 15

Logistics ....................................................................................................................................................... 18

Timelines ..................................................................................................................................................... 19

Team Member Contribution ................................................................................................................... 20

Budget ......................................................................................................................................................... 21

Appendix A: Formative Data Collection Instruments................................................................................... 40

A1 Interview: K-12 Online Administrators/Lead Teachers ...................................................................... 40

A2 Survey: HSE Teachers/Coordinators ................................................................................................... 43

A3 Usability Test: Task/Observation Protocol ......................................................................................... 45

A4 Usability Survey: SUS Usability Reaction ............................................................................................ 46

A5 Management Processes and Expert Review ....................................................................................... 48


05/09/2016

3

Appendix B: Summative Data Collection Instruments ................................................................................. 49

B1 Interview: K-12 Online Administrators/Lead Teachers ...................................................................... 49

B2 Survey: HSE Teachers/Coordinators ................................................................................................... 51

B3 Usability Test: Task/Observation Protocol .......................................................................................... 53

B4 Usability Survey: SUS Usability Reaction ............................................................................................ 54

B5 Management Processes and Expert Review ....................................................................................... 56


05/09/2016

4

Executive Summary This project consists of a formative and summative evaluation for the Mizzou K-12 Co-Teach model whose main objective is to create a bridge between the University of Missouri Lead Teachers and Brazilian HSE Teachers with the use of Canvas LMS as the medium of instruction and communication among the parties. The formative evaluation seeks to address questions regarding the Mizzou K-12/HSE Teacher implementation process, reactions to the Co-Teach model from the point-of-view of the Brazilian teachers, as well as the project management practices used in the model. The goal of this plan is to conduct a thorough evaluation and to present actionable recommendations that will positively affect the Co-Teach course initiative. To develop those actionable recommendations, this plan has already made use of relevant pilot data gathered through exploratory interviews. From that important pilot data, purposes and questions were developed along with direction towards several instruments including a survey, interviews, usability testing and expert reviews. This plan will complete the formative portion of the report with the summative being completed at a later date. Screenshot Example:


05/09/2016

5

Introduction The Co-Teach Team evaluation plan is focused on evaluating the Mizzou K-12 Co-Teach model and objectives. It presents what the Co-Teach model is, provides background information on its mission, objectives, and expected outcomes. It provides information regarding High School Servicos Educacionais (HSE), the University of Missouri’s educational partner and liaison in Brazil along with its purpose and expectations. It provides and overview of what the Mizzou K-12 program is and its expected outcomes, its participants, clients and stakeholders. This plan also includes instructions and sections of both the formative and summative evaluation plan for the program, the questions, methodology, timeline, budget and logistics of the plan to serve as a guiding map for the whole evaluation process. The evaluation plan seeks to find out through the formative and summative evaluation, how successful or not the instructional design is, its implementation and management practices are and how can they be improved to further continue on with the program. The plan questions will be researched and answered through different qualitative and quantitative methods such as surveys, interviews, expert reviews and a usability study. The Co-Teach team, in charge of evaluating the Co-Teach model at Mizzou K-12 and reporting the findings are: Shann Bossaller, Abigayle Fidler, Susan Detrie, and Glorimar Rodriguez. All members have worked harmoniously and synchronously to brainstorm evaluation ideas, conduct interviews, provide feedback and developing and implementing the formative and summative evaluation in order to provide valuable information to the client regarding the program, its implementation, management practices and outcomes to further improve the program. The client (Dr. Krista Galyen of Mizzou K-12 Online) described her overall project needs exactly as: “What are we doing well, what aren’t we doing well? What should we be improving?” She empowered us to look at any and all aspects managing, implementing and executing the Co-Teach course and model. Her ultimate desired outcome is to receive actionable feedback which will positively affect the Co-Teach course model is some tangible way.

Background

Overview The University of Missouri- Columbia College of Education has an accredited online initiative called “Mizzou K-12 Online” whose stated mission is: “To develop students’ educational potential through innovative, high-quality, and an academically rigorous curriculum, supplemented by customizable learning options and exemplary student support.” They have partnered with a private Brazilian company called “High School Servicos Educacionais” (HSE) whose primary purpose is to provide Brazilian students with an intensive English language immersion program paired with the simultaneous goal of their students earning a US high school diploma. To support HSE’s primary purpose, they partnered with Mizzou K-12

http://hseducacional.com.br/site/


05/09/2016

6

Online in August of 2015 to support that objective through a developed blended learning environment having two components, one being online and the other being onsite. The online component is provided by Mizzou K-12 Online through an interactive Canvas LMS platform designed by their highly trained educators who also play a strong role with instructive curriculum development. The onsite Brazilian component is where HSE coordinators and teachers work directly with students, all of which use the online component to support their efforts. The “Co-Teach” course instructional design intends to facilitate collaboration between HSE Teachers/Coordinators and Mizzou K-12 Online educators and is located on the Canvas LMS. That collaboration entails both curriculum development and learning how the Canvas LMS platform operates both as teachers use it and how students experience it. Of important note, as of January 2016, this is Mizzou K-12 Online’s very first use of this approach with either a domestic or international partnership.

HSE In August of 2015, HSE contracted with K-12 Online to supply both human and technological resources to support their clients towards earning a United States high school diploma from an accredited pre-collegiate institution utilizing a blended model approach. HSE had a previous 15 year relationship from another US University institution towards those same goals but that relationship abruptly ended due to significant HSE dissatisfaction of the previous institutions lack of support. HSE’s previous institution contract ended in December of 2015 with the University of Missouri-Columbia service contract beginning in January of 2016. HSE clients who pay for their intensive English language immersion program are the Brazilian parents of high school aged children old enough (15 to 18 years old) to take those courses and currently there are 2100 Brazilian students enrolled to earn their US high school diplomas as part of their English immersion program. HSE works with 48 schools in Brazil which is where their client base derives from. Within each of those schools there are Teachers and Coordinators employed by HSE and Mizzou K-12 Online (via Dr. Krista Galyen) estimates there are between 200 and 300 of them. Their task is to help facilitate the onsite effort of the blended component between the students and the course material supplied (which they also contribute to) on the K-12 Online Canvas LMS platform.

Mizzou K-12 Online Mizzou K-12 Online began their service contract with HSE in January of 2016. The primary interactive means to fulfill the service contract is through use of the Canvas LMS platform. On that platform there are a variety of high school level courses for students to take including English, US History, Speech and Economics. Managing this effort, there are 5 “Lead” Teachers who play a strong role in curriculum development for all of the courses and manage the day to day activities which include answering questions from the HSE Teachers/Coordinators, grading assignments and any other Canvas LMS related


05/09/2016

7

maintenance activities. Currently, the 5 K-12 Online Lead Teachers manage the activities for the 200-300 HSE Teachers/Coordinators, who then manage the activities of the 2100 Brazilian students. To help better manage the Lead Teacher workload, Teacher Assistants were recently onboarded (April 2016). Before granting HSE Teachers/Coordinators full access to all of the efforts courses, they must first pass a basic Canvas LMS competency test to demonstrate capable performance with it. Once that occurs, they then have access to their teaching courses and can begin interacting with students. The HSE Teachers/Coordinators follow the curriculum designed by the Mizzou K-12 Online Lead Teachers to teach their Brazilian students. Lead Teachers make themselves available to answer any HSE Teacher/Coordinator questions related to the educational effort, which is to occur on the “Co-Teach” course discussion area. The most significant potential challenge with evaluating the course will be access to the participants, especially the HSE participants located in Brazil. They may be some combination of unwilling or unavailable which could impact the sample size. A smaller sample size would not only mean less data, it would also mean a more narrow range of data which could impact the based reasoning of final recommendations.

Co-Teach Course The Co-Teach course serves two primary goals (per Dr. Krista Galyen):

● To provide HSE Teachers/Coordinators an area where they can learn how to work on student assignment submissions by using Canvas tools and alternatively as an avenue to experience Canvas as their students do.

● To provide discussion groups where HSE Teachers/Coordinators can have a strong voice collaborating with Mizzou K-12 Online Lead Teachers on all aspects of the education effort.

Part of the Co-Teach model is based off of a Mizzou K-12 Online general training course called “Teaching with Technology” which is an orientation to Canvas for newly hired Teachers who will teach courses outside of the HSE Brazilian effort. Towards the HSE effort, it’s also designed having an open discussion area so as questions and general topics arise, those interactions between HSE Teachers/Coordinators and Mizzou K-12 Online Lead Teachers can be seen by all. This affords the opportunity to engage a larger participating percentage as well as a more efficient disseminating mechanism through the entire community. Those topics can range from Canvas use questions, curriculum development input, dealing with students, course material questions and any other related topic. Through these discussions, the end goal is to empower HSE Teachers/Coordinators towards all educational aspects of the project and to elevate them to the level of “Co-Teachers” working in concert with the Mizzou K-12 Online staff. Potential challenges to evaluating this course include the fact it’s not a traditionally structured course


05/09/2016

8

Purposes The purpose of the formative and summative evaluation: To describe the Co-Teach Course, objectives and its implementation on the K-12 Online Canvas LMS platform. We will be starting our evaluation with a brief description of the Co-Teach Model. The Co-Teach Model, as we have discovered through preliminary pilot interviews with stakeholders, has a specific set of goals, which in turn may or may not align with the instructional design of the Co-Teaching with Technology course we will be evaluating. Through clarifying the program goals as we understand them, only then can an evaluation determine whether the existing instructional design on the Canvas platform aligns with the intentions and purposes of the Co-Teach model as understood by the various stakeholders involved in creating its conceptual framework. Outcome: Identify the major goals of the Co-Teach Model. To identify design improvements to the Mizzou K-12 Online canvas LMS platform: Through clearly identifying the goals of the Co-Teach Model, this evaluation will look to areas for improving the alignment of the instructional design and program goals, as well as make suggestions for increased communication opportunities within the platform and improved user accessibility. During the preliminary pilot interviews of the evaluators for the project, we discovered that stated goals may or may not be clearly exhibited by the instructional design and that perceptions of how the online technology functions may or may not help meet the stated program goals. By way of example, a stated goal of the program administrators at Mizzou Online is that teachers and coordinators in Brazil feel a level of parity and on an equal footing with Mizzou Online teachers and facilitators because in the past the educators in Brazil had not had this experience. Still, in going through the course, the evaluators discovered portions of the course Co-Teaching with Technology were ‘locked’ until the Brazilian teachers complete certain steps and activities. This ‘locked until certain requirements are met’ function of the Canvas LMS is usually used for high school and undergraduate students progressing through a subject, and thus may impact not only the perception of the Brazilian teachers and their ‘equal’ relationship to the Mizzou teaching staff they are working with, but inadvertently undermine a central goal stated by the Mizzou administrators for the program. Through this evaluation, we hope to understand where the instructional design can be strengthened to convey program goals accurately. In addition, this evaluation will examine the flow and sequencing of information and ascertain if it is communicating the underlying goals, as well as delivering information required for teachers to experience the technology in a positive way. Outcome: A list of improvements that can be made to the instructional design and sequencing of information in order to increase alignment with program goals.

http://drive.google.com/open?id=119yqsxe6fhjk1n3MU3fNxTSCIvb1jiKKQj1IaSwYmQo


05/09/2016

9

To ascertain the project management practices of administering the Co-Teach course. This evaluation hopes to better understand the current use of project management principles and organization to further the program goals. During the preliminary, information seeking phase of the evaluation, it became clear that the organizational structure of the Co-Teaching with Technology course had assigned roles to various team members, but that members of the different teams functioning in these various roles had no clear and direct lines of communication between them. With an initial finding that better coordination of the various teams would assist everyone in their jobs, the evaluators hope to target certain areas for evaluation in the hopes of making suggestions to improve overall management of the project. While autonomy is often a prized component of any job, in Co-Teaching with Technology cross-purposes in initiatives will impact the time available to work towards other goals and possibly delay making needed improvements to several other facets of the project that are more critical. Additionally, the organizational structure has the potential affect not only perceptions of participants, but impact full participation. Through evaluating the current state of project management, the course will have the potential to function more efficiently and thus more productively through improvement. Outcome: A list of project management tools and suggestions that can increase productivity on the Co-Teaching with Technology team.

Stakeholders The primary client is Mizzou K-12 Online with the client contact being Dr. Krista Galyen who gave the initial pilot interview information and direction for the evaluation. The evaluation secondarily impacts the rest of the staff and lead teachers on this team, as the key points for the successful delivery of the structure and implementation of the program. Of important note to the implementation of the program, the course in all of its aspects represents Mizzou K-12 Online and any negative experiences could impact the relationship with HSE and the desired development of similar relationships in other parts of the world. In addition to the primary stakeholder, the results of the evaluation are also relevant to:

• Brazilian HSE Administrators, who conduct local student recruitment and Teacher/Coordinator hiring in Brazil.

• Teachers/Coordinators at local schools in Brazil, who work directly with the students in Brazil • Students enrolled in the program, whose academic success may depend on a well-executed

project. • University of Missouri-Columbia College of Education administration, as an institutional partner in

the program. • Lead Teachers at in Mizzou K-12 Online


05/09/2016

10

Decisions The formative and summative evaluation of the Co-teach project will provide actionable information regarding the future state and implementation of the program. This including improvements around the design and delivery of critical information. Likely decisions that will be made as a result of the evaluation are:

● Guidelines for the process of enrolling or onboarding Brazil-based teachers ● Potential design improvements related to the Canvas platform including its use, information

architecture and layout. ● Future improvements to the project management practices leading to better communication and

planning. Results from the evaluation will also help to determine what areas may have the most pressing need for improvement, and what is the most feasible decisions to implement may be, based on all considerations.

Questions In order to be certain that the points laid out in our purposes are addressed, the following questions were established to help direct both the formative and summative evaluation. These questions were established with regard to both overall purposes and the decisions to be made as a result.

Purpose: To describe the Co-Teach Course, objectives and its implementation on the K-12 Online Canvas LMS platform. Formative Question ● How are HSE Teachers/Coordinators onboarded into the Canvas LMS platform? Summative Questions ● What implementation feedback mechanisms are available for HSE Teacher/Coordinator use? ● How effectively are HSE Teachers/Coordinators onboarded into the Canvas LMS platform? Purpose: To identify instructional design improvements to the Mizzou K-12 Online canvas LMS platform Formative Questions ● What recommendations can be made to improve the usability of the Co-Teach course? ● Does the Co-Teach course instructional design support the stated course objectives? ● What are the HSE Teacher/Coordinator reactions to the Mizzou K-12 Online Canvas LMS

instructional design?


05/09/2016

11

● What are the HSE Teacher/Coordinator reactions to the effectiveness of the Mizzou K-12 Canvas LMS instructional design?

Summative Questions ● How effective is the Co-Teach instructional design for supporting the course objectives? ● How effective are the implemented recommendations to improve the usability of the Co-Teach

course? Purpose: To ascertain the project management practices of administering the Co-Teach course. Formative Questions ● What project management practices are being used to administer the Co-Teach course? ● What project management practices might improve project implementation?

Summative Questions ● How effective are the project management practices supporting the stated Co-Teach course

objectives?

Participant Sample Sample for Formative and Summative Plans

Interview Protocol The primary interview protocol sampling will focus on the Mizzou K-12 Online contingent consisting of administrators and lead teachers. They have made themselves readily available and are easily accessible either in person or through Skype or Google Hangouts. Those interviews will be recorded (with permission) transcribed, coded and qualitatively measured making note of concepts, patterns and trends. All interviews will remain confidential. Those Mizzou K-12 Online participants will be: Dr. Krista Galyen, Dr. Katherine Fishman-Weaver, Karen Scales, Stephanie Walter, Jeff Kopolow, Megan Lilien, and Gina Jensen It has been expressly stated that any and all contact with HSE Brazilian employees must initially go through Mizzou K-12 Online administrators and that HSE participation on any level was not guaranteed which represents a potential challenge. Should interviews with HSE become possible those will focus on: HSE Teachers and Coordinators


05/09/2016

12

Survey Protocol The primary survey protocol sampling will focus on HSE Teachers and Coordinators. It is the most efficient way to reach a way to reach the largest Brazilian audience without schedule coordination concerns. It’s also the one method Mizzou K-12 Online stated would be the easiest to solicit participants for. The survey instrument will have both forced response and open ended responses. Those answers will be measured both quantitatively (forced response) and qualitatively (open ended responses). We will attempt to reach the entire HSE Teacher/Coordinator pool through Mizzou K-12 Online with actual collected responses to be determined. Those HSE survey participants will be: HSE Teachers and Coordinators

Usability Testing Usability testing will based on task orientated protocol paired with a think aloud while using the Mizzou K-12 Online Canvas LMS platform. The primary design focus will be the information architecture and instructional design layout. Users will be given a task to perform, then observed and recorded when trying to perform those task. Utilizing Nielsen Norman Group findings, there will be no more than 5 users with no less than 3 attempted. It should be noted that final participation numbers will depend on those willing to volunteer their time. The currently anticipated user participants for this group will be: HSE Teachers and Coordinators

Subject Matter Experts Usability Testing IE Lab Practitioner - The IE Lab is a SISLT run lab focused on usability testing with trained usability practitioners. This project evaluation will utilize that resource having a trained usability practitioner who will be conducting and observing the task orientated protocol and compiling the results. Project Management Practices Certified Project Management Professional (PMP) - Once the current project management practices have been established and documented, consultation with a PMP certified professional (Dr. Aimee Klimczak) will occur. Dr. Klimczak will review those practices and make recommendations. The PMP certified professional credential is a widely recognized professional credential reserved for those with a working knowledge of project knowledge best practices. PMBOK Guide & Standards - The PMBOK guide is produced by PMI (Project Management Institute) and “provides standards, guidelines, rules and characteristics for project, program and portfolio management”. Benchmarking analysis will be used to compare current Mizzou K-12 Online management practices with the PMBOK guide & standards. The PMBOK guide is widely recognized across multiple

https://www.nngroup.com/articles/how-many-test-users/https://www.nngroup.com/articles/how-many-test-users/http://ielab.missouri.edu/#/lab/homehttp://www.pmi.org/certification/project-management-professional-pmp.aspxhttp://www.pmi.org/PMBOK-Guide-and-Standards.aspx


05/09/2016

13

industries as one of the most credible industry standards and reference guides for project management best practices.

Formative Evaluation Methods Formative Instruments: Description, Rationale, Implementation and Analysis Strategy Interview Protocol- The formative interview protocol is a written series of questions to ask Mizzou K-12 Online administrators and Lead teachers. The rationale for this chosen method is due to the ease of access of the Mizzou K-12 Online participants and to offer an opportunity for other information to spontaneously present itself during the interview. These questions will be asked either in person or through some electronic means of exchange like Skype, Google Hangouts or phone. If this evaluation project is able to interview HSE Teachers/Coordinators, a separate interview protocol will be developed and used exploring HSE relevant questions. Those HSE interview questions would have to be asked through some electronic means since the participants are in different countries.

Interview Analysis Strategy- The formative interview session will be recorded (with authorization), transcribed and coded for concepts, trends and patterns which will be noted and potentially become useful data for recommendations.

Usability Testing/Task Protocol- The formative task protocol instrument is focused on testing the Mizzou K-12 Online instructional design in an observed and moderated session by a usability practitioner which is referred to as “Observer as Participant”. The rationale for this is to test against usability concerns that were stated during the pilot data gathering. During a recorded session, Brazilian HSE Teachers and Coordinators will be asked to perform a series of task and encouraged to “Think aloud” while performing those task for later reference if needed. The task will be orientated around the sites usability of information architecture and the ease of locating interactive components supporting the intended instructional design. Once the task protocol is completed, participants will be directed to SUS orientated questionnaire which has both the traditional 10 SUS Likert responses as well as two additional open ended questions. This will help gauge immediate system reactions as well as exploring additional reactions through expanded responses. The questionnaire will be built using Qualtrics.

SUS Reaction Analysis Strategy- The SUS questionnaire will scored and interpreted using the officially designed process as described on the measuring site. The additional open ended questions will be transcribed and coded for concepts, trends and patterns which will be noted and potentially become useful data for recommendations.

Survey Questionnaire- The formative survey questionnaire will be an instrument deployed to HSE Teachers and Coordinators to probe evaluation questions. The rationale for this instrument is that Mizzou K-12 Online stated this would be the easiest way to access these participants and it would afford them

http://www.indiana.edu/%7Eeducy520/sec5982/week_5/qual_data_analy_ex2.pdfhttp://www.measuringu.com/blog/observation-role.phphttp://www.usability.gov/how-to-and-tools/methods/system-usability-scale.htmlhttp://www.measuringu.com/sus.phphttp://www.measuringu.com/sus.php


05/09/2016

14

the opportunity to fill out the survey during timing of their convenience. There will be both forced response and open ended questions and the total time to take the survey will not exceed 15 minutes so they aren’t viewed as timing obstacles by the participants.

Survey Analysis Strategy- Forced answers will be quantitatively compiled and reviewed for trends. Open ended responses will be qualitatively coded and categorized for concepts. Once coded for concepts, trends and patterns will be noted and potentially become useful data for recommendations.

Benchmarking- Built into the formative interview protocol, designed questions will probe current project management practices and be compiled for review and comparison. The rationale for this approach is to use recognized best practices and standards as the comparison to currently utilized management approaches.

Benchmarking Analysis Strategy- Once current project management practices are compiled, they will be benchmarked against the Project Management Institute's PMBOK guide as well as reviewed by a PMP certified professional (Dr. Aimee Klimczak). The PMBOK guide is widely recognized across multiple industries as one of the most credible industry standards and reference guides for project management best practices.

Formative Methods Table

Formative Questions Method Data Collection Technique

Data Analysis Method

How are HSE Teachers/Coordinators onboarded into the Canvas LMS platform?

Interview (To K-12)

Interview Protocol Qualitative: Transcribe, Code, Measure

What recommendations can be made to improve the usability of the Co-Teach course?

Usability Testing

(To HSE)

Observation of Task Protocol

And

Modified SUS Questionnaire

Qualitative (Think Aloud and SUS Open Ended): Document,

Code, Measure Quantitative: Score, Compile,

Measure

Does the Co-Teach course instructional design support the stated course objectives?

Survey (To HSE)

Open-Ended Survey

And

Forced Choice

Survey

Qualitative (open ended): Transcribe, Code, Measure

Quantitative (forced choice):

Scoring, Compiling, Measuring


05/09/2016

15

Formative Questions Method Data Collection Technique


What are the HSE Teacher/Coordinator reactions to the Mizzou K-12 Online Canvas LMS instructional design?

Survey (To HSE)

Open-Ended Survey

And

Forced Choice

Survey


Quantitative (forced choice): Scoring, Compiling,

Measuring

What project management practices are being used to administer the Co-Teach course?

Interview (To K-12)

Interview Protocol Qualitative: Scoring, Compiling, Measuring

What project management practices might improve project implementation?

Expert Review (K-12)

Benchmarking Protocol

Reference 1 Reference 2

Qualitative: Compare, Validate, Measure

Summative Evaluation Methods Summative Instruments-Description, Rationale, Implementation and Analysis Strategy Interview Protocol- The summative interview protocol is a written series of questions to ask Mizzou K-12 Online administrators and Lead teachers. The rationale for this chosen method is due to the ease of access of the Mizzou K-12 Online participants and to offer an opportunity for other information to spontaneously present itself during the interview. These questions will be asked either in person or through some electronic means of exchange like Skype, Google Hangouts or phone. If this evaluation project is able to interview HSE Teachers/Coordinators, a separate summative interview protocol will be used exploring HSE relevant questions. Those HSE summative interview questions would have to be asked through some electronic means since the participants are in different countries.

Interview Analysis Strategy- The summative interview session will be recorded (with authorization), transcribed and coded for concepts, trends and patterns which will be noted and potentially become useful data for recommendations.

Usability Testing/Task Protocol- The summative task protocol instrument is focused on testing the Mizzou K-12 Online instructional design in an observed and moderated session by a usability practitioner which is referred to as “Observer as Participant”. The rationale for this is to test against usability concerns that were stated during the pilot data gathering. During a recorded session, Brazilian HSE Teachers and

http://www.nap.edu/read/11344/chapter/5http://www.nap.edu/read/11344/chapter/5http://www.synergybusiness.com/files/PDF/White_Papers/benchmarking_project_management.pdfhttp://www.synergybusiness.com/files/PDF/White_Papers/benchmarking_project_management.pdfhttp://www.indiana.edu/%7Eeducy520/sec5982/week_5/qual_data_analy_ex2.pdfhttp://www.measuringu.com/blog/observation-role.php


05/09/2016

16

Coordinators will be asked to perform a series of task and encouraged to “Think aloud” while performing those task for later reference if needed. The task will be orientated around the sites usability of information architecture and the ease of locating interactive components supporting the intended instructional design. Once the task protocol is completed, participants will be directed to SUS orientated questionnaire which has both the traditional 10 SUS Likert responses as well as two additional open ended questions. This will help gauge immediate system reactions as well as exploring additional reactions through expanded responses. The questionnaire will be built using Qualtrics.

SUS Reaction Analysis Strategy- The SUS questionnaire will scored and interpreted using the officially designed process as described on the measuring site. The additional open ended questions will be transcribed and coded for concepts, trends and patterns which will be noted and potentially become useful data for recommendations.

Survey Questionnaire- The summative survey questionnaire will be an instrument deployed to HSE Teachers and Coordinators to probe summative evaluation questions. The rationale for this instrument is that Mizzou K-12 Online stated this would be the easiest way to access these participants and it would afford them the opportunity to fill out the survey during timing of their convenience. There will be both forced response and open ended questions and the total time to take the survey will not exceed 15 minutes so they aren’t viewed as timing obstacles by the participants.

Survey Analysis Strategy- Forced answers will be quantitatively compiled and reviewed for trends. Open ended responses will be qualitatively coded and categorized for concepts. Once coded for concepts, trends and patterns will be noted and potentially become useful data for recommendations. Summative survey questions will be used to measure any related improvements based on the formative project phase.

Benchmarking- Built into the summative interview protocol, designed questions will probe current project management practices and be compiled for review and comparison. The rationale for this approach is to use recognized best practices and standards as the comparison to currently utilized management approaches.

Benchmarking Analysis Strategy- Once current project management practices are compiled, they will be benchmarked against the Project Management Institute's PMBOK guide as well as reviewed by a PMP certified professional (Dr. Aimee Klimczak, if available at that time). The PMBOK guide is widely recognized across multiple industries as one of the most credible industry standards and reference guides for project management best practices. Summative benchmarking will be used to measure any related improvements based on the formative project phase.

http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.htmlhttp://www.measuringu.com/sus.phphttp://www.measuringu.com/sus.php


05/09/2016

17

Summative Methods Table

Summative Questions Method Data Collection Technique


What implementation feedback mechanisms are available for HSE Teacher/Coordinator use?

Interview (To K-12 Online)

Interview Protocol

Qualitative: Transcribe, Code, Measure

Survey (To HSE)

Forced Choice Survey

And

Open-Ended

Survey


Measuring


How effectively are HSE Teachers/Coordinators onboarded into the Canvas LMS platform?

Survey (To HSE)


And

Open-Ended

Survey


Measuring


How effective is the Co-Teach course instructional design supporting the course objectives?

Survey (To HSE)


And

Open-Ended

Survey


Measuring


How effective are the project management practices supporting the stated Co-Teach course objectives?

Interview (To K-12)

Interview Protocol Qualitative: Transcribe, Code, Measure


05/09/2016

18

Summative Questions Method Data Collection Technique


How effective are the implemented recommendations to improve the usability of the Co-Teach course?

Usability Testing

(To HSE)

Observation of Task Protocol

And

Modified SUS Questionnaire

Qualitative (Think Aloud and SUS Open Ended):

Document, Code, Measure

Quantitative: Score, Compile, Measure

What are the HSE Teacher/Coordinator reactions to the effectiveness of the Mizzou K-12 Canvas LMS instructional design?

Survey (To HSE)


And

Open-Ended

Survey


Measuring


Logistics In order to develop an effective evaluation plan for the Co-Teach Model at Mizzou K-12, team members need to keep open communication throughout the entire evaluation process and work individually in specified tasks to fill in the gaps of the evaluation. In order to complete the evaluation all team members are responsible for completing client and teacher interviews in order to gather the data needed for the evaluation. During the formative part of the evaluation the group will work together to come up with the parts for the evaluation implementation. Shannon Bossaller will on the background information section of the evaluation plan. Abigail Fidler will work with the decisions section, stakeholders and decisions of the evaluation plan. Susan Detrie will work on the purposes section, method and budget section of the plan. Glorimar Rodriguez will work on the logistics and evaluation plan timeline. All of course not limited to having team members working on other team member section for support. After compiling and putting together the data for developing the evaluation plan, Co-Teach members will work on the data analysis and report. Susan Detrie and Abigail Fidler will work on looking through the data and writing an initial report on the data findings. The data will then be handed in and Shannon Bossaller and Glorimar Rodriguez will work on reviewing the data and writing the final report to hand in to the client.


05/09/2016

19

Timelines The timelines referenced in this section aim to keep track of the evaluation of the Teaching with Technology/Mizzou K-12 Co-Teach Program. The timeline will serve as a roadmap for evaluators to aid them in keeping up with milestones and deadlines that need to be met in order to complete the formative and summative evaluation of the model. The evaluation will take 4 months to complete and members are expected to follow the timelines in order to complete the evaluation in the expected date.

Evaluation Timeline for Mizzou K-12 Co-Teach Program

Formative Evaluation

Initial Meeting/Contract February 21, 2016

Evaluation Planning February 23, 2016 – March 1, 2016

Data Collection Mizzou lead teacher interviews HSE survey Usability Testing Interviews and Expert Reviews

March 2, 2016 – April 2016 March 2, 2016 - March 12, 2016 March 29, 2016 - April 5, 2016 April 4, 2016 - April 8, 2016 April 11, 2016 - April 15, 2016

Summative Evaluation

Data Analysis and Evaluation April 17, 2016 – April 23, 2016

Data Review and Final Report April 24, 2016 - April 30, 2016

Final Evaluation Plan Review May 1, 2016 - May 2, 2016

Evaluation Plan Submission May 3, 2016

Peer Evaluation May 3, 2016

● February 2016: Initial Meeting/Team Contract

○ Members met through the zoom online video conferencing service to talk about the project, look through the evaluation plan and sign the team contract for submission.

● February 2016 – March 2016: Evaluation planning ○ Members met through the zoom online video conferencing service to discuss the

important parts of the evaluation and the steps that need to be taken in order to begin working on the data collection methods and procedures. Members decided to conduct


05/09/2016

20

interviews with the client and the Co-Teach lead teachers in order to gain a background understanding on the problems, difficulties, experiences, among other things being experienced within the model and the expected outcomes and future expectations of the program.

● March 2016 - April 2016 – Data Collection ○ Members met through the zoom online video conferencing service weekly, sometimes

biweekly to discuss the project and important tasks that needed to be completed by order of importance. From March 2nd through March 12 team members conducted meetings with the client and lead teachers either by phone or online video conferences such as Skype, Google Hangout or Zoom to gather the background and discovery information that would shape and give form to our idea and plan for the evaluation. Members kept in touch with each other through Basecamp adding documents and information important for the team and project.

● April 2016 - May 2016: Data Analysis and Evaluation ○ After collecting the necessary data from HSE teachers and Mizzou teachers via surveys

and interviews, as well as conducting the expert evaluations and observations, the data would be transcribed, added and evaluated by Susan Detrie and Abigayle Fidler. Data would then be analyzed by team members and would be added into the evaluation plan.

● May 2016 – Data Review and Final Report ○ After analyzing the data submitted by Abigayle Fidler and Susan Detrie, Shannon

Bossaller and Glorimar Rodriguez will work on drafting and writing up the final data report for submission.

● May 2016 – Co-Teach team data evaluation. ○ Co-Teach team members will submit their peer evaluation.

Team Member Contribution Each evaluator from the Co-Teach Team will take part in the evaluation process in both the formative or summative part of the evaluation. Tasks are assigned individually but are not limited to individual completion. Since this is a team, members are expected to support others if the task becomes too challenging or complex.

Evaluation planning: (February 2016 – March 2016) Susan Detrie Glorimar Rodriguez Abigayle Fidler Shann Bossaller


05/09/2016

21

Data collection: Lead Teacher Interviews: (March 2016) Abigayle Fidler Glorimar Rodriguez Shann Bossaller Susan Detrie

Data Analysis and Report:

Formative Data Analysis and Evaluation: (April 2016) Glorimar Rodriguez and Shann Bossaller Summative/Final Data Report: (May 2016) Abigail Fidler and Susan Detrie

Budget There is no budget for this evaluation from our client, and the evaluation is being conducted by graduate students for a class project and involves teachers, students and administrators from HSE and the University of Missouri High School Online. Generally everything needs to be procured free of charge but a budget was developed to cover the costs to execute the evaluation fully. Projected budget are as follows:

The major components of the budget are as follows: Cost

Zoom Professional Web Conferencing 3 months x 14.99 per month $ 45.00

Dragon Naturally Speaking: Professional transcription software $ 300.00

Final Cut Express Editing Software $ 240.00

Audacity Audio Editing/Noise Reduction (no cost/freeware) $ 0.00

Researchers: 4 researchers x 250 hours each $ 0.00

Basecamp Project Management Software: $20 per month x 3 months $ 60.00

Expert Reviewers: Project Management PMP certified: 1 reviewers x 4 hours each @$125 per hour

$ 500.00

Expert Usability Reviewers (Formative Observation) $26 per hour x 4 hours $ 104.00

Expert Usability Reviewers (Summative Observation) $26 per hour x 4 Hours $ 104.00


05/09/2016

22

The major components of the budget are as follows: Cost

5 Usability testers: Missouri (Formative) 5 testers x 2 hours each @$20 per hour $ 200.00

5 Usability testers: Brazil (Formative) 5 testers x 2 hours each @$20 per hour $ 200.00

5 Usability testers: Missouri (Summative) 5 testers x 2 hours each @$20 per hour $ 200.00

5 Usability testers: Brazil (Summative) 5 testers x 2 hours each @$20 per hour $ 200.00

Total Expenses $ 2,153

Limitations The limitations discussed solely focuses on this evaluation's formative related purposes and questions. While each question had varying degrees of specific limitation affecting it, some limitations affected all of the questions. The 3 primary limitation areas were: Participant Access, Time and Experience and Sample Size and Validity. Participant Access Access to respondents was a consistent issue, especially to HSE personnel. As an example, when calls for usability were sent out by Mizzou K-12 Online, out of approximately 250-300 HSE Teachers and Coordinators, less than 3 responded to the email to participate in usability studies. From the same pool of approximately 250-300 HSE personnel, 13 responded to the survey, a survey which averaged 1.24 minutes to take, and not all survey participants answered all of the questions. There were also times when access to Mizzou K-12 Online personnel was limited due to scheduling challenges which included multiple appointment rescheduling situations. Another limitation encountered was the lack of communication accessibility with HSE personnel. Their feedback would have been an important resource for data collection to help understand the process of teacher onboarding, the objective and focus regarding the Co-Teach model and their partnership perspective of the relationship between themselves and the University of Missouri. Asking for participation in such investigations/evaluations can be problematic with it not being unusual to experience low participation numbers. When reflecting on low participation numbers, one possible explanation could be that many of the same similar Mizzou K-12 Online personnel and HSE groups were being called on for feedback and there was simply not enough feedback capacity to support 4 groups simultaneously attempting to collect data.


05/09/2016

23

Resources: Time and Experience Time was a significant limitation which includes set constraints of assignment due dates and the lag of participant response rates. Another limitation was the inability to commit full time resources delving into every single project aspect like a practicing professional. Real life obligations such as jobs and families limited the time which could be devoted to a major class project. The initial lack of formative and summative evaluation experience was also limiting factor but with the experience of this project, the group now has some experience and would be better prepared for another evaluation project. Lack of formal data analysis skills were also a limitation and would be needed to substantially strengthen qualitative data analysis and results recommendations, especially if response rates would have been higher. Sample Size and Validity With the lack of instrument participants, the question arises of data validity and reliability related to sample size, especially as it relates to the entire usability session and primary K-12 Online Teacher/Coordinator survey. Mizzou K-12 Online administrators reached out to all 250-300 HSE Teachers and Coordinators. Out of that pool, there were only 13 survey participants who participated and 3 who volunteered to participate in the usability observation session and the session closing SUS related questionnaire. While there are results to discuss, the small sample size could be a limitation on the strength of data validity and reliability, though those results could be validated with future evaluation efforts.

Results The following results are based on the 6 formative questions outlined in this evaluation plan. Each question is posed followed by its data, discussion and recommendations based on the discussion and data. The primary recommendation focus areas relate to project management practices and usability considerations. Of note, Interview, Survey and Observation participant identities were coded to help preserve their anonymity based on IRB best practices. 1. How are HSE Teachers/Coordinators onboarded into the Canvas LMS platform? In order have a successful program teachers, assistants, coordinators need to be onboarded and provided information regarding the job they will be performing. After conducting coordinator interviews there was not much information regarding the topic, to their understanding, of how HSE Teachers and Coordinators are onboarded. Quotes from the conducted interviews and transcriptions serve as the data basis for this this discussion and recommendations.


05/09/2016

24

Data “We have zero role in that (the HSE teacher onboarding process). I don't know much about how that works on that side.”

“They are all different skill levels and none of them have teaching degrees. Some of them have so much classroom experience that they might as well have a teaching degree and they're brilliant. Others are brand new and they've never run a classroom or worked with kids. It's such a variety that at this point I feel in my communication with them that they're happy for any kind of support. I think they're experience with the previous University last year was so horrible. They're questions went unanswered, they had nothing, no support in any way. At this point they are just so happy that we respond to them with a message that for now, that's enough though that will change as we work together.” Discussion Interviews with University of Missouri Coordinators and Lead Teachers did not provide much background information into the question we posed and needed answered. It is understood that in order for Mizzou K-12 to work in Brazil, the University has a partnership with HSE to onboard and hire teachers for the program. Although HSE is the one doing the hiring, the University of Missouri program Coordinators do not play a role in the hiring of teachers and is not aware of the onboarding process. Recommendation Transparency should be provided when the hiring of teachers happens on behalf of HSE. Although University of Missouri is partnering with HSE to offer documentation and frameworks for teachers in Brazil, University of Missouri lead Coordinator or Lead Teacher personnel has no idea of the teachers they are working with and their educational background. While University of Missouri is partnering with HSE for hiring, there should be a liaison established among the former and the latter that will keep the former informed of teachers and their educational background, be it anonymously or by providing detailed teacher information.

2. What recommendations can be made to improve the usability of the Co-Teach course?

Data A logical response to this question can be: “Are usability improvements needed”? Probing that response three instruments intersecting with this question were utilized to gather data from HSE personnel. Those instruments were two separate Qualtrics electronic surveys deployed at separate times and remote real time remote usability observation sessions. One survey was given to all of the HSE Teacher/Coordinators (Appendix A2 page 43). The usability observation sessions were conducted over Google Hangouts while sharing screens and using TechSmith’s Camtasia to

https://www.techsmith.com/camtasia.htmlhttps://www.techsmith.com/camtasia.html


05/09/2016

25

record cursor movement and conversation as the user moved through the task protocol. At the end of the usability session, participants were immediately directed to a survey to close the session. The second survey (Appendix A4, page 46) used to close the usability session was based on the SUS measuring survey originally created by John Brooke in 1968. Its designed use was immediately after the usability session to gauge user response to the system along with an opportunity to contribute any other related feedback in a provided open ended text field.

• Appendix A2 Survey: HSE Teacher/Coordinator - The survey shown in Appendix A2 (page 43) had 2 questions relevant to this formative question. Those questions are shown and paired with the number of respondents and the average score. For comparison, the scoring scale is included below in Figure 1.

Figure 1

These responses all averaged to “Agree” suggesting a positive response to both questions. The related survey is further discussed under another formative question (starting on page 30).

• Appendix A3 Usability Observation- Think-Aloud Commentary - During the recorded

observation session, participants were encouraged to “Think-Aloud” as they moved through their task. All three “Think-Alouds” had similar expressions which were mirrored in the open ended comments they entered in the closing SUS usability survey. While there was a mix of commentary, the general theme of frustration seemed to be encapsulated in the open ended text field made available in the SUS survey and available in one of the following segments. The related commentary is noted on page 27 under the “SUS Related Closing Survey- Open Ended Commentary” category.

• A4 Usability Survey: SUS Usability Reaction - The SUS survey is a research proven

instrument designed to measure perceptions of usability. The survey instrument was given immediately after the Co-Teach usability session was conducted, which can be seen here. The below graph (Figure 2) shows a summary all of the responses:

http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.htmlhttps://missouri.qualtrics.com/SE/?SID=SV_8k2JMENrVdML1w9https://missouri.qualtrics.com/SE/?SID=SV_8k2JMENrVdML1w9https://missouri.qualtrics.com/SE/?SID=SV_8k2JMENrVdML1w9


05/09/2016

26

Figure 2

• Measuring SUS Usability Reaction - The SUS survey instrument is designed to use a series

of weighing measures based on each question response to calculate an overall SUS usability score. More can be learned about SUS calculation measurements on the measuringu website. Using the SUS scoring method, a 100 would be a perfect score. Any score above a 68 would be considered above average while below 68 would be considered below average. Anything below a 51 is considered a failing score. The summary below table (Figure 3) is each participant's calculated SUS score with the final

http://www.measuringu.com/sus.phphttp://www.measuringu.com/sus.phphttp://www.measuringu.com/sus.php


05/09/2016

27

averaged measure being 30.83. The specific scoring details can be found in the established raw data google drive folder (SUS Calculator).

Figure 3

• SUS Related Closing Survey- Open Ended Commentary - The SUS related closing survey had a 2 open ended questions to explore additional potential reactions. The entered feedback mirrored the general feedback expressed during the “Think-Alouds” through the task protocol during the observed usability session. Common themes expressed were:

“Not intuitive, needs improvement” “Initially easy turned to frustrating”

“The site was very layered and seemed overwhelming”

One interpretation could be that the feedback seems to mirror the reported 30.83 average SUS usability perception score. That survey feedback commentary can be found in the raw data google folder (SUS Results).

Discussion and Recommendations While the small sampling situation limits recommendations that are strongly data driven, there was enough indication suggesting usability concerns could be further explored. The two primary recommendation areas relate to improved Information Architecture (IA) layout/interaction and further usability testing.

Consider Information Architecture reorganization. The Co-Teach site is a sizeable web effort overflowing with content having 3 primary elements layered within its information architecture. Those elements are:

● HSE Personnel experiencing Mizzou K-12 Online as students would. ○ Gaining user perspective and empathy.

https://drive.google.com/open?id=0Bx5Cu5lZO-Eya3dSYzB3dW9lQms


05/09/2016

28

● HSE Personnel interaction amongst themselves and with Mizzou K-12 Online personnel.

○ Team planning, management and feedback. ● HSE and Mizzou K-12 Online personnel interacting directly with Brazilian

students. ○ Teaching and learning.

Having those 3 elements intermingled creates a layered and complex information architecture promoting an initial sense of overwhelming cognitive overload. By reorganizing the IA showing clear and definable areas, each element would seem more compact with a likely improvement to the expressed “over whelming” commentary from users. Users respond more favorably to a perception of simplicity having a feel of intuitive interaction. Determining the IA layout should be informed by usability testing so that design choices are data driven.

Consider additional usability testing the Co-Teach Information Architecture. Consider creating a Mizzou K-12 Online testing site. Any IA design changes should be informed through usability testing. This would allow for a more tested approach to substantially address discovered concerns and issues. There are two initial testing groups to consider forming, those who have little to no experience interacting on the site and those who are much more experienced interacting on the site. Each of those groups would have different experience perspectives and could offer valuable feedback. From that feedback, a secondary testing platform could be developed having all of the interactivity widgets and features paired with a reorganized IA. Further usability testing on the secondary platform could inform further improvements to the point of eventual full adoption and utilization.

Consider providing an anonymous feedback mechanism. Attributable feedback can have comfort limitations and prevent sending what could be helpful suggestions. By creating an anonymous feedback mechanism users could feel more comfortable submitting potentially helpful thoughts towards improvement without concerns of negative attribution. Options include anonymous email structures and form submissions among others.


05/09/2016

29

Consider installation of an Auto-Suggest Search Box aide. With the current site or a potentially adjusted one, content will still be an issue to navigate. Having an auto-suggest search box could improve site usability as a more interactive site search aide. It would help by facilitating novel query reformulations, by providing an open-ended list of phrases and terms while also encouraging exploratory searches. This more interactive option could be an important tool towards site simplification by users potentially relying on it more than manually sifting through the site for topic discovery. It is recommended the search box has space for 27 characters which accommodates 90% of user searches. Canvas does offer this function as an installed widget. Consider installing Google Analytics This can be a robust data driven tool used to better inform background usability performance and could significantly influence design decisions towards an improved user experience. Analytic data examples include:

● Where do Users both enter and leave your site? ● Analyze User path flows through site. ● See where Users encounter path flow error points causing problems. ● Determine behavioral scenario successes or failures in real or recorded time. ● Discover page popularity and exit frequency. ● Determine User timings which includes:

○ Speed with which Users accomplished goals. ○ How long do specific actions take? ○ Where Users are forced to wait.

Usability Recommendation Summary Usability recommendations represent both short and long term plans of action. Short term examples include installation of the auto-suggest search box, implementing an anonymous feedback mechanism and installing/configuring Google Analytics. Longer term examples relate to information architecture, its usability testing and the testing platform associated with it. While the longer term recommendations represent measured effort, the benefits far outweigh that effort by delivery of a data driven informed design that will positively resonate with users. That importance is further magnified when considering the Mizzou K-12 Online goal of adding a significant student

https://canvas.instructure.com/styleguidehttps://canvas.instructure.com/styleguidehttps://canvas.instructure.com/styleguide


05/09/2016

30

base from other areas of the world. A soundly designed site infrastructure model will better support that expanded user base as well as those already onboarded.

3. Does the Co-Teach course instructional design support the stated course objectives? The goals of the Co-Teach course are to help HSE Teachers/Coordinators learn both how to work through student assignments by using Canvas tools and an avenue to experience Canvas as their students do. It is also intended to provide discussion and communication tools for HSE Teachers/Coordinators to collaborate with Mizzou K-12 Online Lead Teachers on all aspects of the education effort.

Data A survey was given to all Co-Teach course users. 12 users participated in the survey, which featured 9 items to rate and two open response questions. Data related to the design of the course site was also collected in an SUS survey given to observation participants. A summary of results from the item rating is below:

Level of Agreement or Disagreement (where 1 = strongly disagree and 5 = strongly agree)

Question Min

Value Max

Value Variance Total

Responses Mean

1 I understand the purpose of the Canvas course structure

2 5 1.56 10 4

2 I can find what I need in the course interface 2 5 1.21 10 3.9 3 I understand why modules may be locked 1 5 1.86 9 4.11 4 I support locking the modules 1 5 1.78 10 4

5 I understand the Co-Teach Model as described in the Co-Teaching with Technology course

3 5 0.61 9 4.11

6 I support the Co-Teach Model as presented in the Co-Teaching with Technology course

2 5 1.11 9 4.11

7 I understand how to find help for my technical questions

2 5 1.25 9 4

8 The Canvas quizzes are helpful in testing my knowledge

3 5 0.61 9 4.11

9 I feel the plagiarism content is in the correct place.

2 5 1.25 9 3.67

See Appendix 3 for access to the raw data.


05/09/2016

31

In general, responses from course participants regarding placement, tool introduction, and design fell between “Agree” (a rating of 4) and “Strongly Agree” (a rating of 5). Two notable exceptions to this are the placement of the plagiarism content and the ability to find needed content (3.67 and 3.9, respectively). The two questions regarding the use and presentation of the course modules had the highest variances in responses, while the lowest variance was around the use of quizzes within Canvas, and the clarity of the course description and goals. Survey participants were also asked open ended questions related to the most useful and least useful features of the course site. Far less users completed this section (only 3 of the 12). However, those who did listed created tickets to solve errors, handling student work, and grading as most helpful. Least helpful features had minimal responses related to the site but did have one user specifically mention discussion boards. SUS survey data (see Figure 2 on page 26) also asked users general questions about their impression of the site. Discussion Course and SUS Survey data both points to some issues in finding course content easily, but course participants seems to understand the nature and goals of the course, as well as the importance of the “student experience” in learning to use and take quizzes, and work through modules sequentially, even if the content itself is not well organized. Specifically, the topic of plagiarism had a notably lower score than any other. Although interviews with course teachers indicated that this content is important to promote, it could possibly be re located, or tied in with other information to become easier to view and understand. Recommendations Activities that better tie in the use of the discussion boards to highlight the importance of communicating with Mizzou K-12 staff might assist course participants in learning to see its value. Other “student” oriented activities presented this way, such as module progression and quizzes, were met favorably. Additionally, while the model and course content itself seems useful for course participants, both surveys indicated that the placement of some materials on the site were ineffective and difficult to use. To address this, it’s possible that further simplification and streamlining of materials, in addition to ongoing communication, would lessen the onboarding load for HSE staff.


05/09/2016

32

4. What are the HSE Teacher/Coordinator reactions to the Mizzou K-12 Online Canvas LMS instructional design? An interactive Learning Management System, Canvas offers Mizzou K-12 the capacity to connect program coordinators in Missouri, United States with teachers in Brazil. The Canvas LMS has been in use for several months, while its efficiency is being tested and discovered.

Data Coordinators and Lead Teacher interviews from the Mizzou K-12 program were the main source of data collection pertaining to these questions. Those coordinators and lead teachers who responded were open with answering this question. Questions were performed face-to-face, via phone interviews, or through online video conference services, such as Skype, Google Hangouts or Zoom. Taken directly from the transcripts, sample comments included: “Canvas offers a lot of options and capabilities and we are figuring out the most efficient and productive way to streamline the communication. It is successful but we need to work out a few kinks.” “I think in the last document in the first module it's called "Why are my modules locked?” It could be that's not enough information but I think it's an important sweet explanation that says your modules are locked because you're a student, this is what your students are experiencing. This should help you understand what your students are seeing and it's not the same thing you see when you get in the courses.” “They (the teachers in Brazil) ask a lot of questions just clarifying things. For instance, process questions like submitting assignments. Some consistency issues like where the material is located. Everything went together so quickly that sometimes things didn't go where they were supposed to and that needed to be corrected.” Discussion After conducting interviews with University of Missouri Lead Teacher and Coordinators, it was concluded that they are satisfied with the Mizzou K-12 Online Canvas LMS. The LMS offers flexibility and capabilities that allow for a form of communication between the Coordinators in Missouri and the HSE teachers in Brazil. The current Canvas LMS, compared to the previous LMS, which was Moodle, is easier to navigate and more comfortable to work with. Some areas still need improvement and content should be organized more appropriately. Out of the comments from HSE teachers in Brazil, there were complaints about the structure of the content. As a new user of the Canvas LMS, navigation for them it should be intuitive and easy. This would be the same LMS that students will be using in Brazil.

https://www.skype.com/en/https://hangouts.google.com/https://zoom.us/meeting


05/09/2016

33

Recommendation After the evaluation team conducted the interviews and interacted with the Mizzou K-12 Co-Teach model, it was noticed that the layout and structure of the Co-Teach model needed improvement. One recommendation is to include the plagiarism information towards the end on the Co-Teach Model module, as part of the disclosure, so as to not make the instructor feel bombarded from the beginning with ethical and technical information. Also, modules should be open from the beginning, rather than locked, to be opened as the instructor progresses through the model. Those providing support for the Co-Teach model on Canvas should be aware that they are dealing with professional; content should be open and available to them.

5. What project management practices are being used to administer the Co-Teach course? At the start of Mizzou K-12 Online, awareness of the need for communication between team leaders was acknowledged but a centralized communication plan and project management strategy, which linked all team members was not seen generally see as critical to facilitating the on-going program.

Data Formative questions about project management practices were addressed in three interviews with members of the management team (two of the interviews were with members of leadership and one of the interviews was with a member of leadership within the academic/teaching team of Mizzou K-12 Online). Two members of the evaluation team asked these three team members specifically about how communication is structured and the pathways for communication. The interviews were then reviewed by another member of the evaluation team and then passed on to Dr. Aimee Klimczak, who is PMP certified. Dr. Klimczak then reviewed the interviews and conclusions, and gave her feedback and suggestions to the evaluation team. Several of the project management practices were described members of the project as follows:

• Mizzou K-12 Online has no visual representation of team relationships, nor an organizational chart to disperse information about term members, relationships or roles.

• Two of the major teams on the project, instructional design and academic/teaching

operate independently of each most of the time. The two group’s leadership interacts to address administrative duties but generally team members do not meet or exchange information. Often the groups are unaware of what each other are doing.

• Communication is informal and often occurs by proximity, phone, Skype or email. There

is no documentation of communication or project management software in use currently.

• It was stated that issues are addressed as they ‘pop up’.


05/09/2016

34

• It was stated the leadership is relying on ‘active listening’.

• A general newsletter is issued about once a month

• The academic team uses top down communication to disseminate decisions and

information.

• Differences are discussed over Skype, in order to facilitate communication that may be more sensitive.

• There are regular administrative meetings among team leaders and they sometimes work

on Google Docs together, as do the Lead Teachers. Currently there is no direct sharing of that information

• The communication priority is the program mission and the vision, and this seems to be

clear to all on the project but operational priorities and dissemination of information is less clear to team members.

• Lead teachers meet regularly and do communicate in part through using Google Docs,

but do interface at this depth with other teams. Discussion The Mizzou K-12 Online have had some difficulties with the flow of information on the project. The team has philosophical cohesion and a positive outlook towards the project, but formal project management strategies are often absent and there is a lack of information sharing and information dispersal to members of the team. Several of the team members appear to feel this is a product of the online and remote locations of many of the team members, and most seem unaware of the effectiveness of many of the modern project management tools available. Many of the tools now available allow teams to create real spaces for both collaboration and organization, as well as make members feel as though they are part of a community they can reach out to at any time and receive helpful information. Recommendation The leadership team should actively consult a certified project manager for recommendations which would strengthen communication practices, streamline communication and increase dialog and communication between team members. With so many available technology tools, team members need to be made aware of the availability and how easy they are to use, as well as how a relatively short list of project management strategies could make a positive change to the overall project communication.


05/09/2016

35

6. What project management practices might improve project implementation? As Mizzou K-12 Online moves past the launch of its courses more attention can be directed at implementing project management practices to make communication easier among all members of the team, especially between the leadership members, and more critically between leadership and members of their teams.

Data Formative questions about project management practices were addressed in three interviews with members of the management team (two of the interviews were with members of leadership and one of the interviews was with a member of leadership within the academic/teaching team of Mizzou K-12 Online). Two members of the evaluation team asked these three team members specifically about how communication is structured and the pathways for communication. The interviews were then reviewed by another member of the evaluation team and then passed on to Dr. Aimee Klimczak, who is PMP certified. Dr. Klimczak then reviewed the interviews and conclusions, and gave her feedback and suggestions to the evaluation team. The suggestions for strengthening the project management of the program:

• Create a project organization chart that identifies project members and roles - The group leaders in the Mizzou K-12 Online school should create an organizational chart that represents team members, identifies the distinct groups (HSE, Mizzou K-12 Staff: Business, Academic, Instructional Design and Technology) and their relationships to each other so all members are aware of who the team members are and what their roles are within the different groups. This final document should be easily accessible in a central location, and will enable team members to reach out to groups and individual team members.

• Develop clear processes and procedures for the communication of information - The team

should develop a process for general communication within the Mizzou K-12 Online team, where leaders of each group: business, academic and instructional design and technology, regularly post information to a central location so all groups became aware of broader initiatives and activities. This would help team members gain a sense of the bigger picture, as well as understand the broad strokes of the initiatives, activities and problems within each division of Mizzou K-12 Online. This would also allow both team member concerns and good ideas to circulate more rapidly between the different groups and prevent the ‘silo’ effect the group often experiences now.

Without a central project manager, the team must incorporate project management tools that will enable them to create a centralized communication ‘space’ on the web


05/09/2016

36

that can be viewed by all stakeholders. There are several cloud based organizational tools that will allow a team working on projects to move forward on the ‘same page’ and share information. Cloud-based solutions like Slack, Yammer or Hipchat, for example, allow teams to not only converse together, maintain threaded discussions by topic and instantly respond to each other, but also allow for the conversation to carry over seamlessly into the mobile world. In addition, there are programs like Trello, Asana and Basecamp, among the many cloud-based programs that allow for shared documents, threaded discussions by topic with instant notification to all team members, shared calendars with automatic email reminders and mobile access to all posted information, documents, calendars and group conversations. These cloud-based programs eliminate the long email chain, phone conversations and allow discussions that are more like a conversation but with multiple parties, as well as allow people to view large amounts of information organized by topics. This can save individuals in the group many hours formerly spent sending, searching and forwarding email. These programs also allow for quick grouping of collaborators across different program areas in the project, and sets up opportunities for people to interact around a topic who work in different areas. Additionally, since the Mizzou K-12 Online team does not have a central coordinator or project manager, as each smaller scale internal project for Mizzou K-12 Online starts, a point person or project lead should be assigned to direct communication among all the stakeholders on that particular activity and help maintain progress and interaction for that project using the designated project management tools. This project point person can then report back to the leadership team and disperse information from the smaller groups work to the larger community of Mizzou K-12 Online.

• Develop a process to identify, escalate, and resolve issues that arise during the project - As

project management tools provide more opportunities for communication among all stakeholders, the group as a whole will have an increased chance to identify and address issues as they arise. It would be wise for the team to develop a process to address issues in a timely manner. With faster resolution to issues, team members will feel more productive and less like time has been wasted on impractical ideas. Designated leaders from each area (academic, business and instructional design and technology) should all be empowered to call full leadership synchronous meetings where leadership meets via a web conferencing software like WebEX or Zoom. And all team members should be encouraged to use discussion tools to initiate conversations about problems, ideas or procedures in a designated forum so team leaders become aware of issues as they arise and team members have a space to provide feedback to leadership and each other.

https://slack.com/https://www.yammer.com/https://hipchat.com/https://trello.com/https://asana.com/https://basecamp.com/https://signup.webex.com/webexmeetings/US/sem_acquisition.html?DG=01-04-07-CA-12-01-02-06&TrackID=1031998&country=CAhttps://zoom.us/meeting


05/09/2016

37

• Develop a process to review current procedures and existing strategies - With

evaluation plan and report mizzou k-12 co-teach course · 2019. 11. 26. · evaluation plan and...

Documents