Design DocumentChristine Crawford

IDT 520/525Fall 2006/Spring 2007

Richard Van Eck

The Goal

2

The Rationale

Technology and the Internet have grown at incredibly rapid rates, and as a result, distance education is growing rapidly as well. In 2000-2001, “56 percent (2,320) of all 2-year and 4-year Title IV-eligible, degree-granting institutions offered distance education courses for any level or audience.” (US Dept of Education, 2003, ¶2). This statistic does not define the quality of the education provided online, however, and too often courses are simply shifted from face-to-face PowerPoint presentations to hyperlinked PowerPoint files. Technology savvy students expect to be able to participate in their courses from the comforts of home. Educational institutions, particularly institutions of higher education, struggle to acquire resources to purchase technology and stay competitive in the market. Faculty members are required to produce high quality instructional units that can be taught online in varying degrees and methods. The time and effort it takes for this to occur is often overwhelming to instructors.

Historically, course management systems (CMS), such as Blackboard, have been used to facilitate distance education. Course management systems have become tools for instructors to transition from in-person to online classes. But there are many disadvantages to this approach which too frequently result in poorly designed instruction. Studies have found some of the disadvantages to using course management systems to be browser incompatibility, lack of integration with other services, low usability, and simplistic use of the tools (Jafari, McGee and Carmean, 2006). Jafari, et al. (2006) interviewed students, faculty and administrators who cited the following as problems with course management systems; CMS tools that are under used by faculty, a lack of integrated services, and “dumb” systems that cannot perform a task as simple as sending a reminder. Smart classroom systems are a viable alternative to or supplement for posted PowerPoint files and/or lecture notes and course management systems because they have the potential to provide both synchronous and asynchronous contact with students at a distance.

With the approval of a completely online master’s program in Instructional Design and Technology, the College of Education and Human Development purchased a state-of-the-art smart classroom system to provide an in-house means for distance education. The smart classroom was installed in room 202 of the Education building; it is a complex integration of several types of technology. The room contains one high definition camera placed at the far end, eight speakers in the ceiling and four high-quality microphones attached to the walls. An LCD projector and screen is also available to use. On the podium there is an LCD touch panel unit which controls the functions of all of the features available as well as the monitor to the Gateway desktop computer. An instructor microphone is kept in the drawer of the main power console. The console also houses the main power source, the tower for the desktop computer, a Samsung DVD/VHS player, JVC mini DVD/DVD recorder. The unit also has an Elmo digital presenter that can be used for overhead projection and is processed through the unit for distance learners. Recordings can be saved to the IDT server, streamed in real-time to the audience or saved on DVD for later distribution.

This equipment has the ability to record the instructor lecturing in front of the class. Faculty can learn to operate the equipment so that any use of multimedia will be recorded in a viewable way. An advantage to this method of delivery is that faculty accustomed to lecturing in a face-to-face environment will not have to drastically change their instructional methods to adjust to using this equipment. One way to do this is by recording using Adobe Connect Pro (ACP). ACP will not be taught extensively in the training but we will review how to capture video and record a session using ACP. Faculty can also record to mini-DV tapes and/or DVDs which can be sent to students

3

for later viewing. If recorded in ACP, a link can be sent to the student for viewing. Additionally there is the option to stream the session in real time to the web by contacting the IDT department.

Distance education is a very real concern at UND. The president of the University, Dr. Charles Kupchella, states the following in his Strategic Plan II “Building on Excellence:”

Goal Five: Enhance the University’s position as a leader in the creation and application of information technologies to enrich and extend learning and research.

Action Strategy 8: Increase the number of programs offered at a distance by 10 percent until an optimal level is reached. (UND, 2007)

Although distance education is a growing method of course delivery at UND and in other universities, relatively few faculty members in the College have conducted a course using videoconferencing technology. Even those that have used this type of technology will need to be trained on this specific system.

The Problem

The College of Education and Human Development at UND recently purchased and installed a state-of-the-art smart classroom system in room 202 of the Education Building. The purchase was made in order to provide faculty members in the College with a convenient means to conduct distance education courses. Using sophisticated equipment to conduct classes in person, much less at a distance, can be intimidating and in the training environment must be handled delicately with university faculty members. However, the advantages outweigh the disadvantages. In addition to the easy transition of lecture style classes, sessions can be recorded using DVD, mini-DV tape and streamed out one way for students to watch. More ambitious faculty members can use the smart classroom to conduct synchronous class sessions by utilizing the audio conferencing feature for real time sessions and by using Adobe Connect Pro to bring in real time audio and video as well.

The Goal Faculty in the College of Education and Human Services will be able operate the smart classroom equipment to record to DVD or mini-DV tape, stream, conduct audio conferences and conduct and record a videoconference using Breeze.

Works CitedJafari, A., McGee, P., & Carmean, C. (2005). Managing Courses, Defining Learning, What Faculty, Students and Administrators Want. Educause Review July/August 50-70.

University of North Dakota, Strategic Plan II. (2007). Retrieved on June 17, 2007 from http://www.und.edu/stratplan/goals_action_strategies.html.

U.S. Department of Education, National Center for Education Statistics. (2003). Distance Education at Degree-Granting Postsecondary Institutions: 2000-2001 (NCES 2003-017) Retrieved on December 8, 2006 from http://nces.ed.gov/fastfacts/display.asp?id=80.

4

Population and Environmental Analysis

5

The Environment

The University of North Dakota is proactive in providing distance education to its students and support to its faculty participating in distance education. The Division of Continuing Education (http://www.conted.und.edu/), which houses all online courses and programs on campus, offers undergraduate courses, certificate programs and distance degree programs. Continuing Ed also offers faculty support in converting and creating distance courses by providing “The Online Team,” a group of individuals trained in graphic and instructional design. In addition, The Center for Instructional & Learning Technologies (CILT) (http://www.cilt.und.nodak.edu/) offers support to faculty integrating technology into their instruction.

Training will take place on the University of North Dakota campus in the College of Education and Human Development (EHD). The EHD is aligned with the rest of the University in its practical approach to distance ed and several of the EHD programs offer online courses. The departments of Social Work and Instructional Design and Technology, Educational Leadership and Special Education offer masters degrees online. Online doctoral programs can also be earned in Educational Leadership and Teaching and Learning in Higher Education. In addition, thousands of dollars were recently invested into the purchase and installation of smart classroom equipment to be utilized by faculty in this College which was designed and installed by the director of CILT.

Despite this culture of support on campus, a broad spectrum of opinion exists. Some faculty members utilize the distance learning options available to them on a regular basis. Some agree that distance education is a valid concept but participate rarely or not at all. Still others are neutral to the idea or even actively dislike the notion of distance education. Therefore, even ample support does not rule out the possibility that there are faculty members participating in distance education only because they are compelled to do so by the politics of the university.

The smart classroom training will be held in room 202 of the Education Building. The room contains a state-of-the-art smart classroom system which is built upon Windows desktop computer and Microsoft Office products as well as an LCD projector. A high definition camera is mounted on the wall near the ceiling at the far end of the room along with several high quality microphones attached to the walls and eight speakers in the ceiling. The microphones are used with the audio conferencing system which allows verbal communication between the room and distance students, who are at their own computers with broadband and dial-up Internet service.

The system includes an LCD touch pad unit which controls the functions of all of the features available. The unit also has a DVR and a mini-DV tape recorder. There is an Elmo digital presenter that can be used for overhead projection and is processed through the unit for distance learners. Recordings can be saved to the IDT server, streamed in real-time to the audience or saved on DVD or mini-DV tape for later distribution. The room itself is about 20’x 30’ and there are acoustic problems stemming from the concrete walls. Carpets for the walls and floors will be at some point in the future to help dampen the echo.

6

The Learners

The learners will be faculty members of the College of Education and Human Development. The EHD houses the departments of Counseling, Educational Foundations and Research, Educational Leadership, Physical Education and Exercise Science, Social Work and Teaching and Learning. All faculty members will have access to this smart classroom and must be trained on its use.

The skill level of the learners will vary from novice to expert. Those who may have used smart classroom equipment in the past will have some prior knowledge on how to work the system in room 202 and will need only a brief orientation to the system. Others may have minimal experience with smart classrooms and will need more in-depth individual training. Learners may experience some anxiety related to dealing with new technology and may also experience cognitive overload if instruction is not designed with these learners in mind. If participating faculty members are part of the group who are reluctant to participate in online learning, there may also be attitude or motivation issues. Most likely these learners will be accustomed to learning in both print and computer media.

Prerequisite Skills

Learners must have intermediate knowledge of how to use desktop and laptop computers and must know enough about a computer to be able to use whatever programs are needed to teach their courses, such as Word and PowerPoint. Students should also have experience using Adobe Connect Pro or have plans to take a class in Adobe Connect Pro. Prior smart classroom use would be ideal but not necessary. Students should have prior experience with the following items:

desktop computer and USB cords flash drive, uploading files from a flash drive and opening files from a flash drive microphone, volume control and touch panel laptop computers and VGA cords video cameras DVDs, DVD recording

Implications for Design

Training will be designed for inexperienced learners with the option to accelerate wherever possible for those who have more experience with this type of equipment. For example, each student will be asked to complete an Experience Assessment (see page 25) which will indicate their level of prior experience with computers and smart classrooms. More experienced students will be able to move through the information at a quicker pace.

Smart classroom training will be provided by the IDT department and will not be mandatory. The training will be advertised to the College of Education and Human Services and faculty who are interested will contact us. Each student’s motivation will vary depending on his or her attitude toward distance education and the reasons for taking the training. Attitude toward distance education could also affect attitude toward training. Someone who is participating in distance ed under compulsion may be difficult to approach and not very willing to learn. On the

7

other hand, that same person may be more inclined to learn so that he or she can be successful at something foreign and possibly intimidating. Either way, this training is an opportunity to introduce reluctant faculty members to technology that will help them produce high quality distance education. Design can help address these issues by including information about the advantages of distance education, in general and the smart classroom method, in specific. These topics will be covered in Module II.

For learners new to this equipment, cognitive overload is a design concern. The nature of the equipment will make it necessary to bring the students into the room and demonstrate its use step-by-step. If students try to take notes simultaneously, their attention will be diverted; therefore notes on the lesson in the form of a job aid will be provided prior to training for students to review with space for additional notes as the lesson progresses. During each section students will have an opportunity to interact with the equipment and functions just discussed so that through guided practice students will have more than one way to assimilate the information. Faculty of the EHD will benefit from this training and will be well equipped to use the new smart classroom system.

8

Instructional Sequence

9

Instructional SequenceTraining will be face to face. Notes on the lesson in the form of a job aid will be provided prior to training for students to review with space for additional notes as the lesson progresses.

Introduction

Training Objectives

Module I - Regular use of the smart classroomPart 1 – Getting Started

Instruction Begins Experience Assessment

Part 2 – Tour of the system and powering up Diagram of ED 202 Tour of the room Powering Up

Part 3 – Projector, camera and instructor mic Projector Camera Instructor microphone

Part 4 – Using computers with the system Desktop Machine

Module II - Passive (one way) Broadcasting The Recording Process

Part 5 – StreamingPart 6 – Recording to mini-DV tapePart 7 – Recording to DVD

Finalizing a DVD

Module III - Active (two way) Broadcasting.............................................Part 8 – Audio ConferencingPart 9 – Adobe Connect Pro

Fire wire Logging in Start the camera Start the audio conference Record in Adobe Connect Pro Share your screen Share a document

Module IV – What did we learn?Part 10 – Recap

Assessment Performance Checklist

10

Instructional Analysis – Description of Instruction

(see attached Inspiration document for curriculum map)

11

Description of Instruction

The purpose of this lesson is to teach faculty members of the College of Education and Human Services how to use the smart classroom system in room 202 for distance education. Since the equipment has a broad spectrum of capabilities and the learners come from a wide range of experiences, the lessons will be divided into four modules. The instruction will be delivered face-to-face. Due to time constraints, neither the ELMO document camera nor the laptop plug-ins will be addressed in this instructional unit. Prior to the instruction, learners will be asked to take an Experience Assessment. Student manuals will be provided at the beginning of the training session for learners to follow along and take notes as the lesson progresses.

Module I will give learners an overview of regular use of the smart classroom. Module II discuss distance education and how this system can enhance online class sessions. It will also demonstrate how to record and stream a session. When the training session is done the learners will be able to generate a class session using the components of their choosing and record or stream that session. Module III will demonstrate using the audio conferencing feature of the room. It will also demonstrate how to use Adobe Connect Pro to record a class session. In Module IV we will briefly review what we have learned and proceed to the assessment.

Module I will orient the learners to the room and to the equipment. Each piece of hardware will be discussed, including the camera, the wall-mounted microphones, the main power console, the desktop computer, and the cords for using a laptop with the system. Also, the instructor microphone, the LCD projector with screen, the DVD/VCR player, mini-DV tape and DVD recorders and the LCD touch panel. Learners will be shown how to turn on the touch panel and the instructor will give a brief tour of the touch panel to orient the learner to the icons being used for each component. The instructor will demonstrate all components by bringing the various items necessary, such as a VHS tape to demonstrate the VCR player. The instructor will demonstrate how to use the wireless microphone, how to turn on the desktop computer and upload external materials from a flash drive, how to plug in a laptop, how to operate the LCD projector and screen.

In Module II, the process for streaming lectures will be summarized. The instructor will demonstrate how to record a lecture using the mini-DV tape and the DVD recorders.

Students must learn Adobe Connect Pro on their own but Module III will cover how to plug fire wire into the system, how to log in to Adobe Connect Pro using the IDT login and password, how to bring the video feed into the program, how to start the audio conference , how to record and how to end a meeting.

A secondary objective of this lesson is to influence faculty member’s attitudes toward distance education and using the smart classroom to enhance the experience. The need for and the advantages to distance education and how this smart classroom equipment can help meet the need will be woven into the instruction in various places throughout.

12

Objectives Matrix

13

Goal: Demonstrate use of smart classroom equipment designed for distance education

All instruction and assessment will occur face to face. A checklist will be used to assess performance (attached).Skill Objective Sample Assessment1.1.1. Demonstrate how to power up all components

Given a verbal prompt, the student will be able to demonstrate how to power up all components by touching the touch panel every time.

Please power up all components using the touch panel

_____ Main power_____ Gateway power _____ Touch panel

1.1.1.1 Identify main power switch

Given a matching exercise, the student will be able to identify the main power switch by choosing it with 100% accuracy.

Matching components quiz

1.1.1.2 Identify Gateway power switch

Given a matching exercise, the student will be able to identify the Gateway power switch by choosing it with 100% accuracy.

Matching components quiz

1.1.1.3 Identify icons for each component on touch panel

Given a matching exercise, the student will be able to identify each icon on the touch panel by choosing the associated photo for each with 80% accuracy.

Matching touch panel quiz

Matching components quiz

1.1.1.4 Identify each equipment component

Given a matching exercise, the student will be able to identify each component of the smart classroom by choosing the component with 80% accuracy.

Fill in the blank room diagram

.

1.1.2. Demonstrate use of instructor microphone

Given the instructor microphone, the student will be able to demonstrate how to power up, position and control the volume of the instructor microphone by performing each activity with 100% accuracy.

2. Show me how to correctly use the instructor microphone.

_____ Show me how to turn on the instructor microphone_____ Show me how to position microphone on lapel 8” or more from mouth_____ Show me how to use microphone volume control on touch panel

1.1.2.1 Demonstrate use of microphone volume control on touch panel

Given the instructor microphone, the student will be able to demonstrate how to control the volume by turning it up and down while it is on with 90% accuracy.

1.1.2.1 Show me how to correctly use the instructor microphone.

_____ Show me how to use the microphone volume control on touch panel.

14

1.1.2.1.1 Demonstrate optimal position of microphone on lapel

Given the instructor microphone, the student will be able to demonstrate how to position the microphone by placing it on his or her lapel in an optimal position with 100% accuracy.

1.1.2.1.1. Show me how to correctly use the instructor microphone.

_____ Show me how to position microphone on lapel 8” or more from mouth

1.1.2.1.1.1 Demonstrate how to power up instructor microphone

Given the instructor microphone, the student will be able to demonstrate how to power up the microphone by turning it on with 100% accuracy.

1.1.2.1.1.1. Show me how to correctly use the instructor microphone.

_____ Student turns on the instructor microphone

1.1.3. Demonstrate use of camera

Given a verbal cue, the student will be able to demonstrate how to use the camera by manipulating the touch panel buttons with 100% accuracy.

_____ Show me how to turn the camera on using the touch panel_____ Show me how to use the touch panel to control the camera using pan, tilt and zoom.

1.1.3.1 Demonstrate how to turn camera on using touch panel

Given a verbal cue, the student will be able to demonstrate how to turn the camera on by using the touch panel buttons with 100% accuracy.

_____ Show me how to turn the camera on using the touch panel

1.1.3.1.1. Demonstrate pan, tilt and zoom functions.

Given the touch panel, the student will be able to demonstrate how to use the camera by using the pan, tilt and zoom controls with 100% accuracy.

_____ Show me how to use the touch panel to control the camera using pan, tilt and zoom.

1.1.3.1.1.1. Identify pan, tilt and zoom features.

Given the touch panel, the student will be able to point to the pan, tilt and zoom controls with 100% accuracy.

_____ Please point to the pan, tilt and zoom on the touch panel.

1.1.4 Demonstrate use of desktop computer

Given the desktop computer, the student will be able to demonstrate how to turn on the desktop using the touch panel by performing the task with 100% accuracy.

_____1.1.4. Please show me how to turn on the desktop computer using the touch panel.

(Objective 1.1.5, 1.1.5.1 and 1.1.5.1.1 should be deleted)

1.1.5 Demonstrate use of a laptop computer1.1.5.1 Demonstrate how to plug laptop into system cords1.1.5.1.1 Demonstrate touch panel controls for laptop

15

1.1.6 Demonstrate use of touch panel to control projector

Given a verbal request, the student will be able to demonstrate use of touch panel to control projector by turning it on and off using the touch panel with 80% accuracy.

_____ Student uses touch panel to power up projector_____ Student uses touch panel to power down projector

1.1.6.1Demonstrate use of touch panel to control LCD projector

Given the touch panel, the student will be able to demonstrate how to control the projector by turning it on and off using the touch panel with 80% accuracy.

_____ Student uses touch panel to power up projector_____ Student uses touch panel to power down projector

**This objective should be two sequential objectives for turning the projector on and off**

1.2 Demonstrate passive (one way) broadcasting

Given a verbal request the student will be able to demonstrate passive (one way) broadcasting by using all of the methods taught with 80% accuracy.

This objective is assessed in checklist items 1.2.1 – 1.2.3.

1.2.1 Summarize streaming process

Given a verbal request the student will be able to summarize the streaming process by stating the need to contact the IDT department with 100% accuracy.

Please tell me who you need to contact if you want to stream a class session_____ Student states the need to contact the IDT department.

1.2.2 Demonstrate how to record sessions using mini-DV tape recorder

Given a verbal request, the student will be able to demonstrate how to record sessions by recording on a mini-DV tape with 90% accuracy.

This objective is assess in checklist items 1.2.2.1 – 1.2.2.3 as follows:Please show me that you can record to a mini-DV tape…_____ Please point to the play,

record and stop buttons on the touch panel for the mini- DV tape recorder

_____ Student checks that DV is selected on the JVC device.

_____ Student checks that the JVC recorder is on Line 1

_____ Student uses the record icon on the touch panel to begin recording

_____ Student uses the stop icon on the touch panel to stop recording.

1.2.2.1 Demonstrate how to select Line 1 on the JVC recorder

Given a verbal request, the student will be able to demonstrate how to select Line 1 on the JVC recorder with 90% accuracy.

_____ Student checks that the JVC recorder is on Line 1

1.2.2.2 Demonstrate how to select DV on the JVC recorder

Given a verbal request, the student will be able to demonstrate how to select DV on the JVC recorder with 90% accuracy.

_____ Student selects DV on the JVC device.

1.2.2.3. Demonstrate use of touch panel to control recording

Given a verbal request, the student will be able to

_____ Student presses the record icon on the touch panel to begin

16

to mini-DV tape demonstrate how to control recording to mini-DV tape by pressing record and stop on the touch panel with 90% accuracy.

recording _____ Student presses the stop icon on the touch panel to stop recording

1.2.2.3.1 Identify the record and stop buttons on the touch panel for the mini- DV tape recorder

Given a verbal request, the student will be able to identify the record and stop buttons on the touch panel by pointing to them with 90% accuracy.

_____ Student points to the record icon on the touch panel

_____ Student points to the stop icon on the touch panel

1.2.3. Demonstrate how to record sessions using DVD recorder.

Given a verbal request, the student will be able to demonstrate how to record sessions by recording to the DVD recorder with 80% accuracy.

This objective is asses in checklist items 1.2.3.1 – 1.2.3.3.1 as follows:Please show me that you can record to a DVD_____ Please point to the play,

record and stop buttons for the JVC on touch panel

_____ Student checks that DVD is selected on the JVC device.

_____ Student checks that the JVC recorder is on Line 1

_____ Student uses the record icon on the touch panel to begin recording

_____ Student uses the stop icon on the touch panel to stop recording.

Please finalize a DVD _____ Student uses Manual for

finalizing the DVD 1.2.3.1 Demonstrate how to select Line 1 on the JVC recorder

Given a verbal request, the student will be able to demonstrate this step by selecting Line 1on the JVC recorder with 90% accuracy.

_____ Student selects Line 1 on the JVC recorder

1.2.3.2 Demonstrate how to select DVD on the JVC device

Given a verbal request, the student will be able to demonstrate this step by selecting DVD is on the JVC recorder with 90% accuracy.

_____ Student selects DVD is on the JVC recorder.

1.2.3.3 Demonstrate use of touch panel to control recording to DVD

Given a verbal request, the student will be able to demonstrate how use the touch panel to control recording to DVD by pressing record and stop with 90% accuracy

_____ Student uses the record icon on the touch panel to begin recording

_____ Student uses the stop icon on the touch panel to stop recording

1.2.3.3.1 Identify play, record and stop buttons of DVD recorder

Given a verbal request, the student will be able to identify the record and stop buttons on touch panel by pointing to them with 90% accuracy

_____ Student points to the record icon on the touch panel

_____ Student points to the stop icon on the touch panel

1.3.1 Demonstrate audio conferencing feature

Given a verbal prompt, the student will be able to demonstrate the audio conference feature by dialing a local number using the touch panel with 80% accuracy.

_____ Please dial a local number using the touch panel.

17

1.3.2 Demonstrate Breeze Given a verbal request, the student will be able to demonstrate how to use Breeze for web conferencing by performing the tasks involved with 80% accuracy.

_____ Please demonstrate plugging in the fire wire_____ Please log into Breeze using the IDT trainer login and password

_____ Please demonstrate selecting the camera_____ Please demonstrate starting and audio conference in Breeze_____ Please demonstrate entering an audio conference by calling the room_____ Please demonstrate how to record an Breeze session

1.3.2.1 Demonstrate how to plug in the fire wire

Given a verbal request, the student will be able to demonstrate how to connect the fire wire by plugging it into the correct port with 90% accuracy.

_____ Please demonstrate plugging in the fire wire

1.3.2.2 Demonstrate opening and logging in to Breeze

Given a verbal request, the student will be able to demonstrate logging into Breeze by entering the IDT trainer login and password with 90% accuracy.

_____ Please log into Breeze using the IDT trainer login and password

1.3.2.3 Demonstrate how to select the camera

Given a Breeze window and a student manual, the student will be able to demonstrate how to select the camera in Breeze by clicking on the icon with 90% accuracy.

_____ Please demonstrate selecting the camera.

1.3.2.4 Demonstrate how start an audio conference in Breeze

Given a Breeze window and a student manual, the student will be able to demonstrate how to start an audio conference in Breeze by clicking on the proper sequence of icons with 90% accuracy.

_____ Please demonstrate starting and audio conference in Breeze

1.3.2.5 Demonstrate how to enter the audio conference by calling the room

Given a phone number to call, the student will be able to demonstrate how enter the audio conference by calling the room with 90% accuracy.

_____ Please demonstrate entering an audio conference by calling the room

1.3.2.6 Demonstrate how to record a Breeze session

Given a Breeze window, the student will be able to demonstrate how to record a Breeze session by clicking on record meeting with 90% accuracy.

_____ Please demonstrate how to record an Breeze session

1.3.2.7 Demonstrate how to share a document in Breeze

Given a Breeze window, the student will be able to demonstrate how to share a document in Breeze by following the sequence of steps with 90% accuracy.

_____ Please demonstrate how to share a document in Breeze

18

Assessment

19

Smart Classroom TrainingExperience Assessment

Please answer these questions to the best of your ability by placing a “X” in the appropriate response area.

1. How many years have you been using computers?

<---------------------------------------------------------------------------------------------------------------->( ) ( ) ( ) ( ) ( )<1 1-3 4-6 7-9 >10

2. How often do you use computers to teach?

<---------------------------------------------------------------------------------------------------------------->( ) ( ) ( ) ( ) ( )

Almost Never Not Often Sometimes Often Always

3. How often do you use PowerPoint and an LCD projector to teach?

<---------------------------------------------------------------------------------------------------------------->( ) ( ) ( ) ( ) ( )

Almost Never Not Often Sometimes Often Always

4. Have you ever taken a class where a smart classroom was used by an instructor?

( ) ( )Yes No

5. Have you ever used a smart classroom yourself?

( ) ( )Yes No

6. How often do you use smart classrooms to teach?

<---------------------------------------------------------------------------------------------------------------->( ) ( ) ( ) ( ) ( )

Almost Never Not Often Sometimes Often Always

7. Please rate your ability to learn new technical skills (based on your success in the past).

<---------------------------------------------------------------------------------------------------------------->( ) ( ) ( ) ( ) ( )

Difficult Somewhat Can sometimes Somewhat EasyDifficult do it Easy

20

Please fill in the blanks with the name of each component.

21

Objective 1.1.1.3.1

A. Audio conferenceB. Media volumeC. DVD playerD. ComputerE. HelpF. Projector powerG. Cam controlH. Mic volumeI. Doc camJ. LaptopK. RecordL. VCR player

Matching touch panel quiz

Objective 1.1.1.3

Directions:Please enter the letter of each icon.

22

A. VGA cord for laptopsB. Touch panelC. SpeakersD. Main powerE. Mini-DV tape/DVD recorderF. USB extensionG. CameraH. VHS playerI. Audio cordJ. PC powerK. Boundary microphone

Matching components quiz

Objective 1.1.1.3.1.123

24

Performance Checklist

Check off each task as the student completes it. Please do not provide guidance.

Using your student manual as a resource, enter the room as though you were going to teach a class and answer the questions asked or perform the tasks that I request. I can’t give you any assistance as you complete this assessment. Use your student manual as a reference.

Powering up the system

Please turn on the system by turning on the:

_____ Main Power 1.1.1.2_____ Gateway power -insert Objective #_____ Touch panel 1.1.1.1

Projector, camera and instructor mic

Please show me how to use the projector by using the touch panel. 1.1.6

_____ Student uses touch panel to power up projector 1.1.6.1

_____ Show me how to turn the camera on using the touch panel 1.2.3

_____ Please point to pan, tilt and zoom on the touch panel. 1.2.3.1.1

_____ Show me how to use the touch panel to use pan, tilt and zoom. 1.2.3.1

Please show me how to correctly use the instructor microphone. 1.1.2

_____ Show me how to turn on the instructor microphone 1.1.2.1.1.1

_____ Show me how to position microphone on lapel about 8” from mouth 1.1.2.1.1

_____ Show me how to use the microphone volume control on touch panel 1.1.2.1

_____ Show me how to use the touch panel to power down projector 1.1.6.1

Computers

_____ Please show me how to select the desktop computer using the touch panel. 1.1.4

_____ Student uses the touch panel icon to interface with the desktop computer 1.1.4.1

Streaming

Please tell me who you need to contact if you want to stream a class session

_____ Student states the need to contact the IDT department 1.2, 1.2.1

Recording

Please show me that you can record to a mini-DV tape…25

_____ Please point to the play, record and stop buttons on the touch panel for the mini-

DV tape recorder 1.2, 1.2.2.3.1

_____ Student checks that DV is selected on the JVC device. 1.2.2.2

_____ Student checks that the JVC recorder is on Line 1….1.2.2.1

_____ Student uses the record icon on the touch panel to begin recording 1.2.2.3

_____ Student uses the stop icon on the touch panel to stop recording. 1.2.2.3

Please show me that you can record to a DVD…1.2, 1.2.2, 1.2.2.1

_____ Please point to the play, record and stop buttons for the JVC on touch panel

1.2.2.3.2

_____ Student checks that DVD is selected on the JVC device. 1.2.2.2

_____ Student checks that the JVC recorder is on Line 1….1.2.2.1

_____ Student uses the record icon on the touch panel to begin recording 1.2.2.3

_____ Student uses the stop icon on the touch panel to stop recording. 1.2.2.3

Please finalize a DVD 1.2.3.4

_____ Student uses Manual for finalizing the DVD 1.2.2.4.1

Audio conferencing

Please show me how to use the audio conferencing feature

_____ Please dial a local number using touch panel 1.3.1, 1.3

Breeze

_____ Please demonstrate plugging in the Fire Wire 1.3.2.1

_____ Please log into Breeze using the IDT trainer login and password 1.3.2.2

_____ Please demonstrate selecting the camera 1.3.2.3

_____ Please demonstrate starting an audio conference in Breeze 1.3.2.4

_____ Please demonstrate entering an audio conference by calling the room 1.3.2.5

_____ Please demonstrate how to record a Breeze session 1.3.2.6

_____ Please demonstrate how to share a document in Breeze 1.3.2.7

26

Assignment 7Instructional Sequence/Matrix & Strategies

27

*This section to be revised in future versions

Skill Part 1 Part 2 Part 3 Part 4 Part 5 Part 6 Part 71. Demonstrate how to power up all components1.1 Identify each equipment component1.1.1 State functions of components1.1.2 List names of all components1.2 Identify icons for each component on touch panel2. Demonstrate use of instructor microphone2.1 Demonstrate how to power up instructor microphone2.2 Demonstrate optimal position of microphone on lapel2.3 Demonstrate use of microphone volume control on touch panel3. Demonstrate use of desktop computer3.1 Demonstrate touch panel controls for desktop computer3.2 Demonstrate how to plug in a flash drive.3.3 Demonstrate how to upload files from a flash drive3.4 Demonstrate how to safely unplug a flash drive3.5 Demonstrate how safely unplug a flash drive3.5.1 State the procedure for safely removing a USB drive.4. Demonstrate use of a laptop computerTime 15 min 10 min 15 min 10 min 15 min 20 min 15 min

28

4.1 Demonstrate how to plug laptop into system cords4.1.1 Identify laptop VGA cord5. Demonstrate use of DVD/VCR player5.1 Demonstrate use of touch panel to control DVD/VCR player5.1.1 Identify touch panel buttons for insert, play, stop, eject, etc for DVD/VCR player6. Demonstrate use of LCD projector6.1 Demonstrate use of touch panel to control LCD projector6.2 Demonstrate switching between LCD & DVD7. Demonstrate use of camera7.1 Demonstrate use of touch panel to control camera using pan, tilt and zoom.

8. Summarize audio conferencing feature8.1 State process to dial out to conferencing site using touch panel macro9. Demonstrate recording lecture sessions9.1 Demonstrate recording on mini-DV tape or DVD9.1.1 Identify play, record and pause features of DVR on the touch panel9.1.2 Identify play, record and pause features of mini- DV tape recorderTime 15 min 10 min 15 min 10 min 15 min 20 min 15 min

29

9.2 Demonstrate opening and logging in to Breeze9.3 Demonstrate bringing video into Breeze software10. Summarize the procedure for streaming lecture sessions11. Choose smart classroom equipment for generating online classes11.1 State functions of system11.2 State advantages of using this equipment11.3 State advantages of distance education in higher ed12. List most common errors and their solutionsTime 15 min 10 min 15 min 10 min 15 min 20 min 15 min

30

Assignment 8Formative Evaluation Report

31

Formative Evaluation Report

32

Formative Evaluation Report

This report will describe and analyze the formative evaluation process for faculty training on the smart classroom in room 202 of the Education building on the UND campus. This report is divided into four main sections; Plan, Description, Outcomes and Interpretation. The Plan provides an outline of how the formative evaluation was designed and includes one-to-one and small group evaluations. The Description explains how the evaluations progressed. The Outcomes section contains the data obtained from both evaluations and the Interpretation analyzes this data.

I. The Plan

Control of Variables

AptitudeAptitude will be controlled by including learners of below average, average and above average aptitude. Aptitude will be measured using an Experience Assessment, a tool which measures experience with computers, smart classrooms and technical ability (see Appendix A). Questions on the instrument will be assigned equal weights and one through five points for each response.

The Experience Assessment contains a total of ten questions. Question 1 (1-5 pts) measures the learner’s experience with computers. Question 2 (1-5 pts) measures how frequently the learner uses computers to teach. This question was included because intermediate computer knowledge is a required prerequisite for the course. Question 3 (1-5 pts) measures how often the instructor uses PowerPoint and an LCD projector to teach classes. This question acknowledges a learner’s experience with low level technology in the classroom. Question 4 (1-5 pts) measures whether a learner has experienced the smart classroom setting as a student. Familiarity with the smart classroom environment as a student can provide an instructor valuable insight into good and bad instructional strategies with the equipment. Question 5 (1-5 pts) measures whether the student has used a smart classroom to teach. Previous experience with the smart classroom could impact the pace of the instruction for this student. Both questions 4 and 5 have yes/no responses. Positive responses will receive the full value of five points and negative answers will receive only one point. Question 6 (1-5 pts) measures frequency of experience with smart classrooms. This question was included because even if the learner’s experience is minimal, prior increases the probability that the learner will succeed and may impact the pace of the instruction as well. Question 7 measures the student’s ability to learn new technical skills. Question 8 (1-5 pts) measures whether the student has taken a course using distance education and question 9 (1-5 pts) measures whether the student has taught online using distance education. Both questions 8 and 9 have yes/no responses. Positive responses will receive the full value of five points and negative answers will receive only one point. Question 10 is an open ended question. Answers will be transcribed, reported and categorized. A chart will be created to illustrate the results. Possible total scores from nine to forty-five will indicate aptitude ranking. Participants will be faculty members at UND or people who have the potential to teach or who are teachers.

ProcessDue to the hands-on nature of the smart classroom, all instruction will be conducted face-to-face. The one-to-one sessions will be conducted by the same trainer and the group sessions will be conducted by a different facilitator. A checklist derived from the objectives will also help to control for process and insure that each item is addressed.

33

SupportAll training will be conducted in the smart classroom located in the Education building, room 202. Support will be provided by the Center for Instructional Learning Technology (CILT). The director of this department designed the room and CILT is responsible for its maintenance. A phone number is posted for anyone experiencing technical problems to call when in need of assistance.

One-to-OneThree subjects of varying ability, below average, average and above average, will be chosen to participate in this stage of the evaluation. Each training session will be conducted on a one-to-one basis with the trainer. The trainer will proceed to teach the learner about the smart classroom by demonstrating each aspect and asking the learner to demonstrate as well by using a guided practice approach. Performance will be measured using the Performance Checklist (Appendix B). The instructor will stop as questions or problems arise and get feedback from the learner as to how different aspects of the instruction could be improved. An Attitudinal Assessment (Appendix C) will be administered following the training. All responses to the Experience Assessment and the Attitudinal Assessment will be recorded using a matrix of questions and learners (Tables 1&2). Instruction will be revised based on learner feedback and the results of the post-training Attitudinal Assessment. One-to-one instruction will be completed within a two-week period.

Small GroupEight learners of below average, average and above average aptitude will participate in the training after the one-to-one evaluations are finished and revisions have been made. These eight learners may or may not be taught simultaneously. Training will be conducted in a similar manner as in the one-to-one phase; face-to-face instruction in the smart classroom. There will be no discussion of instruction or learner feedback during the instruction stage of the small group phase. The Performance Checklist (Appendix B) will be used during instruction to monitor learner progress. This stage takes place during the two weeks following one-to-one instruction.

EvaluationPerformance will be evaluated based on the stated objectives. The Performance Checklist and three paper-based quizzes will be used to evaluate each objective. Failure of the learner to complete any objective accurately will lead to analysis of the instruction and related instruments. A table will indicate the results for each objective and evaluation questions. Items that are below 100% accurate will be analyzed and modified as necessary.

All information will be analyzed for consistency and patterns by objective, assessment and learner both individually and across type. Matrix tables (Tables 3&4) containing objectives for each item will be used to perform item and performance analyses. Items within each objective will be evaluated, and any inconsistencies will be considered indications that modifications to the instruction or to the instruments may be needed. Problems with the instrument may include items that may not measure what they are supposed to measure according to the corresponding objective, the question may be unclear or too difficult. Success for each objective will be measured and compared to success rates of other objectives to examine overall consistency. Student performance across all checklist items and assessment items will be evaluated for consistency and patterns. Deficient items and unsuccessful objectives will be analyzed by

34

learner. All information will be analyzed across objectives, items and learners and any inconsistencies and patterns will be studied for indication of need for modification.

II. DescriptionOne-to-One

It is important to note that the smart classroom in Education Room 202 was not complete until a few weeks prior to the commencement of training. Much of the material was compiled and written prior to completion of the system in order to prepare as well as possible for the implementation phase of instruction. During much of the training many of the glitches were discovered and worked through which took a lot of time and made it difficult to get though the material. The goal was to have as much worked out as possible before the group training. The learning task analysis was changed a few times throughout the process which also effected the assessment and the results.

Since this is instructor-led training, a facilitator was chosen to conduct the training while the designer watched, took notes, and answered questions as needed. The facilitator was also an IDT graduate student who had a great deal of technology experience (especially audio/visual) and some experience with room 202 working on a different project. The facilitator was asked to read the green text from the instructor manual verbatim in order for the designer to evaluate the instruction that had been written.

The placement of practice items was something we struggled with. During the first one-to-one we turned everything off after the first demonstration and asked the participant to go through the process of turning it all back on. This was repeated only a few minutes later as an assessment item while the assessment checklist was utilized. This seemed to take up a great deal of time within too close of a timeframe. The format was changed so that the learner practiced many of the activities simultaneous to the instructor describing them. The student manual came to be seen as more of a tool than a reference and the learners were encouraged to listen to the facilitator’s instructions while watching the manual and practicing the activities when asked. This was effective in the one-to-one sessions but less so in the group sessions where learners were taking turns practicing. Those items they practiced themselves seemed to be better understood than those that were practiced by others. However, they did learn to turn to the student manual when questions arose during the assessment.

Participant A

Participant A was a graduate student and teaching assistant in the Communications department. Her score on the Experience Assessment was very slightly below average. She was chosen because of her experience with distance education and her desire to learn the system. She was also an IDT student. Her experience and knowledge was helpful in identifying areas that needed improvement. Participant A spent 2 hours on the instruction and was still unable to complete the training. This was due to a combination of reasons, including the frequent discussion that occurred throughout the session regarding the system and clarifying how things worked, the materials and how to improve them, as well as her need to leave at a certain time. Participant A referred to the student manual quite frequently during the session.

35

The first thing noted was that the information reviewed in the student manual was not specific enough. I added page numbers to and more specific verbiage to help with that. In the “Tour of the Room” section it was noted that there were eight speakers in the ceiling rather than four. This was edited for the next version of the manual. After turning on the main power the facilitator mentioned that the user should wait a few seconds before turning on the Gateway power switch. This was also added to the verbiage. The facilitator also noticed that the touch panel does not, in fact, default to the computer interface as indicated by the manual. This section was deemed irrelevant and removed from the text to save time in the training. The sequence and length of time needed for turning the projector on and off became a question which was investigated and edited in the text of the manuals. In the camera section, we added a note to watch the camera for movement to indicate whether it had been turned on. The note on the camera presets was edited and a separate screen shot was added for the pan, tilt and zoom section. The instructor microphone was unexpectedly on mute and therefore the volume control could not be used in the touch panel. This issue was not resolved until later in the evaluation process when a note was included in the text. Although Participant A was not able to meet one of the sub objectives of the goal to demonstrate use of instructor microphone (1.1.2), she did attempt to complete the task by taking the correct action and therefore this objective is considered met. There was not enough time to complete the instruction and assessment so this data is incomplete as well.A photo of the microphone clip was taken and inserted into this section as well.

After reviewing the “Recording Process” diagram it was reconfigured to more accurately reflect how recording is accomplished with the system. Details of the process of recording to mini-DV tape were carefully reviewed and refined. For example, directions were modified to include a reminder to look at the touch panel to determine if a tape starts playing when it is inserted it into the console. The Note in the “Recording to DVD” section and the Tech Tip in “Finalizing a DVD” was edited for accuracy. On the Job Aid for finalizing a DVD, a tech tip was added. Participant A and the facilitator went very carefully through this job aid and many changes were identified, including some of the verbiage which had been taken directly from the manufacturer’s manual. Participant A suggested that a break be added to the manual at this point due to her sense of cognitive overload but that did not become a priority in the project due to time constraints.

In Module III, we noted a need to make sure the learner kept the microphone on during the audio conference demonstration. It was also mentioned that there should be a note in the manuals about the fire wire cable being difficult to plug in. In the section on Breeze the facilitator suggested that the Breeze link be made a shortcut on the desktop. There were technical issues with the making Breeze detect the camera. We spent about 20 minutes trying to figure it out until we had to move on to the assessment without finishing the training. The assessment was negatively impacted by the lack of time to review the instruction on the Breeze module. The primary goal, demonstrate use of smart classroom equipment designed for distance, was not met, but a great deal of progress was made with the revisions that resulted from this session.

Participant B

The participant is the Dean of the College of Education and Human Development. His score on the Experience Assessment was below average. He was chosen because of his role as the Dean and his desire to become familiar with the technology used by his faculty. Participant B was the least experienced of the one-to-one learners but his lack of experience helped identify several

36

areas that were unclear and/or too long winded. He scored 100% on the matching touch panel quiz, 91% on the matching components quiz and 100% on the Fill-in-the-blank quiz which indicates that he met the requirements for objectives 1.1.1.3 and 1.1.1.4. The participant was encouraged to use the student manual as a tool to help him answer the checklist item questions. Experiences with this participant confirmed the idea that the student manual should be used as a tool during the instruction rather than presented as a reference later.

The first area of improvement was the objectives presented in the student and instructor manuals. They had not been re-written from the five component format and needed to be changed. They were revised after this session. In the “Getting Started” section, it was decided that the words, “green text to indicate when the instructor should be speaking” should be in green text to draw the facilitator’s eye to it. Also in this section, under Job Aids, the words, “This comprises the majority of the student manual” were deleted. In the part of this section which was spoken by the facilitator, some of the verbiage and order of information was changed and some was removed for flow and to save time.

In Part 2, some of the text was seen as repetitive and it was removed. Also, instructions to the facilitator to refer to the room diagram while pointing out the various components were added. The diagram entitled, “Tour of the System” was confusing and missing some components – changes were made upon revision. In the “Tour of the Room” section, a sentence about acoustic materials being added to the walls and floors was added. In the section on powering up, a practice item was added. The section on Shutting Down was moved to later in the training as was the practice item for turning the system back on. This practice item was moved because we felt that it was taking too much time and it was too simple to do so soon after it was presented in the training.

In the “Projector” section we added a practice item for the learners to try turning the projector on and off. When we had problems with it, we realized that it seemed to be going into sleep mode. A tech tip on the projector was written and added to the section. A practice item was also added to the “Camera” section so that the learner could try turning the camera on and watch it respond. To save time, the camera preset section was deleted. In the “Instructor Microphone” section duplication was removed and a tech tip was added regarding the mute button. Laptop objectives were still in use during this session but were later deleted due to time constraints.

We decided that it would save time to move the Module I checklist to the end of the training with the Module III checklist. In the Introduction to Module II we added a summary of “The Recording Process” diagram. In Part 6, some of the wording in the Tech Tip was changed for accuracy. In the practice section we thought a photo of the correct type of DVDs for the system would be helpful and we discovered a circle on the photo had been moved to an incorrect spot. In Part 7 we changed the “Discussion/Demonstration” section into a Practice session in which the facilitator could talk the learner through the steps of recording to a DVD. This was a section in which a few more idiosyncrasies with the system were identified and addressed by removing a note and going straight to the “Finalize a DVD” section. There were a few steps in the “Finalizing a DVD” job aid which needed to be edited and added. Also, a sentence about how to use the remote control with the touch panel was added.

In the section on Breeze, item 1.3.2.1, plugging fire wire, was not added until a later revision of the task analysis and assessment and was therefore not assessed or calculated in this participant’s information. Although it wasn’t covered with this participant due to time constraints, a few small

37

changes were made to the audio conference section to shorten and simplify the process. We felt it was more important that the Breeze portion of the instruction be assessed since it was not assessed in the previous session. Due to this omission, the learner did not meet the objective of entering an audio conference through Breeze by calling the room. If audio conference had been reviewed in the previous checklist item, he may have had the practice he needed to accomplish it for this objective.

Participant C

Participant C was an adjunct faculty member of the Department of Teaching and Learning. With above average experience, she achieved 100% on the Matching Touch Panel Quiz, 100% on the Matching Components Quiz and 84% on the Fill-in-the-Blank Quiz. She was an advanced user who employed the system to record her lecture sessions on DVD prior to this training. She had many good suggestions which were incorporated into the manuals.

The Getting Started section still seemed to drag on for a long time with a lot of lecturing going on. There was too much information not totally relevant being given to the learner and we decided it would be better to simply cut a lot of it out. Other minor edits were made to the text spoken by the facilitator to make for better flow.

Participant C suggested that the phrase “To use when streaming” be added to the “Tour of the system” diagram for clarity. It was added. Minor directions were added to the “Tour of the Room” section for the facilitator.

In Part 3 a Note was removed to save time. Due to the participant’s previous experience with the system, we asked her if she had any trouble with the projector. She said that she watches the touch panel to see if the OFF button turns red or the ON button turns blue. This tip was added to the instruction contained in the manuals. In addition, a practice section was moved from after the facilitator discussion to before so that the learner could experiment with the buttons while the facilitator talked about them.

A tech tip was added to tell learners that the longer the camera control buttons on the touch panel are held down, the quicker the camera moves. Practice sections were added here as well.

Participant C brought up a question regarding switching back and forth between media and microphone volume. After working on trying to find an answer for a while, we decided it was a question we would have to ask CILT. The question was never resolved and should be investigated for future revisions.

Participant C suggested putting a section on the USB extension after the photo of the USB ports. This suggestion was not taken because the “Plugging your laptop” section was removed from the instructor manual. The “Plugging in your laptop” section was removed from the instructor manual because we came to consider it a prerequisite. It was left in the student manual and placed at the back in the appendix.

Participant C felt that the word “simple” should be removed from the summary because it might make someone who does not find it simple, feel stupid. This was changed.

38

The student did not remember the steps to take in order stream a session and at this point it became clear that this needed to be emphasized in the instruction.

A few minor wording changes were made to the “Recording to mini-DV tape” section. The text on the JVC diagram was changed to more accurately match that which is on the actual component. The facilitator had made the suggestion in each prior one-to-one to “wave to yourself to make sure the DV tape is not playing back.” Participant C pointed out that the red light on the device stays on when it is recording. Both of these tips were added to the instruction. The wrong photos were cut and pasted into the “Recording to DVD” section from another section. This was corrected.We also determined that a further break down of the objectives “record to mini-DV tape and DVD” into smaller steps in the checklist and a detailed job aid on how to finalize a DVD were needed. These objectives were written as a result and included in the subsequent checklists.

In the “Finalizing a DVD” segment, a clarification was made about finalizing and un-finalizing DVDs. In the Job Aid, a tech tip was added about timing out and a few more clarifications were made.

Time constraints had kept us from working on the audio conference section with Participant B and Participant A had not done well with the task. We spent time with Participant C working on the item and decided to remove the “dial a long distance number using a touch panel” checklist item and did not ask Participant C to complete the task. Participant C suggested we explain more about voice over IPs so a brief statement was added to the “audio conference” section. A practice item was also added and the information on VoIPs was moved to another section. The volume section was circled on the photo for clarification.

Since this is a section that was not completely finished by either of the previous participants, we found several areas for improvement as we worked through it. A “something more” section was removed due to time constraints and a screen shot was deleted. A few screen shots were cropped to focus the learner in on the relevant content. In this segment, we decided to switch the camera and audio conference sections so that we could ensure that the camera is set up before the audio conference begins. This is helpful if there are camera initialization problems. Clarifications and corrections were made to the phone dialing section.

While going through the checklist items for Breeze, we found that the objectives needed to be refined a bit to include plugging in the fire wire. Objective 1.3.2.6, “demonstrate how to record a Breeze session” was omitted due to time constraints and the learner’s request to review the other material on Breeze.

After running through this instruction with three participants it was clear that it was originally written with far too much information to cover in the allotted time. It was also a challenge to continue finding glitches within the system that effected how the instruction was written. Ideally this instruction would have been better prepared had we performed at least one more one-to-one so that we could test the Breeze section more thoroughly. Still, there were far fewer changes on the next round of revisions so there was some progress achieved.

39

Small Group

First Session

Participant 1The first small group session was scheduled with two people, one male and one female, but only the female attended. Participant 1 was the Technology Coordinator for the department of Teaching and Learning. She was chosen because of her need to learn the system for her position in the department. As the Technology Coordinator for the College, Participant 1 was very familiar with the equipment and scored above average on the Experience Assessment. As such, she scored 100% on the Matching Touch Panel Quiz, 92% on the Matching Components Quiz and 100% on the Fill-in-the-Blank Quiz, which means she met the constraints for objective 1.1.1.4.

There were a few formatting issues with the table of content which were changed, and few of the page numbers referred to in the student manual were in need of correction.

In the “Power Up” section two Tech Tips and some of the instructor dialog was removed to save time. The Summary/Transition was also deleted for time. More minor changes were necessary to the Projector section and Practice Item was removed. While Participant 1 was trying to turn on the camera there were some technical issues which were remedied by rebooting the system. A Note was created to inform learners of this possibility. In the Camera section the word “camera” was replaced with the phrase, “cam control” to align with how it shows on the touch panel. The words, “further from” were added to the description of zoom for clarity. The type of batteries used for the Instructor Microphone was added to that section as well as a photo of correct mic placement.

When asked how to start a streaming session, Participant 1 looked around for a moment and then said I don’t know. The facilitator directed her to the student manual and she then answered the question correctly. In the “Recording to mini-DV tape” section, part of the introduction was removed because it was a duplication and a Tech Tip was removed from the instructor’s manual. This tip was left in the student manual for reference. Some of the wording was changed in the instruction and a Tech Tip on the HD/DVD switch was added. Edits were made to the graphic as well.

In Part 7, the information on DVDs and DVD-RWs was removed from the instructor presentation to save time but it remains in the student manual for reference. Some of the graphics here were edited as well and the order was changed for clarity. A tip for finalizing was added for clarity to the Finalize a DVD job aid and other minor edits to the text and graphics were made as well. The Tech Tip at the end of the section was removed but remains in the student manual.

In the section on Breeze several nonessential items were removed from the Instructor dialog to save time. Text and photos that had been added received minor edits and a “more reading” section was removed as well. Clarifications were made in the text on using audio conference in breeze and minor but important instructions were added.

40

Second Session

The second session of the small group was comprised of participants 2, 3 and 4 who completed the training in room 202 of the Education building. All participants are UND employees. Only Participant 3 is a faculty member of the Teaching and Learning Department. She was above average on the Experience Assessment due to her frequent use of smart classrooms. For this reason she entered the training thinking that she would not be learning anything new but ended up being enthusiastic about utilizing the distance features of the room for recording her class sessions. Participant 2 is the doctoral program coordinator for Teaching and Learning and took part in the training only in the interest of helping a student. She had a below average score on the Experience Assessment. Participant 4 is an instructional designer for the UND Aerospace program. Her responsibilities include training faculty and students in Breeze; her Experience Assessment score was above average. Her participation in the training was to help a fellow IDT student and to learn more about using smart classrooms for instruction.

The facilitator for this session was the director of the center for instructional and learning technologies and one of the people who designed the smart classroom. Since this was the first time she had reviewed the training, she took extra time discussing and in some cases correcting the instruction. Even so, the instruction was much more coherent in this session than in previous sessions thanks to the feedback of previous participants and much revision. The training did go over two hours with the assessment.

Many of the problems found during this session were typographical or formatting issues. The few substantial issues were related to system design problems that the facilitator clarified. Several of the arrows in the diagram labeled “The Recording Process” had to be removed or changed.

Participant 2Participant 2, who scored below average on the Experience Assessment, scored 100% on the Matching Touch Panel Quiz, 82% on the Matching Components Quiz and 100% on the Fill-in-the-Blank Quiz. While using the touch panel for microphone volume control, Participant 2 did not touch the microphone icon and turned changed the media volume instead. For objective 1.2.1, summarize the streaming process, the participant named CILT as the department to contact. The phone number to get help from CILT is posted on the podium next to the monitor and this is what people seemed to be turning to when asked about streaming. Final revisions include a note that the phone number on the podium is not the one used for streaming.

Participant 2 struggled with plugging in the fire wire, in part because she did not have an opportunity to practice during the session.

Participant 3Participant 3 was an above average learner who scored 100% on the Matching Touch Panel Quiz, 73% on the Matching Components Quiz and 100% on the Fill-in-the-Blank Quiz. Despite her low score on the Matching Components Quiz, her average for the objective was 91%, therefore she still reached the goal of 80% or better on objective 1.1.1.4. While attempting to demonstrate how to control the microphone volume using the touch panel, participant C failed to touch the mic icon before pressing volume and therefore changed the media volume instead. We

41

found this to be an issue that needed to be emphasized more in the instruction and revised the section accordingly. Although the learner did demonstrate Objective 1.1.6.1, “power down the projector using the touch panel,” the system was not ready to be shut down yet. A note was made in the instructor manual to ensure that this would not be an issue in future assessments.

When asked to summarize the streaming process, Participant 3 chose to contact the CILT department rather than the IDT department. More emphasis was given to the distinction of the different roles each department plays in support of this room. For objective 1.3.2.1, demonstrate plugging in the fire wire, the participant attempted to plug the fire wire into the USB extension on the podium rather than into the JVC recorder. Again, this may have been a result of lack of practice during the session and not enough use of the student manual as a tool. The facilitator provided guidance at this time by showing her the photos of the fire wire in the manual and where they should be plugged in. She then followed the instructions and continued on to the next objective.

Participant 4As the most experienced learner in the group, participant 4 scored very high on all assessments. She scored 100% on the Matching Touch Panel Quiz, 100% on the Matching Components Quiz and 100% on the Fill-in-the-Blank Quiz. The one question that she got wrong was when asked who should be contacted to stream a class session and she gave the feedback that the question should be rephrased; which it was.

III. Outcomes

Practice Items

The original version had practice items interspersed throughout the instruction but we found that the way they were presented interrupted the flow of learning and took far too much time. After experimenting with placement of practice items during the one-to-one sessions and the first group session, we found it to work best when we walked learners through the steps while they performed tasks. We also found that the student manual should be used more as a tool during the instruction rather than a reference. Many times we found ourselves reminding learners to use it while performing tasks, even tasks that we talked them through, so that they would feel comfortable doing so when they were on their own in the assessment and in practical use. In addition, participants in the second group session were not all able to perform practice items because there was not enough time or space to allow each learner to perform each task as it was taught. Each learner did get a chance to perform some tasks with the instructor, but all had to watch others perform as well. It was noted that those that performed the practice items seemed to perform better on the assessment on those particular items than those who did not. However, the changes made for trial and error made it difficult to accurately measure the use of practice items and their effectiveness over time.

42

Final Test

IV. InterpretationPractice items

Due to the trial and error used when designing practice items, data was not gathered. In the future I would keep class sizes as small as possible and include practice items after each module. Practice would be directly relevant to the material just covered and would be supported by the student manual. In this way the material will be reinforced and the learner will become accustomed to using the student manual as a tool.

The following is data, discussion and results of quizzes testing verbal information from objectives 1.1.1.3 and 1.1.1.4.

Matching Touch Panel Quiz

Participant A B C D E F G H I J K LA x x x x x x x x x x x xB x x x x x x x x x x x xC x x x x x x x x x x x x1 x x x x x x x x x x x x2 x x x x x x x x x x x x3 x x x x x x x x x x x x4 x x x x x x x x x x x x

ResultsNo errors or changes in this instrument occurred and all participants achieved a score of 100%.

The Matching Components Quiz

Participant A B C D E F G H I J K LA x x x x x* x x x x x* x**B x x x x -* x x x x x* xC x x x x x x x x x x x x1 x x - x x x x x x x x x2 x x x x - x x - x x x3 - x x x x - x x - x x4 x x x x x x x x x x x

KEY: X = correct; - = incorrect; shaded = item was removed/added upon revision*There was only one line for this item (mini-DV tape recorder/DVD recorder) to be answered so a line was added as a result. The extra line was later removed and the answer was changed to Mini-DV tape/DVD recorder to accommodate one line. **This text for this item was missing from the list for Participant A and was added as a result of the discovery to become item K.

DiscussionItems E and J, the mini-DV tape/DVD recorder, was listed twice in the answer key and did not have two slots for answers. Despite this, all but the least experienced participant (B) answered the question correctly by inserting both answers in one slot before it was fixed. Participant 2, the

43

next least experienced participant, got it wrong after it was fixed. In spite of the errors on the assessment instrument, the participants scored an overall average 85%.

Fill-in-the blank quiz

Participant Camera Projector Screen Server Podium Boundarymicrophones

A x x x - - xB x x x x x xC x x - x x x1 x x x x x x2 x x x x x x3 x x x x x x4 x x x x x x

DiscussionOn the original document the touch panel was not labeled. These items should have been labeled and choices should have been given for answers. Beginning with Participant B the facilitator was asked to encourage the learner to refer to the student manual when completing this quiz. Participant C decided to try to do it from memory. An overall average of 71 % was reached with this quiz.

ResultsOut of 121 possible correct answers, participants only got 10 wrong, which is an average of about 83%, therefore this objective (1.1.1.4) was met.

PracticePractice items could not be measured due to the trial and error of their use and placement.

44

Conclusions and Future Revisions

Overall this training went very well and accomplished much of what it was designed to do. The one-to-one settings were very productive in that they allowed learners to experiment with the equipment in the context of the instruction while simultaneously testing the instruction for accuracy. This was especially helpful because of the newness of the system. Although one of the participants in the one-to-one sessions was not able to finish the performance checklist and did not, therefore, meet the main goal of the training, many typographical, formatting and instructional errors were identified and corrected. One more one-to-one session would have been helpful to test the Breeze section more thoroughly.

The group setting was more challenging because of the space and time available. Practice items were done by various people within the group which meant that no one person could have hands-on experience with every component. This instruction was designed as facilitator-led to expose learners to the equipment and give them a chance to experience handling it first-hand but it was unrealistic for each person to do so in this group setting. Some of the assessment scores suffered because those items learners practiced themselves seemed to be better understood than those that were practiced by others.

An issue with the design of the touch panel caused some concern for most of the learners; the mic and media volume are controlled by the same volume control icon. A user must press the mic icon to switch to the mic volume but there is no indication as to whether it is in use or how to switch back to media volume. Unless this feature is changed, future training should include a note on the problem to make learners aware of it.

In the future, a strong emphasis on the student manual as a tool is recommended. The manual is laid out logically with step-by-step instructions for all of the required tasks. Familiarity with the manual will help students with the assessments and subsequent use of the room. If taught in a group setting, I recommend that the more difficult items, such as plugging in the fire wire, are practiced by each participant. In the future I would keep class sizes as small as possible and include practice items after each module. Practice would be directly relevant to the material just covered and would be supported by the student manual. In this way the material will be reinforced and the learner will become accustomed to using the student manual as a tool.

During the Implementation stage, revisions were made to the objectives in the task analysis which resulted in changes to Performance Checklist. These revisions were not reflected in the matrix until after the implementation phase. The data gathered and analyzed is based on the format of the task analysis prior to revision. Numbers are indicated in the table where necessary.

45

V. Appendices

46

Appendix AAll Participants

Figure 1

Figure 2

47

Table 1. Response to experience questions on Experience Assessment.

Questions1 2 3 4 5 6 7 8 9 Total

Participant Average Average %

A 5 5 3 1 1 1 4 5 5 30 3.33 67B 5 5 2 1 1 1 3 1 5 24 2.67 53C 5 4 4 1 5 5 4 5 1 34 3.78 761 5 4 4 1 5 2 5 1 5 32 3.56 712 5 4 4 1 1 1 4 1 5 22 2.89 493 5 4 4 1 5 4 4 5 1 33 3.67 734 5 5 3 5 5 1 4 5 1 34 3.78 76Average 5 4.43 3.43 1.57 3.29 2.14 4 3.29 3.29 23 3.38 51One-to-One Composite 5 4.67 3 1 2.33 2.33 3.67 3.67 3.67 3.26

Small Group Composite 5 4.25 3.75 2 4 2 4.25 3 3 3.47

Below Average Learners 5 4.67 3 1 1 1 3.67 2.33 5 2.96

Above Average Learners 5 4.25 3.75 2 5 3 4.25 4 2 3.69

Answers to Question 10

Participant A: Technical difficulties in sound, video or connection.Participant B: The time it takes to develop the course and then the new skills one might need to learn.Participant C: 1) The fact that the work is all safely hidden away in my computer 2) not knowing the people fact to face.Participant 1: No opportunity for hands-on experience.Participant 2: My experience is in correspondence – which is not the same as distance education. I never have a time when I can connect with a student except over e-mail. I think the biggest drawback is that sometimes when students are writing culturally insensitive or factually untrue responses, I believe that a face-to-face conversation would be beneficial. Or, a redirection to a reading that might help them reframe their thinking. This is hard to do in correspondence. I can certainly suggest a link or an article to read, but I have no way of knowing if they follow through with it.Participant 3: One barrier is that there is limited personal interaction with peers. The course I took as a biology class so it wasn’t as critical there, but in some education courses, we are teaching and modeling effective personal interactions. On the flip side, I found that I needed to know my content very well, as the grade was 100% objective based on my test scores. One further disadvantage with the class I took related to this was that the tests were open book, as they were taken at home. In that regard, I could see a potential for academic dishonesty. Participant 4: Trying to get student skills and technology up to speed so that everyone can focus on the subject matter.

Figure 348

Figure 4

49

Appendix BOne-to-One Participants

Figure 5

Figure 6

Figure 7

50

Figure 8

51

Appendix CSmall Group Participants

Figure 9

Figure 10

52

Figure 11

53

Appendix D Assessment

54

Table 5. One-to-One “Practice Checklist” Items

Objective 1 1.1 1.1.1 1.1.1.1

1.1.1.2

1.1.1.3

1.1.1.4

1.1.2 1.1.2.1

1.1.2.1.1

1.1.2.1.1.1

1.1.3 1.1.3.1

1.1.3.1.1

1.1.3.1.1.1

1.1.4 1.1.4.1

1.1.5 1.1.5.1

1.1.5.1.1

1.1.6 1.1.6.1

1.2

Student A x x x x x x x x - x x x x x x x x - - - x - -

Student B x x x x x x x x x x x x x x x x x x x x x x x

Student C x x x x x x x x x x x x x x x x x x x x x x x

# Correct 3 3 3 3 3 3 3 3 2 3 3 3 3 3 3 3 3 2 2 2 3 2 2

% Correct 100 100 100 100 100 100 100 100 66 100 100 100 100 100 100 100 100 66 66 66 100 66 66

% Mastery

100 100 100 100 100 100 100 100 66 100 100 100 100 100 100 100 100 66 66 66 100 66 66

Objective

1.2.1 1.2.2 1.2.2.1

1.2.2.2

1.2.2.3

1.2.2.3.1

1.2.3 1.2.3.1

1.2.3.2

1.2.3.3

1.2.3.4

1.3 1.3.1 1.3.2 1.3.2.1

1.3.2.2

1.3.2.3

1.3.2.4

1.3.2.5

1.3.2.6

1.3.2.7

# %

Student A - - - - - - - - - - - - - - - 38 42

Student B x x x x x x - - x x x - x x x 38 92

Student C - x x x x x x x x x x x x - x 38 95

# Correct 1 2 2 2 2 2 1 1 2 2 2 1 2 1 2 68 60

% Correct 33 66 66 66 66 66 33 33 66 66 66 33 66 33 66 92 79

% Mastery

33 66 66 66 66 66 33 33 66 66 66 33 66 33 66 100

Items shaded purple were not objectives until later in the evaluation process.

55

Table 6. Small Group “Practice Checklist” Items

Objective 1 1.1 1.1.1 1.1.1.1

1.1.1.2

1.1.1.3

1.1.1.4

1.1.2 1.1.2.1

1.1.2.1.1

1.1.2.1.1.1

1.1.3 1.1.3.1

1.1.3.1.1

1.1.3.1.1.1

1.1.4 1.1.4.1

1.1.5 1.1.5.1

1.1.5.1.1

1.1.6 1.1.6.1

1.2

Student 1 x x x x x x x x x x x x x x x x x x x x x x x

Student 2 x x x x x x x x - x x x x x x x x x x x

Student 3 x x x x x x x x - x x x x x x x x x x x

Student 4 x x x x x x x x x x x x x x x x x x x x

# Correct 4 4 4 4 4 4 4 4 2 4 4 4 4 4 4 4 4 1 1 1 4 4 4

% Correct 100 100 100 100 100 100 100 100 50 100 100 100 100 100 100 100 100 100 100 100 100 100 100

% Mastery

100 100 100 100 100 100 100 100 50 100 100 100 100 100 100 100 100 100 100 100 100 100 100

Objective

1.2.1 1.2.2 1.2.2.1

1.2.2.2

1.2.2.3

1.2.2.3.1

1.2.3 1.2.3.1

1.2.3.2

1.2.3.3

1.2.3.4

1.3 1.3.1 1.3.2 1.3.2.1

1.3.2.2

1.3.2.3

1.3.2.4

1.3.2.5

1.3.2.6

1.3.2.7

# %

Student 1 - x x x x x x x x x x x x x x x x 40 98

Student 2 - x x x x x x x x x x x x x - x x x x x x 41 93

Student 3 - x x x x x x - x x x x x x x x x x x x x 41 93

Student 4 - x x x x x x x x x x x x x x x x x x x x 41 98

# Correct 0 4 3 3 4 4 4 2 3 4 4 4 4 4 3 4 4 4 4 4 4 155 95

% Correct 0 100 100 100 100 100 100 75 100 100 100 100 100 100 75 100 100 100 100 100 100 154 95

% Mastery

0 100 100 100 100 100 100 75 100 100 100 100 100 100 75 100 100 100 100 100 100 100

56

Figure 12

Figure 13

Figure 14

57

.

Figure 15

58