dt2119: speech and speaker recognition; course analysis vt2018

DT2119: Speech and Speaker RecognitionCourse Analysis VT2018

Course analysis carried out by (name, e-mail):

Giampiero Salvi, [email protected]

COURSE DESIGN

Briefly describe the course design (learning activities, examinations) and any changesthat have been implemented since the last course offering.

In VT2018 the course consisted of 12 lectures, 3 computer labs each evaluated with an oralpresentation and a final project evaluated with a written report, peer review and a poster session.Special care was put into aligning the learning outcomes with the teaching, learning activitiesand assessment. This is a course that, besides the specific topic, intends to teach the basicresearch paradigm, based on performing scientific studies, peer review, and public presentations.

The main differences with previous years were the following:

• the third lab on deep learning was completely rewritten and now is based on Pythonand TensorFlow. This improves the pedagogical value of the lab because it lets studentsimplement most of the methods used,

• improvement were made to lab 1 and 2 to address some of the feedback received in VT2017,

• grading criteria were improved,

• some improvements were made to the software available at Parallel Data Centre machinesto simplify use for the students

• credit to the Google Cloud Platform was obtained under an academic licence and theservice was made available to the students

THE STUDENT’S WORKLOAD

Does the students’ workload correspond to the expected level (40 hours/1.5 credits)?If there is a significant deviation from the expected, what can be the reason?

The students who participated in this course belonged to eight different programs (CDATE,CINTE, CMIEL, D, TMAIM, TMETM, TMLEM, TSCRM) and had, therefore different back-grounds. Consequently, the workload exhibits large variations, Besides one student who reported33–35 hours/week load, most students range from 3–5 to 18–20 hours/week. On average, thework load is 12.5–14.5 hours/week which is low compared to the 20 hours/week required for thiskind of courses. There is still room to include more teaching and learning activities, for examplea fourth lab.

(of 22)

DT2119 Speech and Speaker Recognition • Vårterminen 2018Giampiero Salvi

THE STUDENTS’ RESULTS

How well have the students succeeded on the course? If there are significant differ-ences compared to previous course offerings, what can be the reason?

There were 55 students applying and registering to the course (source Canvas). Of them, 43were active during the course and passed at the end. Below you can see the evolution in time forthe course (until 2016 the course code was DT2118). The increase in number of students seemto have stopped in 2018. This may be due to the course reaching its limit, also considering theever increasing course offering on Machine Learning topics at KTH.

2011

2012

2013

2014

2015

2016

2017

2018

10

20

30

40

50

Year

#stud

ents

# students in DT2118/9

registeredactivepassed

The milestones during the years are:

VT2015: the matlab exercises and computer lab based on the HTK software package weresubstituted for three python based labs on Feature Extraction, HMMs and ContinuousSpeech Recognition in order to improve the alignment between the learning outcomes, theteaching/learning activities and the assessment. However, Lab 3 was still based on theHTK software package.

VT2016: An alternative version of Lab 3 was introduced to include the Deep Learning modelsthat became state-of-the art in speech recognition. However, this lab was still based onpre-made software packages and offered limited learning opportunities for the students

VT2017: An introduction to signal processing was added to the course to help student lackingthat background (mainly computer science students).

VT2018: Lab 3 was completely redesigned in Python/TensorFlow to allow the students toimplement most of the methods, giving them a better opportunity to learn.

(of 22)


OVERALL IMPRESSION OF THE LEARNING ENVIRONMENT

What is your overall impression of the learning environment in the polar diagram?If there are significant differences between different groups of students, what can bethe reason?

Only 13 students responded to the survey. The overall impression is that the students highlyenjoyed the course and found it stimulating (Q1=5.6). They find the course less challengingcompared to previous years (Q4=4.8). This probably is due to the different distribution ofstudents that joined the course this year that are mainly from the machine learning master. Itis, therefore possible, to design more challenging tasks in the course. Similarly the backgroundof participants seems to be adequate (Q17=5.8) and the students thought that the assessmentwas fair and honest (Q16=5.6). The lowest score (4.4) was for Q13: “I understood what I wasexpected to learn in order to obtain a certain grade”, which indicates that the grading criteriacan still be improved. It is to note that 4.4 is not a low score.

The system identified several subgroups. The distinction between men and women is ratherinteresting and shows that women are much more positive than man in their feedback. Thequestions with the larges difference were Q1 (interest), Q6 (“The atmosphere on the coursewas open and inclusive”), Q7–Q8 (clear goals and organisation), Q9–Q11 (understanding ofsubject matter), Q12: “The course activities helped me to achieve the intended learning outcomesefficiently”, Q21 (collaboration) and Q22 (support). This is very encouraging. The distinctionbetween international master students and total, does not show interesting differences.

ANALYSIS OF THE LEARNING ENVIRONMENT

Can you identify some stronger or weaker areas of the learning environment in thepolar diagram - or in the response to each statement - respectively? Do they havean explanation?

From the numeric results of the survey, it seems that the course does a good job in most ofthe areas. Compared to previous years, the students who apply to this course seem to have amore fitting background resulting in lower work loads and lower perception of the course beingchallenging. This means that the assignments can be extended both in terms of coverage ofthe course topics and in terms of difficulty. Also, in the past years, when the course was runon KTH Course Web, one of the most appreciated feature was the availability of the teacherto answer question through the forum. Now, since Canvas has been introduced, the score onquestions related to feedback have decreased. I intend to investigate if this is due to the forumfunctionality in Canvas that is not as user friendly as in Course Web, or if I have been less presentdue to other engagements (the Machine Learning Master Admission process takes a significantamount of my time during the course). However, the open questions on Canvas and feedbackfrom the teacher were extremely positive, making me think that the lower numerical scores aresimply due to the different scoring traditions of international students.

Throughout the years I have been meaning to find the resources to organise meetings withthe students during the project to check progress and provide some form of supervision. PerhapsI can achieve this by recruiting some teaching assistants with a good background in the subject.

(of 22)


ANSWERS TO OPEN QUESTIONS

What emerges in the students’ answers to the open questions? Is there any goodadvice to future course participants that you want to pass on?

There is a majority of very positive answers to open questions which is very encouraging. Thelab material was in general considered very positively, and students were extremely happy aboutthe way the course was organised on Canvas. Many had difficulties using the computationalresources at PDC, and were glad that the alternative of Google Cloud Platform was madeavailable in the course. Some improvements are required for lab 3, and the students would liketo have TA sessions during the labs and project to get assistance.

PRIORITY COURSE DEVELOPMENT

What aspects of the course should primarily be developed? How could these aspectsbe developed in the short or long term?

The course has reached maturity. However there is still room for improvements:

• The instructions of lab 3 should be revised.

• Also, writing lab 3, I realised that the instructions of some aspects of lab 2 could be mademore general so to fit both labs and simplify the work for the students.

• I will consider adding a fourth lab on continuous speech recognition. This is an importantaspect in the course and it would be good if the students dedicated some extra time onit. The problem so far has been making sure that the students have enough time to carryout four labs without affecting the results of the project. However, the reported work loadindicates that there is room for adding more tasks in the course.

• The grading criteria need to be improved as some students reported they were not surewhat they were expected to do to obtain a certain grade.

• I will try to define more Canvas quizzes to cover those topics that are not covered by labsand project. This has also been requested by the students.

RESULT OF THE STUDENT SURVEY

In the next pages I report the result of the student survey.

(of 22)


DT2119 - 2018-06-06

Antal respondenter: 46

Antal svar: 13

Svarsfrekvens: 28,26 %

ESTIMA

TED W

OR

KLO

AD

Com

ments

Com

ments (I w

orked: 9-11 timmar/vecka)

I can't really remem

ber the hours/week but it felt like a good w

ork load/credits ratio.

Com

ments (I w


There was no w

ay near enough time to read the reading m

aterial associated with the lectures. (B

ut maybe this w

as not the idea)

LEAR

NIN

G EXPER

IENC

E

The polar diagrams below

show the average response to the LE

Q

statements for different groups of respondents (only valid responses are

included). The scale that is used in the diagrams is defined by:

1 = No, I strongly disagree w

ith the statement

4 = I am neutral to the statem

ent7 = Y

es, I strongly agree with the statem

ent

Note! A

group has to include at least 3 respondents in order to appear ina diagram

.

KTH

Learning Experience Questionnaire v3.1.3

Meaningfulness - em

otional level

Stim

ulating tasks

1. I worked w

ith interesting issues (a)

Exploration and ow

n experience

2. I explored parts of the subject on my ow

n (a)3. I w

as able to learn by trying out my ow

n ideas (b)

Challenge

4. The course was challenging in a stim

ulating way (c)

Belonging

5. I felt togetherness with others on the course (d)

6. The atmosphere on the course w

as open and inclusive (d)

Com

prehensibility - cognitive level

Clear goals and organization

7. The intended learning outcomes helped m

e to understand what I w

as expected to achieve (e)8. I understood how

the course was organized and w

hat I was expected

to do (e)

Understanding of subject m

atter

9. I understood what the teachers w

ere talking about (f)10. I w

as able to learn from concrete exam

ples that I could relate to (g)11. U

nderstanding of key concepts had high priority (h)

Constructive alignm

ent

12. The course activities helped me to achieve the intended learning

outcomes efficiently (i)

13. I understood what I w

as expected to learn in order to obtain a certaingrade (i)

Feedback and security

14. I received regular feedback that helped me to see m

y progress (j)15. I could practice and receive feedback w

ithout being graded (j)16. The assessm

ent on the course was fair and honest (k)

Manageability - instrum

ental level

Sufficient background know

ledge

17. My background know

ledge was sufficient to follow

the course (f)

Time to reflect

18. I regularly spent time to reflect on w

hat I learned (l)

Variation and choices

19. I was able to learn in a w

ay that suited me (m

)20. I had opportunities to choose w

hat to do (m)

Collaboration

21. I was able to learn by collaborating and discussing w

ith others (n)

Support

22. I was able to get support if I needed it (c)

Learning factors from the literature that LEQ

intends to examine

We tend to learn m

ost effectively (in ways that m

ake a sustained, substantial, and positive influence on the w

ay we think, reflect, act or

feel) when:

a) We are trying to answ

er questions, solve problems or acquire skills

that we find interesting, intriguing or im

portant

b) We can speculate, try out ideas (intellectually or practically) and learn

from experience, even before w

e know much about the subject

c) We are able to do so in a challenging yet supportive environm

ent

d) We feel that w

e are part of a community and believe that other people

have faith in our ability to learn

e) We understand the m

eaning of the intended learning outcomes, how

the environm

ent is organized and what is expected of us

f) We have sufficient background know

ledge to manage the present

learning situation

g) We can learn inductively by m

oving from specific exam

ples and experiences to general principles, rather than the other w

ay around

h) We are challenged to develop a proper understanding of key

concepts and successively create a coherent whole of the content

i) We believe that the w

ork we are expected to do w

ill help us to reach the intended learning outcom

es

j) We can try, fail, and receive feedback in advance of and separate from

any summative judgm

ent of our efforts

k) We believe that our w

ork will be considered fairly and honestly

l) We have sufficient tim

e to learn and devote the time necessary to do

so

m) W

e believe that we are in control of our ow

n learning, not manipulated

n) We can w

ork collaboratively with other learners struggling w

ith the sam

e problems

Literature

Bain, K

. (2004). What the B

est College Teachers D

o, Chapter 5, pp.

98-134. Cam

bridge: Harvard U

niversity Press.

Biggs J. &

Tang, C. (2011). Teaching for Q

uality Learning at University,

Chapter 6, pp. 95-110. M

aidenhead: McG

raw Hill.

Elmgren, M

. & Henriksson, A

-S. (2014). A

cademic Teaching, C

hapter 3, pp. 57-72. Lund: S

tudentlitteratur.

Kem

ber, K. &

McN

aught, C. (2007). E

nhancing University Teaching:

Lessons from R

esearch into Aw

ard-Winning Teachers, C

hapter 5, pp. 31-40. A

bingdon: Routledge.

Ram

sden, P. (2003). Learning to Teach in H

igher Education, C

hapter 6, pp. 84-105. N

ew York: R

outledgeFalmer.

Com

ments

Com

ments

Com

ments (I am

: Svensk student i årskurs 1-3)

Thanks for an interesting course! :)

GEN

ERA

L QU

ESTION

S

What w

as the best aspect of the course?

What w

as the best aspect of the course? (I worked: 3-5 tim

mar/vecka)

The labsThe labs and the final project

What w


mar/vecka)

Very relevant and w

ell written assignm

ents throughout the course.The project, allow

ing for us to do something w

e are interested in within the field.

Good labs. C

lear lab instructions on the first two labs.

What w


mar/vecka)

The freedom to choose alm

ost any speech related subject for the final projectFöreläsningarna, att det var m

ycket arbete i grupp, att projektet var så fritt och redovisningsformatet m

ed posters.

What w


mar/vecka)

the projectCom

ing into contact with speech preprocessing

What w


mar/vecka)

That it covered a lot of topics

What w

ould you suggest to improve?

What w

ould you suggest to improve? (I w


Maybe propose subjects for projects

It suppose to have course quiz

What w



Maybe em

phasize a bit on speech characteristics when m

odels are presented. More guidance during the project

The lab instructions on the third lab.

What w



Improve the assignm

ent descriptions. Som

e people including me lost a lot of tim

e trying to understand what the description text w

ants us to do and how

. There w

ere cases where the description w

as apparently not very carefully written and these things w

ere fixed only after someone

poses question on the discussions.

- Det hade varit trevligt att ha alla quizzar.

- Labb 3 tog väldigt mycket tid och inkräktade därm

ed på projektet. Mycket av det berodde på att m

ängden data var såpass stor så våra datorerhade problem

med att hantera den.

What w



better defined assignments w

ithout bugs in the skeletons T

he signal processing lecture was not sufficient to understand the reasons behind the different m

anipulations of the speech signal.

Less focus on recreation of historic techniques, more critical analysis of them

(what are their draw

backs, and where do they com

e from), and

more contact w

ith modern techniques and how

they try to solve some of the draw

backs.

What w



Add prerequisite courses

Add TA

help sessions for the labs and the project

What advice w

ould you like to give to future participants?

What advice w

ould you like to give to future participants? (I worked: 3-5 tim

mar/vecka)

pay more attention to the final project

What advice w


mar/vecka)

Try to start the project as early as possible.Learn about sound theory to better understand the labs

What advice w


mar/vecka)

Hitta en labb-/projektpartner du trivs m

ed för ni kommer att tillbringa m

ycket tid ihop.

What advice w


mar/vecka)

start the lab ASAP as you w

ill find a lot of bugsGet quite fam

iliar with the preprocessing pipelineIs there anything else you w

ould like to add?

Is there anything else you would like to add? (I w


noIs there anything else you would like to add? (I w


No



Jag använde inte kurslitteraturen alls men hade viss nytta av Jurafsky &

Martins S

peech and language processing.



please pay attention to the lab specsModern A

SR still seem

s like it's just an interface above a really bad UX design, and the course did not help change that view

. It may not have

been in the course's aims though. In any case, I do not feel I have any greater appreciation for the technology and its applications, it still seem

slike a terrible w

ay to interact with com

puters

SPECIFIC

QU

ESTION

S

Arbetsinsatsen var jäm

nfördelad under kurstiden

Arbetsinsatsen var jäm

nfördelad under kurstidenYes

No, there w

as too much w

ork in a short period of time for Lab2 and Lab3 (given that w

e were doing the project at the sam

e time). O

n the other hand there w

as too much tim

e given for Lab1. How

ever, I assume the labs 2 and 3 w

ould take far less time to im

plement if w

e had better instructions in the assignm

ent description.Yes that is som

ething I like about the course !Y

es, it was evenly distributed and felt fair

Yes, m

aybe except for the last projectYes

Yes, a part from

lab3 being to heavyyesyesDet m

esta av arbetet lades på labb 3 och projektet som båda var på slutet.

Laborationerna hjälpte mig förstå de teoretiska aspekterna

Laborationerna hjälpte mig förstå de teoretiska aspekterna

Significantly

Yes but I learned a lot of stuff during the actual presentation from

the TA

s. I believe all that stuff should be included in the description or at least hints should be given so that w

e do not squander our time.

Yes definitely but it w

as sometim

e hard to meet the deadlines w

ith my 2-3 other courses

Yes

Yes

Yes

to an extend. They were not alw

ays clear on what their aim

was, or w

hat particular part of the whole recognition process w

e were dealing w

ith in a concrete and focused w

ay. Lab 3 was particularly badly form

ed, with the task being unclear both in its nature and in its necessity, since w

e already w

ere doing the same thing w

ith HMMs. It w

as a good discussion on the pros and cons of each method at least

yesJa

Informationsflödet i C

anvas var tidsenlig och användbar

Informationsflödet i C

anvas var tidsenlig och användbarYes

Yes

Yes

To some extend yes

Very helpful

Yes

Yes

Yes, the teacher did a very good job w

ith canvas. He even added a deadline for us to apply for pdc accounts so w

e wouldn't forget

yesyesJa, C

anvas funkade överlag väldigt bra

Beräkningsresurserna var adekvat (PD

C/G

CP/A

WS)

Beräkningsresurserna var adekvat (P

DC/GCP/AWS)

PD

C w

as unusable in the end, AW

S w

as very helpful and easy to use. P

s : I used AW

S too m

uch and consumed m

ore than my free credit. T

he billing dashboard is not clear, and seems to take tim

e to refresh. We did not run into any issues but I w

as afraid that running certain stuff on GCP would m

ake us run out of credits. Yes but kind of hard to use som

etimes.

Yeah, I like that you added google clouds. They are so easy to use

They were a bit com

plicatedPDC was very hard to use and w

e spent many hours trying to get it to w

ork. Once it w

as working it w

as extremely hard to get a tim

e slot for running our jobs. The google cloud platform

which w

e used instead, was how

ever very convenient and user friendly! GCP was. W

e were unable to use P

DC after correctly setting everything up

GCP

Föreläsningen om PDC kändes inte alls anpassad efter oss och vad vi behövde veta och klargjorde inte särskilt m

ycket. Vi använde istället

Google C

loud och det fungerade utmärkt, lätt att kom

ma igång och kvoten räckte m

er än väl. Vi hade dock behövt m

er minne än förslaget i

instruktionerna, men det visste vi ju nte när vi skapade vår V

M och det gjorde att vi fick hoppa över vissa körningar.

RESPO

NSE D

ATA

The diagrams below

show the detailed response to the LE

Q statem

ents.The response scale is defined by:

-3 = No, I strongly disagree w

ith the statement

0 = I am neutral to the statem

ent+3 = Y

es, I strongly agree with the statem

ent

X = I decline to take a position on the statem

ent

Com

ments

Com

ments

Com

ments

Com

ments

Com

ments (M

y response was: +2)

Yes, but som

etimes frustrating, especially lab3 since this lab w

as pretty extensive, stressful and a bit unclear

Com

ments

Com

ments (M

y response was: +3)

Students w

here very helpful! And there w

as a very open and inclusive atmosphere in the class!

Com

ments

Com

ments

Com

ments

Com

ments

Com

ments (M

y response was: +1)

Most of the tim

e

Com

ments

Com

ments

Com

ments (M

y response was: +3)

Very w

ell structured powerpoints and lectures!

Com

ments

Com

ments

Com

ments (M

y response was: +1)

Lite oklart hur projektet kommer att betygsättas.

Com

ments

Com

ments (M

y response was: +2)

How

ever, I think it could be good with helpsessions w

ith TAs as w

e had in the DD2424

Com

ments

Com

ments

Com

ments (M

y response was: X

)No assesm

ent yet, still waiting for project grades

Com

ments

Com

ments (M

y response was: -2)

Without any background in m

achine learning, advanced machine and signal processing it w

as sometim

es very challenging to follow the course.

I would recom

mend adding them

as prerequisites or recommended prerequisites on the course page

Com

ments

Com

ments

Com

ments

Com

ments (M

y response was: -2)

During the project only

Com

ments

Com

ments

dt2119: speech and speaker recognition; course analysis vt2018

Documents