vtechworks.lib.vt.edu · web view2020-05-14 · deep learning course. final report. instructor:...

Deep Learning Course

Final Report

Instructor: Dr. Edward A. FoxCS 4624: Multimedia, Hypertext, and Information Access

Virginia Tech, Blacksburg VA 2406105/11/2020

Client: Amirsina Torfi

Team: Brendan Bolon | Samuel Burton | Harrison Ellis | Matthew Robinson | Andrew Whelan

1

Table of ContentsTable of Figures.......................................................................................................................3

Table of Tables........................................................................................................................4

Executive Summary.................................................................................................................5

Introduction............................................................................................................................5

Objective....................................................................................................................................5

Deliverables............................................................................................................................... 6

Client..........................................................................................................................................6

Team..........................................................................................................................................6

Requirements..........................................................................................................................8

Open Source Deep Learning Course..........................................................................................8

PyTorch...................................................................................................................................... 8

ReStructuredText Documentation.............................................................................................8

Design.....................................................................................................................................9

Implementation.....................................................................................................................10

Evaluation.............................................................................................................................10

User Manual..........................................................................................................................11

Homepage............................................................................................................................... 11

Course Introduction.................................................................................................................12

Introduction to Deep Learning Module...................................................................................12

What is Deep Learning?...........................................................................................................13

Why Deep Learning?................................................................................................................14

Applications of Deep Learning.................................................................................................14

Basics Module..........................................................................................................................15

Logistic Regression...................................................................................................................16

Derivatives and The Chain Rule...............................................................................................17

Gradient Descent.....................................................................................................................18

Simple Neural Networks..........................................................................................................19

Simple Logistic Regression.......................................................................................................21

Neural Network Module..........................................................................................................22

Overview..................................................................................................................................23

Computation............................................................................................................................24

2

Activation Functions................................................................................................................ 25

Backpropagation......................................................................................................................26

Simple Neural Network Classifier.............................................................................................27

More on Deep Neural Networks Module.................................................................................29

Regularization..........................................................................................................................30

Dropout................................................................................................................................... 31

Gradient Setbacks....................................................................................................................32

Batch Normalization................................................................................................................ 33

Final Notes and Citations.........................................................................................................33

Developer Manual.................................................................................................................34

Course Layout..........................................................................................................................34

Contribution.............................................................................................................................37

Methodology.........................................................................................................................38

Lessons Learned....................................................................................................................46

Timeline/Schedule...................................................................................................................46

Problems..................................................................................................................................47

Solutions.................................................................................................................................. 48

Solutions.................................................................................................................................. 49

Future Work.............................................................................................................................49

Acknowledgements...............................................................................................................50

References.............................................................................................................................51

3

Table of FiguresFigure 1: Class Organization and Structure....................................................................................9Figure 2: Homepage View............................................................................................................11Figure 3: Course Introduction View.............................................................................................12Figure 4: Deep Learning Module..................................................................................................12Figure 5: What is Deep Learning Page.........................................................................................13Figure 6: Why Deep Learning Page..............................................................................................14Figure 7: Applications of Deep Learning Page..............................................................................14Figure 8: Basics Module...............................................................................................................15Figure 9: Logistic Regression Page...............................................................................................16Figure 10: Derivations and Chain Rule Page................................................................................17Figure 11: Gradient Descent Page................................................................................................18Figure 12: Neural Network Architecture Page.............................................................................19Figure 13: Neural Network Documentation.................................................................................20Figure 14: Logistic Regression Documentation Page...................................................................21Figure 15: Neural Network Module.............................................................................................22Figure 16: Neural Network Overview Page..................................................................................23Figure 17: Neural Network Computation Page............................................................................24Figure 18: Activation Functions Page...........................................................................................25Figure 19: Backpropagation Page................................................................................................26Figure 20: Neural Network Classifier Page...................................................................................27Figure 21: Neural Network Classifier Code Documentation........................................................28Figure 22: Deep Neural Networks Module..................................................................................29Figure 23: Regularization Page.....................................................................................................30Figure 24: Dropout Page..............................................................................................................31Figure 25: Gradient Setbacks Page..............................................................................................32Figure 26: Batch Normalization Page...........................................................................................33Figure 27: rST Paragraph..............................................................................................................34Figure 28: rST Code Block............................................................................................................ 34Figure 29: rST Code Block on Page Example................................................................................35Figure 30: rST Embedded Links....................................................................................................35Figure 31: rST Embedded Block on Page Example.......................................................................35Figure 32: rST Table..................................................................................................................... 36Figure 33: rST Table on Page Example.........................................................................................36Figure 34: Pull Request Process...................................................................................................37Figure 35: Workflow Diagram......................................................................................................46

4

Table of TablesTable 1: Example Persona and Goals...........................................................................................38Table 2: Tasks and Subtasks of Goals for Personas......................................................................41Table 3: Schedule.........................................................................................................................47

5

Executive Summary

Our project produced an open source course for deep learning that is hosted on GitHub. Our course is broken up into modules, each with sub-sections that are designed to give users a better understanding of how deep learning works and how to write code for deep learning. The sub-sections for each module have documentation describing the concepts in-depth and are accompanied by PyTorch code to show how the concepts work in practice. Under instruction from our client, Amirsina Torfi, we built this course following the format of courses he has designed before. Users are expected to move sequentially through the modules, as the concepts build on previous concepts.

The deep learning course assumes that users have experience with writing code and with computer science concepts. The course also contains a large amount of linear algebra concepts and expects the user to have a background in that field. Users will still have much to gain without this background, but the course was designed with some previous knowledge in mind. Two aspects of our project are the design of the website using reStructuredText and our GitHub repository of PyTorch code. Our documentation was done using reStructuredText. The GitHub repository contains all of the source code for each module, written in PyTorch. Making the course open source and free was done to make it accessible to everyone, as there are not many free open source courses for deep learning. In order to test our course, we gave our fellow classmates as well as a few fellow CS majors access to our course. After taking the course, we were provided constructive feedback from our peers, and we also received feedback via a Canvas quiz that our classmates were able to take as a knowledge test.

IntroductionDeep Learning is a subfield of Machine Learning that is focused around creating a multitude of neural networks that are able to learn from data and predict outcomes. Deep Learning is a field that has, in recent years, become increasingly applicable in the workforce due to its many useful applications. Though the demand for deep learning experience has been increasing, there is a lack of open source courses online that individuals can use to learn the basics. So, we have created an open source course, “A Deep Learning Course with PyTorch”, for individuals everywhere to learn the basics of deep learning.

ObjectiveOur team has the objective of creating an open source course for deep learning using the PyTorch framework. We will educate our users about what deep learning is, why they should be interested in it, and what possible real-world applications it can have. We will provide resources to teach the fundamentals of linear algebra that are needed in order to implement deep neural networks effectively. We will also educate our users on what neural networks are, how they work, and how to create them using PyTorch. We will ensure that our users have the resources

6

needed so that they can understand all of the basic aspects of what goes into creating a fully functioning deep learning algorithm.

A key aspect of this course is that it is open source. Thus, anyone will have the chance to use the provided source code to its full potential, and others can add to its curriculum.

PyTorch was chosen as the framework to use because it is generally considered easier to use than the other popular deep learning framework, TensorFlow. We believe it is best to teach our users the basics of deep learning using a framework that is easier to use and understand than others, since this is supposed to be an introductory course.

DeliverablesThe deliverables for this capstone project are:

1. An open-source repository of topic documentation hosted on GitHubhttps://github.com/machinelearningmindset/deep-learning-course

2. Source code developed using Python and PyTorch

3. Comprehensive documentation on the following modules:a. Introductionb. Basicsc. Neural Networksd. Deep Neural Networks

ClientThe client for our project is Amirsina Torfi, a Ph.D. student at Virginia Tech. He is the founder of Machine Learning Mindset [17], a non-profit organization that strives to enhance how AI is leveraged in today’s world. He is a developer and researcher who has dedicated the bulk of his professional life to Artificial Intelligence and its corresponding subfields such as machine learning and deep learning. His public GitHub repositories currently receive over ten thousand monthly readers, as well as having over five thousand developers from around the world currently benefiting from them [17]. Amirsina has been personally ranked as one of the top 20 GitHub Python developers in the United States. His dedication to this field is a large part of what has inspired our team to complete this open-source project.

TeamOur team consists of the following students: Brendan Bolon, Harrison Ellis, Sam Burton, Matt Robinson, and Andrew Whelan. We are all majoring in Computer Science and have plans to graduate from Virginia Tech in the Spring of 2020. We took on the deep learning course because we wanted to learn about how deep learning differs from machine learning, and also

7

explore how deep learning fundamentals could help us in our post-graduation careers. Below there is some background information regarding each member of our team and what role they served during the completion of this project.

Sam Burton is interested in cloud architecture and cloud development, specifically on the AWS platform. He will begin his professional career working for AWS as a Solutions Architect, helping companies of all sizes to solve their cloud-related problems. He hopes to one day become an expert in the architecture of cloud computing so that he can apply his problem solving skills to this new, advanced technology that is rapidly changing the world. For the deep learning course, Sam is in charge of creating the skeleton of the GitHub repository. He has created the structure and layout of the course. He has also helped in the research and development of many individual modules.

Matt Robinson is most interested in big data analytics. After graduation, he will begin his professional career as a data scientist for IBM, where he will be using cutting edge data mining technologies to identify new patterns and make new discoveries. For the deep learning course, Matt is the team leader, which means he is in charge of submitting assignments as well as in charge of communication with the client. Matt is also responsible for research and PyTorch assistance within each of the modules.

Andrew Whelan is a student with a focus on data structures, security, and systems engineering. He will begin his professional career as a Systems Engineer at Northrop Grumman Mission Systems working with data models, research, and maintenance in the defense sector on various projects. With this experience in the defense sector, he hopes to be involved in development security operations, where cyber security is integrated in the development of new technologies. For this deep learning course, Andrew works to create PyTorch source code with Sam, assists in documentation of various concepts, and oversees the basics module in the course.

Harrison Ellis is a Computer Science student focused on Software Engineering. Upon graduating, he will start his career at Ally Financial as a Software Engineer on a Digital Channels Technology team. Previous experience applying engineering practices to the development of web-based applications will provide him with a seamless transition to this role. Along with developing client-facing software, he will also gain knowledge of automating deployment processes. Harrison’s primary responsibilities pertaining to this deep learning course include documenting concepts and creating detailed tutorials for the covered topics.

Brendan Bolon is studying Computer Science and is most interested in Software Engineering and applications development. Upon graduation Brendan will start his professional at Capital One as a Software Engineer as part of their Technology Development Program. He plans to use his background in full stack application development to help apply and develop the most cutting-edge technologies in the banking industry. Brendan’s primary role in development of the deep learning course has been creating documentation and outlining tutorials for the various topics.

8

RequirementsWorking with our client, we came up with a set of requirements that he wanted us to meet during the creation of the deep learning course. Our biggest requirement is to develop a functional and well documented open-source course that can help users learn about deep learning concepts and some of the algorithms behind those concepts. We chose at the start of the project to write all of our source code using PyTorch, which is an open source Python framework that specializes in machine learning and deep learning applications. Finally, we chose to write all of our documentation in reStructuredText (rST) [6] to stay consistent with our client’s past work on GitHub [17].

Open Source Deep Learning CourseOne of the main reasons our client wanted this course to be open source is because of the lack of material on the Internet that teaches deep learning basics. Coming into this project, our team was experienced in Python but had no knowledge about how deep learning algorithms worked. Our plan is to compile all the information we can from every trusted reference that we can find into one GitHub course that anyone can access. By providing source code and documentation that helped our team understand deep learning, we hope to provide users a way to learn basic deep learning concepts in one repository.

PyTorchOur client gave us the choice to use PyTorch or TensorFlow as our deep learning framework; we ultimately decided on PyTorch. The main reason we chose PyTorch over TensorFlow was because our client is more familiar with PyTorch and said that it would be easier for him to assist us while writing source code. He also mentioned that the learning curve for TensorFlow is much steeper and that learning it in a semester would be a lot more challenging compared to PyTorch. This was not an easy decision for us because TensorFlow is predominantly used in the workforce, but because we are new to both frameworks and only have a semester to develop code, we chose the more intuitive framework.

ReStructuredText DocumentationWe decided to use reStructuredText to document and display our course materials. We chose this framework over markdown because it was consistent with the past GitHub projects that our client has completed and also because of its close association with Python documentation.

9

DesignThis deep learning course is organized in the structure described in Figure 1. The overarching course is divided into four modules which are further divided into three to five subtopics.

Each subtopic may contain well-commented Python scripts using PyTorch and various linear algebra packages for users to read and experiment with on their own. These modules are designed to be linear for users to follow and to help them focus on material that they require. Some subtopics may include fully written programs or snippets of code to give an idea of certain functions used.

Implementation

Figure 1: Class Organization and Structure

10

This course is built entirely inside of a GitHub repository called deep-learning-course and is a part of the Machine Learning Mindset GitHub repository [17]. We have chosen to have this learning course be on GitHub so that the course itself and all files associated with it are highly available and accessible to those who are interested in diving deep into the source files. The structure of the course is made using ReStructuredText, a type of markdown language that allows for more technical implementations.

The course begins with a table of contents as well as a link to a book written by our client, “Deep Learning RoadMap - A Resource Guide”, in the case that any individual would be interested in further reading about deep learning. We then provide a brief introduction to the field of deep learning, and our motivations for creating this course [17]. As you can see in Figure 1, the course is then broken down into four modules, each containing sub-modules that pertain to a specific aspect of deep learning. We wrap up the course with a note and thanks from the creators.

EvaluationKeeping this course simple and professional is extremely important. Our code must compile and run correctly, and all information contained within the tutorials must be well formatted and correct so that our course will be credible. For this to happen, we created a system to ensure that every member of the group has conducted the research and taken proper notes when doing so before beginning to write a tutorial or code. Afterwards, our system has each member of the group read over every bit of documentation and code that is created so that no typos are missed, everything is formatted correctly and flows well, and most importantly so that all information contained within our course is correct.

We have tested our product by having a small group of our peers, who are interested in deep learning, take our course and evaluate it. The people who participated in the course were able to provide us with feedback in two separate ways. The first way that our peers have provided us with feedback is by taking a pretest before the start of our course and a post-test after the completion of our course. This feedback is important to us because we are able to see if the material we have created is getting absorbed by our peers as we intended. The second way that we have gotten feedback is through an end of survey response where our peers are able to provide us with insight on how to improve the class. From these surveys, we have been able to learn about what we did well and what we have to improve upon to make this the best course possible within our given time frame.

Finally, our client will also be reading all of our documentation and code to ensure that it is correct, and representative of how he intends the course to be.

User Manual

11

Homepage

Upon navigating to the course page, a user will be presented with a table of contents and a link to view a book written by our client as shown in Figure 2. Each link in the table of contents will direct the user to the specific position on the page where that module is located, facilitating navigation through the course.

Course Introduction

Figure 2: Homepage View

12

Secondly, users will see an introduction to the course as shown in Figure 3. They will have the opportunity to read the motivation behind why we created this course, and why we think people should be interested.

Introduction to Deep Learning Module

After reading the introduction and motivation, users will see four separate modules like the one above, as can be seen in Figure 4. Each module contains submodules pertaining to a specific

Figure 3: Course Introduction View

Figure 4: Deep Learning Module

13

area that is needed in order to be able to write a successful deep learning algorithm. Some modules only contain documented tutorials while others contain source code. The Introduction to Deep Learning module begins by containing documentation that teaches users what deep learning is, why it is important, and different applications for it. To see the tutorials, users will simply click on the word, “Tutorial”. Once they have clicked, they will be redirected to a new page such as the one in Figure 5.

What is Deep Learning?The “What is Deep Learning” page gives a brief background on deep learning and goes into

supervised learning and the different types of data on which to train deep learning algorithms. This section is divided into 5 subsections, Background, Supervised Learning, Structured Data, Unstructured Data, and Neural Networks as shown in Figure 5.

Why Deep Learning?

Figure 5: What is Deep Learning Page

14

The “Why Deep Learning page” as shown in Figure 6, explains why deep learning is important and has been gaining popularity in recent years. This page specifically highlights three main reasons: Data, Advancements in Computation, and Innovation in Algorithms.

Applications of Deep Learning

Figure 7: Applications of Deep Learning Page

Figure 7 shows the “Applications of Deep Learning” page explains some of the different real-world and current applications of deep learning.

Figure 6: Why Deep Learning Page

15

Basics Module

The basics module as shown in Figure 8 contains content for basics of linear algebra that users will need to know in order to understand how deep learning works. The sub-topics of this module are logistic regression, derivatives and the chain rule, and gradient descent. These tutorials are also accessed by clicking on the blue, “Tutorial” link.

Logistic Regression

Figure 8: Basics Module

16

Figure 9 shows the top of the Logistic Regression Page. The logistic regression documentation page begins with a brief description introductory paragraph and is followed by links to the three separate sections. The notation section explains some commonly used logistic regression notation. The next section explains the sigmoid function and the final section shows how cost functions can be used to assess the sigmoid function and thus train parameters on data.

Derivatives and The Chain Rule

Figure 9: Logistic Regression Page

17

Figure 10: Derivations and Chain Rule Page

The tutorial page shown in Figure 10 discusses some basic calculus and how it is used in deep learning. The section’s primary purpose is to provide some clarifications about information discussed in the gradient descent section.

Gradient Descent

18

The gradient descent tutorial page (see Figure 11) begins with a visual representation of the concave cost function, J(w, b). The page provides examples of how gradient descent works on high and low levels, and also how it pertains to machine learning.

Simple Neural Networks

Figure 11: Gradient Descent Page

19

Figure 12: Neural Network Architecture Page

The module on neural networks architecture is the first time in this course that users will be able to attempt writing code based on steps provided in this module. The module begins by giving users basic information about neural networks such as what they are and what their architecture looks like (see Figure 12).

The module then provides the steps (see Figure 13) that should be taken by the users in order to write their own simple neural network, with pictures of code provided. The source code is also available under the label “code” for users to look at a completed simple neural network.

20

Figure 13: Neural Network Documentation

Once this module has been completed, users are able to navigate to the next module using a link at the bottom of the page.

21

Simple Logistic Regression

Figure 14: Logistic Regression Documentation Page

The simple logistic regression tutorial page (see Figure 14) gives an in-depth breakdown of the logistic model that we trained. This model takes input from an online PyTorch dataset that contains images of handwritten numbers (ranging 0-9) and outputs its prediction of what number is being represented. The example output displayed at the end of the documentation shows the loss statistics as the model gets more accurate after hundreds of iterations.

22

Neural Network Module

Figure 15: Neural Network Module

Figure 15 shows the outline of the Neural Networks Module. This module explains what neural networks are and how they are used in deep learning. This module starts off with a brief introductory overview of the module. Computation is the next section, followed by the activation function section, which explains what activation functions are and how they are used between layers of the neural network. The last section is backpropagation which explains how neural networks are modified to get more accurate results. All of the tutorials can be accessed by clicking on the tutorial links.

23

Overview

Figure 16: Neural Network Overview Page

The overview tutorial (see Figure 16) gives an explanation about what a neural network is and how it fits into deep learning. The tutorial also shows how a neural network is represented graphically and in code.

24

Computation

Figure 17: Neural Network Computation Page

Figure 17 shows the Computation Page. This computation section describes how to compute the final value for a neural network on a specific training item. This computation is also called forward propagation.

25

Activation Functions

Figure 18: Activation Functions Page

Figure 18 is the Activation Functions page which outline various activation functions that can be used in the creation of a neural network. This section shows all the different activation functions and explains how activation functions are used. The activation function section also explains why ReLU is the most widely used activation function.

26

Backpropagation

Figure 19: Backpropagation Page

The backpropagation section (see Figure 19) describes how to calculate the derivatives of the cost function working from right to left across the network.

27

Simple Neural Network Classifier

Figure 20: Neural Network Classifier Page

Figure 20 shows the Neural Network Classifier Page which outlines a simple neural network classifier code. Users will be able to follow the implementation of the code step by step and learn what each function does on a sample CIFAR10 dataset, a popular dataset that can be imported into Python. This section is inspired by a PyTorch tutorial found online [10] but will go further into detail with every concept used in the tutorial.

28

Figure 21: Neural Network Classifier Code Documentation

Figure 21 shows the step by step organization of the Neural Network Classifier page.

29

More on Deep Neural Networks Module

Figure 22: Deep Neural Networks Module

The “more on deep neural networks” module (see Figure 22) gives a more in-depth overview of neural networks. The first two sections and last section of the module explain the different strategies to fix the problem of over-complex neural networks. The third section, Gradient Setbacks, discusses problems and limitations associated with gradient descent. The tutorials for each subject can be accessed by clicking on the tutorial link next to the topic.

30

Regularization

Figure 23: Regularization Page

Figure 23 shows the Regularization page that explains the problem of overfitting and how regularization can be used to fix it. In this section, the main two methods for regularization are explained.

31

Dropout

Figure 24: Dropout Page

Figure 24 shows the Dropout Page. This section explains the dropout method and how it can be used to solve overfitting for overly complex neural networks.

32

Gradient Setbacks

Figure 25: Gradient Setbacks Page

The gradient setbacks section (see Figure 25) discusses issues and limitations that come with gradient based deep learning. The section also discusses recommended improvements for the setbacks.

33

Batch Normalization

Figure 26: Batch Normalization Page

The tutorial on batch normalization (see Figure 26) discusses how to improve performance and results on networks with many hidden layers. The standardization methods are described as well as the overall effects of using batch normalization.

Final Notes and CitationsThe page ends with directions on how to submit a pull request for the GitHub repository, a note from the creators, and citations for the course.

34

Developer ManualThe developer manual gives aspiring developers insight on how we created our Deep Learning Course repository and ways that they can continue the work we have completed.

Course LayoutAll of our content is developed in Python and reStructuredText. We chose Python as our programming language because it is very easy to learn and has extensive deep learning libraries, such as PyTorch. We decided to use reStructuredText as our markup language because it offers a wide array of markup features for Python development and is consistent with our client’s past projects. Below we have listed some examples of reStructuredText documentation that we used within our repository:

Figure 27: rST Paragraph

Figure 27 shows a reStructuredText format of paragraphs. Paragraphs written in reStructuredText require no specific markup to differentiate them from other elements in a document. They can be separated by headings to make them easier to read, but to add onto a section of the repository, a developer can simply keep writing directly onto the end of a given section.

Figure 28: rST Code Block

Code blocks differ slightly from paragraph blocks in reStructuredText (see Figure 28) but are still fairly easy to grasp. Instead of using images to represent code within our documentation, we used a built-in feature of reStructuredText to display the code. This is done by indenting the code segment that you would like to display beneath the text line “.. code:: python”. After

35

doing this, you have a nicely highlighted code block that fits seamlessly into the documentation, as seen in Figure 29.

Figure 29: rST Code Block on Page Example

Figure 30: rST Embedded Links

We use embedded links all throughout our repository for a number of reasons. In the section above, we have links to three references and then a link to the next section of the course titled “Why Deep Learning?” (see Figure 30). These links can also be used in the midst of a paragraph which is a very useful quality of reStructuredText. The code is displayed in Figure 31.

Figure 31: rST Embedded Block on Page Example

Figure 32: rST Table

36

Figure 32 shows the structure of tables in reStructuredText. Tables are used on the homepage of our course, with one for each module. The primary purpose of these is to keep the sections organized. The tables that can be seen in Figure 33 make it easier for the user to know what a module covers and how to move through it in an orderly fashion.

Figure 33: rST Table on Page Example

Contribution

37

Figure 34: Pull Request Process

Figure 34 shows the page on the pull request process. This process for contributions to the course are described in the Pull Request Process section of the homepage.

MethodologyList of possible users and goals that our system supports

38

Persona Goal

Students: People pursuing a degree in a computer related field. Often use online tutorials to learn more. Students could also be people in the workforce who have graduated from university but would still like to learn deep learning.

1. Learn the linear algebra needed to write effective deep learning algorithms

2. Write a neural network algorithm in PyTorch for deep learning.

3. Learn the various applications of Deep Learning

Teachers: Individuals who are teaching classes in a computer related field. They will see the importance of keeping up to date with new, groundbreaking technologies.

1. Help students experiment in writing their own deep learning algorithms in PyTorch, and possibly add to the course content

2. Give feedback to Course CreatorsTable 1: Example Persona and Goals

Table 1 contains example personas and their goals to view our system on the user side.

Break down of each goal into units of tasks and subtasks

39

Student: Teacher:

41

Table 2: Tasks and Subtasks of Goals for Personas

Table 2 shows steps to achieve a goal and its subtasks on the user side. This is outlined in the creator and system side in the following section implementation based service for each attack.

Implementation based service for each taskStudents Goal 1: Learn the linear algebra needed to write effective deep learning algorithms

Sub-task: Navigate to course page

Inputs: Link to course, Web search/GitHub search

Outputs: Deep learning course page

Libraries, Functions, & Environments: Web browser, GitHub

Sub-task: Visit linear algebra overview documentation

Inputs: Link provided to linear algebra documentation from course page

Outputs: Detailed documentation of linear algebra concepts

Libraries, Functions, & Environments: GitHub

42

Sub-task: Read our breakdown of linear algebra fundamentals

Inputs: Documentation of linear algebra concepts

Outputs: Students understanding essential linear algebra concepts

Libraries, Functions, & Environments: GitHub, Coursera Deep learning Course, Practical Linear Algebra for Machine Learning By: Amirsina Torfi

Sub-task: Make connections to basic deep learning concepts

Inputs: Knowledge of basic linear algebra concepts and basic deep learning concepts

Outputs: Students realizing the math behind deep learning fundamentals

Libraries, Functions, & Environments: GitHub, Coursera Deep Learning Course, Practical Linear Algebra for Machine Learning By: Amirsina Torfi

Students Goal 2: Write a neural network algorithm in PyTorch for deep learning.





Sub-task: Scroll to Table of Contents

Inputs: User scroll

Outputs: Table of Contents section

Libraries, Functions, & Environments: Web Browser, GitHub

Sub-task: Click on Neural Network

Inputs: User click

Outputs: Neural Network section on GitHub page


Sub-task: Click on Neural Network Architecture Documentation

Inputs: User click

Outputs: Neural Network Architecture Documentation page

43

Libraries, Functions, & Environments: Web Browser, GitHub, reStructuredText hyperlink/layout

Sub-task: Read Course Material

Inputs: Course Documentation

Outputs: Student understanding/reference

Libraries, Functions, & Environments: N/A

Sub-task: Click on Python Code

Inputs: User click

Outputs: Basic Neural Network python code GitHub repository

Libraries, Functions, & Environments: Web Browser, GitHub, reStructuredText hyperlink/layout, PyTorch

Students Goal 3: Learn various applications of deep learning

Sub-task: Navigate to open source Deep Learning Course




Sub-task: Surf course and click “Applications” sub-topic

Inputs: Link to Applications documentation

Outputs: Documentation of relevant Deep Learning Applications

Libraries, Functions, & Environments: GitHub, Coursera Deep Learning Course, Journal/web references

Sub-task: Read page about applications

Inputs: Documentation of Deep Learning Applications

Outputs: Knowledge about different uses of deep learning

Libraries, Functions, & Environments: GitHub, Coursera Deep Learning Course, Journal/web references

Sub-task: Click links to references at bottom of page to further explore more deep learning applications

44

Inputs: List of references used in research

Outputs: More data and applications that can be further studied/analyzed

Libraries, Functions, & Environments: GitHub, journal/web references

Teachers Goal 1: Help students experiment with writing their own deep learning algorithm in PyTorch

Sub-task: Help students with any questions they have when writing they own deep learning algorithms

Inputs: Students questions related to their code

Outputs: Successfully implemented deep learning algorithms written by students using PyTorch

Libraries, Functions, & Environments: PyTorch

Sub-task: Encourage students to use information in modules to write deep learning algorithms

Inputs: Users viewing individual module tutorials

Outputs: Students initial attempts at PyTorch deep learning algorithms

Libraries, Functions, & Environments: PyTorch

Sub-task: Tell students to navigate and learn about each individual module

Inputs: Tutorial page with links to each module

Outputs: Students learn the basics of deep learning


Sub-task: Provide students with course link

Inputs: Students clicking on the link to the course

Outputs: Students have access to the course


Teachers Goal 2: Give feedback to Course Creators



45



Sub-task: Scroll to Table of Contents

Inputs: User scroll

Outputs: Table of Contents section


Sub-task: Click on Contact Us

Inputs: User click

Outputs: Contact Us Section

Libraries, Functions, & Environments: Web Browser, GitHub, reStructuredText hyperlink/layout

Sub-task: Click on email hyperlink: [email protected]

Inputs: User click

Outputs: User proprietary e-mailing system

Libraries, Functions, & Environments: Web Browser, GitHub, user email server, reStructuredText hyperlink

Sub-task: Write email and send to Course Creators

Inputs: User email/message

Outputs: sent email to course creators

Libraries, Functions, & Environments: Web Browser, GitHub, user email server

46

Set of workflows

Figure 35 shows the complete workflow diagram for the student and user side along with the system design to supplement how our course assists in achieving user goals.

Lessons Learned

Timeline/ScheduleDue to the nature of our project and team, an immense amount of research was and is required. We created our schedule to ensure that we had allocated 2-3 weeks to create each module. We cycled through a 2-3 week process of research, compilation of notes, experimentation with code, implementation/testing of code, and writing documentation. Table 4 shows the “due date” or milestone date in the first column, topics covered in the second column, and action taken on said topics in the last column. Table 3 outlines the cycle of our process of research and implementation.

Date Topics Covered Action Taken

02/14/2020 - Linear Algebra Review/PyTorch Review, - Machine Learning Course Review

Research, Notes

02/21/2020 - Introductory Module- Basics- Neural Networks- Deep Neural Networks

Research, Notes

Figure 35: Workflow Diagram

47

- PyTorch- GitHub Page Setup

02/28/2020 - Introductory Module: What, Why, Applications

Documentation

03/06/2020 - Basics Module:- Logistic Regression- Derivatives + Chain Rule- Gradient Descent

Research, Code Implementation

03/13/2020 - Basics Module:- Logistic Regression- Gradient Descent

Code Implementation

03/20/2020 - Basics Module:- Logistic Regression- Derivatives + Chain Rule- Gradient Descent

Documentation, Module Organization in GitHub page

03/27/2020 - Neural Networks:- Overview- NN Architecture- Representation

Research, Documentation, Code experimentation

04/03/2020 - Neural Networks:- Computation- Activation Functions- Backpropagation

Research, Code Experimentation

04/10/2020 - Neural Networks:- Overview- NN Architecture- Representation- Computation- Activation Functions- Backpropagation

Documentation, Code Implementation

04/17/2020 - Deep Neural Networks:- Regularization- Dropout- Gradient Setbacks- Batch Normalization

Research, Code Experimentation

04/24/2020 - Deep Neural Networks:- Regularization- Dropout- Gradient Setbacks- Batch Normalization

Documentation, Code Implementation

04/26/2020 - Testing/Peer Reviews Evaluation

Table 3: Schedule

48

ProblemsThe biggest problem that we have faced so far this semester is the communication difficulties we have faced due to COVID-19. Prior to the shutdown of Virginia Tech, we met weekly with our client and met as a team twice a week on campus to work on the course. After the shutdown, we have done most of our client communication through Zoom and Slack. Zoom serves as a decent alternative to our weekly meetings, but it makes it hard for our client to help us with our coding problems. In terms of our team meetings, we still try and meet up twice a week, but it is definitely more challenging to make progress compared to when we could utilize campus resources.

A second problem that we faced was the learning curve we all dealt with when trying to understand deep learning concepts and reStructuredText. Deep learning algorithms are very powerful, but they are not easy to totally grasp, so we had to take a lot of time researching and practicing different techniques. We were also all new to reStructuredText, so it took some time before we could start exporting all of our documentation onto the course webpage.

The last major problem that we faced was deciding how we should balance our time between coding and documentation. This course is designed to help teach users deep learning basics. That requires a lot of documentation on how each algorithm is designed to work. With that being said, providing relevant source code is a difficult task that requires a lot of time and critical thinking.

SolutionsWe tried to address all of the communication issues that we faced due to COVID-19 by being more active on Slack and by making sure that we communicated over slack about meeting upcoming deadlines. It has taken a lot more planning and out of class conversations to make sure that we are on track, but that is the only way to ensure that we are completing our work in a timely manner.

In terms of dealing with the steep learning curve associated with deep learning, we all took the first four weeks of the project auditing a Coursera Deep Learning Class [13]. The videos and readings provided by this class gave us the basic understanding of deep learning that we needed to dive right into creating the course. Specifically, it gave us a lot of information that we could include in our documentation and a good base to start writing some source code. More research was definitely needed outside of this course, but it was very helpful in balancing the learning curve. Learning reStructuredText was new, but learning it was not nearly as difficult. It took some time but once we all had a feel for it, it became second nature.

When deciding how to balance our time, we made the decision toward the beginning of the project to spend 60% of our time working on documentation and 40% of our time writing relevant source code. The reason we decided to divide our time up like this is because our client emphasized that our biggest objective is to provide helpful, well written explanations of deep

49

learning concepts. The code portion of our course serves more as an example of how these concepts are used practically.

SolutionsWe tried to address all of the communication issues that we faced due to COVID-19 by being more active on Slack and by making sure that we communicated over slack about meeting upcoming deadlines. It has taken a lot more planning and out of class conversations to make sure that we are on track, but that is the only way to ensure that we are completing our work in a timely manner.

In terms of dealing with the steep learning curve associated with deep learning, we all took the first four weeks of the project auditing a Coursera Deep Learning Class [13]. The videos and readings provided by this class gave us the basic understanding of deep learning that we needed to dive right into creating the course. Specifically, it gave us a lot of information that we could include in our documentation and a good base to start writing some source code. More research was definitely needed outside of this course, but it was very helpful in balancing the learning curve. Learning reStructuredText was new, but learning it was not nearly as difficult. It took some time but once we all had a feel for it, it became second nature.

When deciding how to balance our time, we made the decision toward the beginning of the project to spend 60% of our time working on documentation and 40% of our time writing relevant source code. The reason we decided to divide our time up like this is because our client emphasized that our biggest objective is to provide helpful, well written explanations of deep learning concepts. The code portion of our course serves more as an example of how these concepts are used practically.

Future WorkOur team has now completed all four modules and their corresponding subtopics, written source code where it was needed, used reStructuredText to create the format and layout of the course, and completed extensive research pertaining to the topic of deep learning. To continue expanding upon this course following graduation, more coding tutorials will be added to enhance the quantity and quality of content that is included in the course for our users. Since this course is open source, contributions will be taken from any individual who believes they are able to contribute good content. The GitHub course page will continue to be maintained and regularly updated with any information the team feels is necessary to include in the course.

50

AcknowledgementsWe would like to acknowledge our client, Amirsina Torfi:

- Ph.D. Student at Virginia Tech- Top 20 GitHub Python developer in the U.S. by GitAward- Top 100 GitHub Python developer in the world by GitAward- Founder of Machine Learning Mindset: www.machinelearningmindset.com- Email: [email protected] GitHub: https://github.com/astorfi

51

References[1] Anon. 2018. 4.7. Forward Propagation, Backward Propagation, and Computational Graphs Colab Open the notebook in Colab. (2018). Retrieved April 25, 2020 from http://d2l.ai/chapter_multilayer-perceptrons/backprop.html

[2] Anon. 2019. ReLu. (May 2019). Retrieved April 7, 2020 from https://deepai.org/machine-learning-glossary-and-terms/relu

[3] Anon. 2019. torchvision.datasets. (2019). Retrieved April 25, 2020 from Anon. 2019. https://pytorch.org/docs/stable/torchvision/datasets.html#mnist

[4] Artem Oppermann. 2019. Deep Learning Academy. (October 2019). Retrieved April 25, 2020 from https://www.deeplearning-academy.com/p/ai-wiki-regularization

[5] Asad Mahmood. 2019. Logistic Regression on MNIST with PyTorch. (April 2019). Retrieved April 25, 2020 from https://towardsdatascience.com/logistic-regression-on-mnist-with-pytorch-b048327f8d19

[6] Eli Bendersky. 2017. reStructuredText vs. Markdown for technical documentation. (February 2017). Retrieved April 7, 2020 from https://eli.thegreenplace.net/2017/restructuredtext-vs-markdown-for-technical-documentation/

[7] Elvis. 2020. A Simple Neural Network from Scratch with PyTorch and Google Colab. (February 2020). Retrieved April 25, 2020 from https://medium.com/dair-ai/a-simple-neural-network-from-scratch-with-pytorch-and-google-colab-c7f3830618e0

[8] F D. 2017. Batch normalization in Neural Networks. (October 2017). Retrieved April 25, 2020 from https://towardsdatascience.com/batch-normalization-in-neural-networks-1ac91516821c

[9] Jason Brownlee. 2019. 14 Different Types of Learning in Machine Learning. (November 2019). Retrieved April 7, 2020 from https://machinelearningmastery.com/types-of-learning-in-machine-learning

[10] Jason Brownlee. 2019. A Gentle Introduction to Batch Normalization for Deep Neural Networks. (December 2019). Retrieved April 25, 2020 from https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/

[11] Jason Brownlee. 2019. A Gentle Introduction to Dropout for Regularizing Deep Neural Networks. (August 2019). Retrieved April 25, 2020 from https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/

52

[12] Jürgen Schmidhuber. 2014. Deep learning in neural networks: An overview. (October 2014). Retrieved April 7, 2020 from https://www.sciencedirect.com/science/article/abs/pii/S0893608014002135

[13] Ng, Andrew, et al. “Deep Learning by Deeplearning.ai.” Coursera, http://www.coursera.org/specializations/deep-learning

[14] Originally published by Suryansh S. 2020. Gradient Descent: All You Need to Know. (April 2020). Retrieved April 25, 2020 from https://hackernoon.com/gradient-descent-aynk-7cbe95a778da

[15] Ritchie Ng. 2020. Logistic Regression with PyTorch. (2020). Retrieved April 25, 2020 from https://www.deeplearningwizard.com/deep_learning/practical_pytorch/pytorch_logistic_regression/

[16] Sagar Sharma. 2019. Activation Functions in Neural Networks. (February 2019). Retrieved April 6, 2020 from https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6

[17] Torfi, Amirsina, et al. “Machine Learning With Python.” GitHub, Machine Learning Mindset, 1 Dec. 2019, http://github.com/machinelearningmindset/machine-learning-course

[18] Vikashraj Luhaniwal. 2019. Why Gradient descent isn't enough: A comprehensive introduction to optimization algorithms in... (May 2019). Retrieved April 25, 2020 from https://towardsdatascience.com/why-gradient-descent-isnt-enough-a-comprehensive-introduction-to-optimization-algorithms-in-59670fd5c096

[19] “Training a Classifier.” Training a Classifier - PyTorch Tutorials 1.5.0 Documentation, pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html

vtechworks.lib.vt.edu · web view2020-05-14 · deep learning course. final report. instructor:...

Documents