Digital Media: Why Do We Need It, and How Do We Teach It

as REAL Computer Science?

Jennifer BurgDepartment of Computer Science

Wake Forest UniversityWinston-Salem, NC 27109http://www.cs.wfu.edu/~burg

burg@wfu.edu

This material is based on work supported by the National Science Foundation

under Grants No. DUE-0127280 and DUE-0340969.

04/19/23 2

Overview of Curriculum Material:

•Primer*•Computer Science •Art*

*Primer and Art books are being written byDr. Yue-Ling Wong, Wake Forest University.

Three companion books to bepublished by Prentice-Hall, withon-line learning supplements.

04/19/23 3

Computer Science Material

04/19/23 4

Curriculum Material Creates a Bridge Between Digital Media as Computer Science

and Digital Media As Art

• Primer (Yue-Ling Wong) – foundational material for computer scientists and artists; tied more to practice, but introducing the science behind the tools in a “gentle” way

• The Science of Digital Media (Jennifer Burg) – for computer science majors; focuses on the science and mathematics behind digital media tools like Photoshop, Audition, Illustrator, Premiere, Flash, and Director

• The Art of Digital Media (Yue-Ling Wong) – for digital art students

04/19/23 5

Components of The Science of Digital Media

• Textbook• Interactive tutorials• Worksheets• Programming exercises• MATLAB mathematical modeling exercises

(all currently available on the web at

http://digitalmedia.wfu.edu)

04/19/23 6

• All material is freely available on the web at http://www.wfu.edu.

• Answer sheets to worksheets and implemented programs are at a password-protected site.

• This talk will focus on The Science of Digital Media.

04/19/23 7

How I Use the Material• A course called Digital Media covers Chapters 1 through 5 and is taught to

computer science majors (juniors and seniors). – Pre-requisites are Calc I, CS1 and CS2, and data structures.– Chapters 1 through 5 cover digital imaging and digital audio.– Digital video is introduced in the course (briefly).– In class, we cover the science and mathematics of digital media.– “Behind the scenes,” students do a game program as their semester-long term

project, putting together digital photographs, vector graphics digital sound, and interactivity using Macromedia Director and its scripting language Lingo.

– Students are expected to learn “hands-on” things on their own.– There is a pre-test and post-test for each chapter.– Students also do worksheets and on-line tutorials. (Note that in this course, we

do not do the programming exercises.)

• Second course called Digital Video and Animation, does an overview of central concepts of Chapters 1 and 5 and then focuses on Chapters 6-8, which cover digital video and multimedia programming.

04/19/23 8

Software Needed for the Courses Described on Previous Slide

• Image processing (e.g., Photoshop)• Vector graphics (e.g., Illustrator or Freehand)• Digital audio processing (e.g., Audition or Sound Forge)• Video processing (e.g., Premiere or Final Cut)• Multimedia programming (e.g., Director of Flash) (I use

Director).• Optional -- A programming environment (e.g., C++ or

Java), if you choose to do the programming assignments.

• Optional – MATLAB or Mathematica for mathematical modeling exercises. (If your students don’t have access to MATLAB or Mathematica, but YOU do, you can do the mathematical modeling exercises as in-class demos.

04/19/23 9

Alternative Ways to Use Computer Science Curriculum Material

• Teach a course in Algorithms for Digital Media and focus on the programming assignments.

OR• Focus on only one medium per course (digital imaging,

audio, or video).OR• Cover all of the book (The Science of Digital Media) in an

undergraduate course, but not in complete depth. Then use the entire book again in a graduate course and go into more depth in the mathematics and science.

04/19/23 10

Discussion:Are courses in Digital Media suitable for a

rigorous computer science program?

Is Digital Media REALLY computer science?

Yes! Not only is it computer science, but it’s a more interesting way to engage the students’ interest in science.

04/19/23 11

Imagine• You are students.

• I’m your teacher teaching you about two-dimensional arrays.

“Two-dimensional arrays are data elements stored in contiguous blocks of memory, and blah blah blah…”

Scribble scribble scribble on the board…

04/19/23 12

OR…I can show you what you can do with two-

dimensional arrays.

Image dithering

04/19/23 13

04/19/23 14

What can students learn in this exercise?

• Two-dimensional arrays, matrix operations• Dynamically vs. statically allocated arrays• Array/index vs. pointer/offset notation• File I/O• Not to mention

– Loops– Control structures– Function call and return– Etc.

04/19/23 15

And there are other digital media topics

that deal with two-dimensional arraysConvolutions and unsharp mask

1 -3 1

1

1

- = -1 5 -1

-1

-1

2*original - blur mask = unsharp mask

2

04/19/23 16

Imagine• You are students.

• I’m your teacher teaching you about trees.

“A tree consists of nodes. A non-empty tree has a root node, and n child nodes, where n >= 0, and blah blah blah…”

Scribble scribble scribble on the board…

04/19/23 17

OR…I can show you what you can do with a tree.

Octree algorithm for indexed color

04/19/23 18

indexed color by Photoshop our own implementation of indexed color

04/19/23 19

With some fine-tuning of the algorithm, we can do even better.

This version has more color gradient.

04/19/23 20

Octree Algorithm for Indexed ColorWhen the algorithm reaches a point where it wants to create a 257th leaf node

This is what an image looks like when the node with the least children is collapsed. Note that there is more color detail in the sky.

This is what an image looks like when the node with the most children is collapsed.

Photographs courtesy of Kristian Damkjer, who used this curriculum material at the University of Florida in the fall of 2005.

04/19/23 21

What can students learn in this exercise?

• The tree data structure

• Recursion and recursive descent of a tree

04/19/23 22

Students also can be motivated to learn mathematics through digital media

Show how colors can be modeled mathematically and understood as occupying a graphical space.

Black

Green Yellow

RedCyan

Blue Magenta

White

Blue (0,0,1) Cyan (0,1,1)

Magenta (1,0,1)

Red (1,0,0) Yellow (1,1,0)

Green (0,1,0)A

B

04/19/23 23

Students can be given a motivation for learning logarithms

Human perception of sound intensity is proportional to the logarithm of the air pressure amplitude.

This is why we use decibels to measure sound. Decibels are defined in terms of logarithms.

04/19/23 24

We can capture the students’ interest in fairly complex mathematical operations – e.g., the Fourier transform.

04/19/23 25

In short, relating concepts to images, sound, motion, and interactivity motivates students to learn.

04/19/23 26

Where does digital media* currently fit in the computer science curriculum?

• Not included in ACM’s list of major subdisciplines within computer science – computer science– information systems– software engineering– computer engineering– information technology

(See http://www.acm.org/education/curric_vols/CC2005_Final_Report2.pdf.)

*I’ll use the term “digital media” to denote a field of study and thus will consider it a singular noun. (Just didn’t want you to think I don’t know my grammar. )

04/19/23 27

Where does digital media currently fit in the computer science curriculum?

• In Computer Science Computing Curricula 2001, not included as one of the essential courses or areas within traditional computer science

– i.e., architecture, graphics, databases, operating systems, algorithms

– The closest thing is “Graphics and Visual Computing,” with some elements covered in Net-Centric Computing, Algorithms and Complexity, and Programming Languages

– Seehttp://www.computer.org/portal/cms_docs_ieeecs/ieeecs/education/cc2001/cc2001.pdfandhttp://www.cs.wfu.edu/~burg/papers/BurgWongStrokSIGCSE2004.pdf

04/19/23 28

Where does digital media currently fit in the computer science curriculum?

• Digital media courses in computer science departments exist mostly as electives.

• Digital media is sometimes covered superficially in CS100-type courses.

• Digital media also appears as – an interdisciplinary field of study– a course or courses within disciplines outside of

computer science (e.g., art, communications)– a department all its own

04/19/23 29

Why do we need digital media?

•Enrollments are falling in computer science.

•Even students who like working with computers may perceive computer science as irrelevant and detached from the REAL world of computers.

•We don’t always create a good link between science and practice. Digital media is an ideal area in which to make this connection.

04/19/23 30

Why have computer scientists been reluctant to include digital media in the

computer science curriculum?

• It seems inherently interdisciplinary. – (Well, that’s one of its charms!)

• It seems SOFT! – (Really, it’s NOT! Consider the topics in the

computer science curriculum material at http://digitalmedia.wfu.edu.)

04/19/23 31

Combining the science of digital media with the practice of digital media

• Emphasize the science in classroom teaching, readings, worksheets, etc.

• In “the background,” students are working with application programs such as Photoshop, Audition, Premiere, Flash, Director, and/or Javascript.

• I have my students do a game program as their term project, developed incrementally as they study each medium.

• Another good assignment is an on-line resume/porfolio.

04/19/23 32

Just in case you wondered: Why do I use Director instead of Flash

for on-line tutorials?

• Director is more powerful than Flash in the following ways:– Better support for video formats.– Better support for sound.– More Xtras.– Allows for launching of external applications.

• Director now allows you to use Javascript syntax as an alternative to Lingo.

• See http://www.macromedia.com/software/director/resources/integration/flash/quick_comparison.html

04/19/23 33

After the break

• Do tutorials and exercises

• Do a mathematical modeling worksheet

• Discuss ways to use the material

• Discuss your own curriculum needs

• Talk about good projects and assignments

• Talk about whether digital media really should be in computer science

04/19/23 34

The Challenge

• Link science to practice

• Show that knowing the science results in better practice

• Get scientists and artists communicating and collaborating on creative projects