once more, with feeling

O n c e m O r e , w i t h f e e l i n gTHOUGHTS & EvidEncE FROM THE MakinG OF MOTiv

Russell Maschmeyer, Interaction Design MFA Candidate, 2011 | School of Visual Arts

Russell MaschMeyeR. OncE MORE, wiTH FEElinG.

00 CONTENTS01 INTrOduCTION02 ExplOraTION03 INvESTIgaTION04 rESEarCh05 EvOluTION06 ExECuTION07 rESulT08 CONCluSION09 ThaNk yOu10 rEFErENCES

0 0

Russell MaschMeyeR. OncE MORE, wiTH FEElinG. INtroductIoN • 5

I’m kind of cheap. I don’t spend much on clothes or housing or vacations. I don’t own a car. But I do spoil myself with one, somewhat bizarre hobby: I collect vintage audio recording equipment. I have what amounts to a small, but professional stu-dio tucked away in a closet and a small encyclopedia’s worth of sound engineering know-how tucked away in the nooks and crannies of my brain. I collect because I’m fascinated by the tools and the processes of music making. Music is magical stuff.

Collecting musical technology has taught me—and this is why music is fascinating to me—that musical magic isn’t born of the newest technology. There’s as much magic in a Daft Punk performance as in a Sam Cooke a cappella. What makes mu-sic magical is the way it can be bent toward expression.

I knew while preparing (mentally/physically/emotionally) for the thesis year, that I had to find a project that would combine my love of music, process, and technol-ogy. I wanted my thesis to be something I might never have been able to do outside the safe walls of my graduate studio environment. I wanted to make something real, that people could experience first hand. I wanted people to feel just a little bit of magic. I wanted jaws to drop. I wasn’t thinking small.

INTrOduCTION

0 1

Russell MaschMeyeR. OncE MORE, wiTH FEElinG. ExploratIoN • 7

I kept a pocket journal during the early part of my second-semester. On the cover I wrote “Thesis Ideas” and jogged down every idea that came to mind. It was in-formal, but useful. The second entry reads: “making music -> pattern & deviation, surprise.” It was the beginning of what Steven Johnson, author of Where Good Ideas come From, would refer to as my slow hunch.

Over the course of the summer and into the following semester I explored the purpose of pattern and variation in music. Through reading, thinking and writing I realized I was chipping away at much a larger truth: music and physical movement are inextricably linked. Music instinctively drives us to move, and movement drives our instinctive understanding of music. Our brains are hard-wired for it.

Then I turned my attention to instruments. I began considering the strengths and weakness of both digital and traditional instruments when judged against this need for connection between movement music. When mastered, traditional in-struments become extensions of the musician’s body, allowing musicians to ex-press themselves through movement. New digital instruments seem to lack the same level of connection to physicality. I wanted to find some way to bring a higher degree of expressiveness to digital instruments.

ExplOraTION

0 2

school of VIsual aRts. MFa in inTERacTiOn dESiGn.8 • ExploratIoN

they weren’t born that way. they didn’t get that way by accident,

either. they got that way by chronic, repeated abuse. that’s

not a digital problem, that’s a physical problem. It’s still about

an industrial system that cruelly sacrifices human flesh for the

sake of dysfunctional machinery. they sit, they type, they stare

in screens. all day, every day.”

Bruce sterling, shaping things

Of course we do this to ourselves for a reason. The computer gave us so

many opportunities to work abstractly, to expand beyond the physical, to

create a document that wastes no paper and can be edited with little to

no effort, to model reality. We encased these functions in a form that fit

the means of the time: screen, keyboard, mouse. Back then, technology

couldn’t be expressive. Since it did the job well enough, we stuck with it—

for 50 years now.

“Initially I thought it might be interesting to do something musical, but I hesitated. I didn’t want to concoct a new instrument interface…”

But technology has increased on exponentially since then, and we have

the capability to do so much more. We have the ability to create, touch

and manipulate virtual spaces and objects in ways only dreamed of a few

years ago, in ways that seamlessly blur the distinction between “IRL” ob-

jects and those of the virtual world. We should celebrate and honor our

evolution by finding new and more physical ways to handle the digital. We

should find ways to de-abstract our computational world in ways that still

carry the benefits of virtualization.

So, in a nutshell. That’s what I want to do.

FiRST THOUGHTS On a THESiS, and EvOlUTiOn

Though work won’t begin until next semester, a lot of

us here at chez SVA IxD have been racking our brains all

semester to come up with some brilliant thesis ideas.

Slow going at first, but my creative juices got a jump-start down in Savan-

nah during IxD ’10. There were so many brilliant ideas on display it was

hard not to come away with a few of your own. Like the effect of hearing a

faint tune in the distance and interpreting your own original melody from

the fractured bits you hear. That’s happened to you guys too, right?

Initially I thought it might be interesting to do something musical, but I

hesitated. I didn’t want to concoct a new instrument interface. I’ve seen

that done a lot (see: Tenori-on, Otto, and or draw your own), and while that

would be fun and fascinating in its own way, it felt too obvious. I want this

project to guide my career for at least a few years to come. So I wanted a

broader scope.

I’ve been thinking about evolution. After hundreds of thousands of years,

we’ve become highly physical beings: gangly arms, upright posture, joints

and bones that withstand long arduous walks, teeth that can cut and

grind nearly anything, hands that can perform millions of coordinated,

nuanced manipulations. We’re amazing physical beings. Yet, we spend all

of our time sitting, staring into projected light computer terminals, han-

dling virtual objects with a “digital finger” that has less nuance than my

pinky toe. We punch 78 or so keys repeatedly to “talk” to friends or to

“compose” a piece of music. We’re not using our bodies, and so they are

failing us. Bruce Sterling, in his book Shaping Things, puts it pretty well:

“…the heavy duty programmers…are commonly portly guys with

wrist supports, thick glasses and mid-life heart attacks.

B l O g p O S TMARch 29, 2010

{Above} Bruce Sterlin’s Shaping Things was (and is) and incredibly important book to me as an interac-tion designer and lover of technology. It brings to light some of the ways our technological existence is misaligned with our evolution as a species.


field notes

Page one of my ‘Thesis Ideas’ notebook. The second entry, ‘making music -> pattern & deviation, surprise,’ turned out to be strangely prescient.


EvERy ExTEnSiOn

I’m continuing to work my way through Adam Green-

field’s Everyware, an amazing book that continues to

blow my mind.

In “Thesis 43” Greenfield quotes Marshall McLuhan (who incidentally has

been named the patron saint of Wired Magazine). Marshall coined the

terms “global village” and “the medium is the message” as well as wrote

Understanding Media: The Extensions of Man in 1964, his seminal and

most widely known work. Anyhow, here’s a great sum up from the Wiki-

pedias:

“Mcluhan’s insight was that a medium affects the society in

which it plays a role not by the content delivered over the me-

dium, but by the characteristics of the medium itself. Mclu-

han pointed to the light bulb as a clear demonstration of this

concept. a light bulb does not have content in the way that a

newspaper has articles or a television has programs, yet it is

a medium that has a social effect; that is, a light bulb enables

people to create spaces during nighttime that would otherwise

be enveloped by darkness. he describes the light bulb as a me-

dium without any content.”

Wikipedia, “Marshall Mcluhan”

So that’s all some intense background just to get to a great idea. Green-

field quotes McLuhan from Understanding Media. McLuhan brilliantly

points out:

“Every extension is [also] an amputation”

B l O g p O S TApRIL 16, 2010

Amazing. here we are, a society hell bent on extending our reach through

phones, through computers, through “seamless integration” and yet all

along the way we’re unwittingly losing perhaps as much as we gain. The

mediums we create are built to carry out specific tasks efficiently, but by

doing so they have a tendency to restrict our options for accomplishing

that task by other means. We begin to learn the “One” way to do it, when

in fact there are infinite ways. The medium begins to restrict our thinking,

our imagination, our potential.

The extent to which a certain medium is adopted to perform other tasks,

it begins to restrict those as well. This is exactly the case against com-

puters. We’ve shoved a lot in there and not all of it fits very well. The tool

has begun to ill-fit the tasks. Our attentions are now paid so often to how

to learn a new piece of software or hardware. Seems like a lot of wasted

energy. This leads me to principle Number One of my thesis project:

principle I: Focus on the task, not the tool

Who knows how many principles I’ll have by the end of next year. But that

feels like a good start to me.

{Above} Adam Greenfield’s every-ware: the Dawning age of ubiquitous computing was a source of inspira-tion, driving me to reconsider the ways we shape our technological ecosystem and awakening me to the amazing work in the field of ubiquitous com-puting.

{below} Marshal McLuhan’s under-standing Media is essential reading for anyone interested in media, technol-ogy, or design.


field notes

Another page from my ‘Thesis Ideas’ notebook. As I contin-ued to try and escape my innate desire to work in the musi-cal space, my ideas became more and more disjointed.


{left} This statement was written after only a few weeks contemplating my interests. It’s surprising how little I’ve deviated from the core belief: that physical motion is essential to cre-ating expressive music.

cOndUcTinG in THE BOxan early thesis statement

Musicality is the intersection between movement and

sound. Music and dance have been bound to each other

since before our species can remember itself, and the

two cannot be split without a loss of efficacy. They are

two sides of the same phenomenon—like electricity

and magnetism. Musicality is motion. Motion is musi-

cality. To say it plainly: without an understanding and

fluency in motion, one cannot be musical.

A computer’s interface is generally dumb to motion; therefore, it cannot

understand or engender musicality. Sure, people make fantastic music

on computers, but we wage an interface battle with computers in order to

encode our inherent musicality into an inhospitable environment. Even so,

more and more musicians and music professionals on both the young and

experienced ends of the spectrum have been turning to the computer to

inspire, create, and mix their music. It’s called “mixing in the box.” I have

to think hard to find a less appealing phrase.

conductors were the first audio engineers. With a deftly rising, falling, and

swooping hand accompanied by a metronomic wand they controlled what

listeners heard. Volume, speed, dynamics, character, and sometimes

content. They shaped raw music into experience, into motion by using

nothing more than motion. Modern engineers do it with mixing boards,

pan pots, reverb units, delays, flangers, EQs, compressors, expanders,

and—God, save us—computers. In order to interface with a single piece of

music, an audio engineer may stare at and tweak hundreds of disparate,

clunky, asynchronous interfaces, each one a dozen times over.

I’d like to investigate the role gesture might play in educating computer-

ized musical engineering interfaces as to the natural motions of music.

possible applications include:

01 humanizing a beat sequencer to generate a more dynamic “per-

formance” and a deeper kinetic response to computer generated

beats.

02 Re-imagining the interfaces of a mixing engineer to perform

more like those between conductor and orchestra

03 Approaching audio effects such as compression, EQ, reverbs, de-

lays, et al using gestural sculpting

There have been inroads to educating computers about motion in mu-

sic. Wii Music is a great example of an interface that allows for a playful

physicality to be returned to a previously arranged piece of music as the

participants control dynamics through motion activated Wiimotes. Tod

Machover’s Media Lab group, Opera of the Future, is creating richly physi-

cal instruments which can be played intuitively through gesture and other

types of physical computing. A central theme in Tod’s work is breaking

down barriers for non-musicians with unusual instruments so that they

can begin creating rich music without formal training.


Thesis Preparation Liz Danzico Feedback: April 26, 2010

For Russ Maschmeyer This is an exquisite pursuit, one that is rich with possibilities to explore. As an area to pursue, it’s both personal and pragmatic, progressive and has legacy. I’d be enthusiastic to see this progress as a thesis project. Some people and considerations: Control: Both a conductor and a computer relegate the control to one or a series of pre-defined individuals. In addition to motion, might you be considering democratizing the process of composition to a larger set of consumer or composers? Learning curves: What might the learning curve be for these types of new interfaces? At the intersection of game and movement, is some Tetris-related research: Epistemic Action increases with skill (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.51.3279&rep=rep1&type=pdf) Kars Alfrink at Leapfrog has been exploring the intersection of interaction design, music composition, and game design. One of the references I see online is to this “opera” that uses large-scale urban games and an opera he composed (http://leapfrog.nl/blog/archives/2009/05/29/announcing-a-hybrid-game-opera-for-monster/). While not at the interface level, Kars has been an interaction designer with some depth, and might be an insightful resource for you. Amit Pitaru (http://www.pitaru.com/) whose work has been focused on, primarily art, the work of music and interactive experiences. See also bodies as instruments: “How Bodies Matter: Five Themes for Interaction Design” (http://hci.stanford.edu/publications/2006/HowBodiesMatter-DIS2006.pdf)

{RIGHt} As usual, a thorough and thoughtful response from Liz Danzico to my early thesis statement.

ExploratIoN • 13Russell MaschMeyeR. OncE MORE, wiTH FEElinG.

From June 1st through August 30th of 2010, I took a time-out from New York and set up shop in San Francisco. I was interning at Ap-ple, an amazing experience allowing me to explore a new field without have to throw my entire being into the work. After the first year of SVA Interaction Design, I definitely needed to recoup some extra mental bandwidth. So, despite working forty hour weeks and being three-thousand miles away from Jessica, it was still a relaxing summer.

San Francisco was beautiful and I’ve always enjoyed lonely adventures in strange plac-es. I could have squat-thrusted the weight of a buffalo after walking all those hills.

During these three months I began think-ing in depth about my career goals and how my thesis project for the coming year might fit in. I knew that I wanted to spend my life working on technologies that honor human evolution. I want to create experiences that strengthen our physical and mental development, while driving social progress. I realized that digital instruments suffer from a lack of fluency in human motion. When I put two and two together, I began to see this thesis focus as a microcosm of a larger technologically systemic issue.

SuMMEr SOMEWhErE ElSE

14 • ExploratIoN school of VIsual aRts. MFa in inTERacTiOn dESiGn.

While in San Francisco, I spent each weekend morning at Boogaloo’s, a small cafe on Valencia Street in the Mission District. I’d order a heart-healthy breakfast like juevos rancheros or eggs benedict with bacon on the side and I would dive into Levitin’s this Is your Brain on Music.

Call me a pusher, but if you’re interested in music on any level, get this book. It took the main stage in shaping my thesis concepts—not to mention my thoughts about music and human evolution on the whole.

Fascinating on all accounts.

a MaJOr INFluENCE



“a pianist may play five

notes at once and make one note only slightly louder than the others, causing it to take on an

entirely different role in

our overall perception of

the musical passage.”

“Musicians generally

agree that groove works best when it is not strictly metronomic—

that is, when it is not

perfectly machinelike.”

“Groove has to do with a

particular performer or a

particular performance,

not with what is written on paper.”

“the genius of his playing

is that he keeps us

on our mental toes by

changing aspects of the pattern every time he plays it, holding just

enough of it the same to

keep us grounded and

oriented.”

There are enough insights in this Is your Brain on Music to launch a thousand the-sis projects, but the key insight for me was that our neurological interest in music is rooted in expecation and deviation. We build up libraries of music in our brains and we begin to expect certain patterns in the music we listen to at both a macro and micro level. Music only gets interesting to us when a performance deviates in a controlled way from our expectations.

ExprESSION IS BOrN OF varIaTION

PaGE 72 PaGE 170 PaGE 171 PaGE 171

This passage, spanning page 210-211 in This Is Your Brain on Music was one of the most influential for me. The concept of a hidden, primal link between physical-ity and musical expression—an inborn, physical understanding—was incredibly powerful.

Moreover that musical expertise can take both a technical and an emotional form was central to my attempts at separating the two. Without this passage I might have ended up somewhere very different.

MOvEMENT & MuSIC


Howdy Russ!

Welcome back to the East Coast (albeit it briefly).

I was chatting with Jessica last night about SVA stuff. They’ve inquired whether I would like to be an advisor, which I think would be fun - but with the right advisee. I enjoyed our brief conversation earlier this year, and wanted to pitch myself as a potential advisor, if you think I’d be a good match for whatever you have planned.

That is all!

Best, nf

E - M a I lFROM: NIchOLAS FELTONdaTE: AUGUST 6, 2010TO: RUSSELL MASchMEYER

18 • ExploratIoN school of VIsual aRts. MFa in inTERacTiOn dESiGn.


I had a lot of questions about what traits were important in an advisor when it came time to choose. Was the most important thing that my advisor be an expert in the field I was exploring? I found it helps, but it’s not the most important thing. Was it important my advisor have some celebrity to potentially launch my project into the limelight? Actually, that would have been a negative if he/she was too in demand to meet regularly. Was it important my advisor have a specific technical knowledge? I found a technical understanding was important, but technical skill wasn’t paramount. After all, my advisor wouldn’t be building my project.

The important thing, to me, was connecting with someone I respected, but most im-portantly: someone I liked and someone who liked me too. You can find technical as-sistance in a lot of places, but it’s much harder to find a collaborator. That’s the role your advisor should play.

Luckily, Nicholas Felton made this decision easy on me. He asked me if I’d like to be his advisee. I like and respect Nick a lot. So I said ‘of course.’

I did seek the help of a few others over the course of my thesis year including Robin Bargar (who really is an expert in the field I’m exploring) as well as Larry Legend. Not to mention thesis instructors Jenn Bove and Paul Pangaro (it really does take a vil-lage). A lot of great people contributed valuably to my thesis project, but not in the way Nicholas Felton did by being there every week, genuinely interested in the prob-lem I was trying to solve and the way I was going about solving it.

Hats off to Nicholas.

www.feltron.com

ON ChOOSINg aN advISOr


ExTEnSiOnS & MaSTERy

Interesting article today on Scientific American con-

cerning our use of tools and the level to which our brain

can assimilate tools as parts of the body. This is some-

thing I’ve thought a lot about as a musician and Inter-

action Designer.

The article provides a good jumping off point for a new thesis principle I’d

like to share:

principle II: The most widely mastered tool is the human body.

For the most part, the instruments we have accrued developed out of

historical context and technological constraints. The acoustic guitar,

the drum set, various percussive sticks and boxes and tubes, the violin,

the piano, etc. have all—after a certain point—stopped progressing with

technology. The sound of the instrument and the physical form that pro-

duces that sound became cherished tradition.

Most of us have picked up an acoustic guitar at least once. Maybe some-

one showed you how to play a chord. You pressed down, but the strings

hurt your fingers and when you strummed you heard more muted plunk-

ing than you heard notes. If you were stubborn, you picked it up at least a

few more times and figured out how to make the chord sound good. If you

were really stubborn you learned all the other chords, grew a few calluses

on your fingers, and maybe even strummed a few songs you still remem-

ber. If you’re the <1% of the population who reaches musical mastery,

you picked up that guitar and didn’t put it down until your fingers were as

lingual as your tongue.

“humans, and some other animals, are able to use tools as

additions to the body. When we use a long pole to retrieve an

object we couldn’t otherwise reach, the pole becomes, in some

sense, an extension of our body.”

patrick haggar & MattheW r. longo

So, to the issue of mastering these extensions; making them part of our

bodies. If you’re mastering a back-scratcher, it probably won’t take you

10,000 hours. Mastering a guitar on the other hand could take you years;

the piano a lifetime. These hurdles drive a lot of people away from creat-

ing and participating in music. But there may be a secret to bringing them

back: movement. Nearly everyone dances. Why? Because it comes natu-

rally. There’s no learning curve.

“First, from the brain’s perspective, the body is by far the most

familiar object in the world: the body, as William James elegantly

put it, is ‘always there.’

patrick haggar & MattheW r. longo

I’d like to try and bring these musical extensions we’ve created back into

the body. Maybe I’ll fail. Maybe the problem is just too difficult. Maybe to

be truly expressive and musical with any tool, including your body, you

must spend years obtaining a certain level of mastery. But if the body re-

ally is the most widely mastered tool, certainly it’s got to shave some time

off that 10,000 hours.

ApRIL 22, 2010

B l O g p O S TSEpTEMBER 7, 2010

{left} My first post of the Thesis Develop-ment semester. This article drove ideas about the body being played as its own instrument, as long as someone could build a computer sys-tem to sufficiently interpret it.

{RIGHt} According to Malcolm Gladwell (out-liers), it takes 10,000 hours of experience for someone to become an expert, music notwith-standing. My hope was that transforming the body into an instrument might shave off a few of those hours, since any grown person is al-ready a master of their own physicality.

10,000Hours of practice required to become an expert musician.


Russell MaschMeyeR. OncE MORE, wiTH FEElinG. INvEstIgatIoN • 23

I was naïve. I assumed that I was the first to discover a link between music, move-ment and gestural interfaces. I was spectacularly wrong.

I stumbled on more research papers, experiments, instruments, art projects and exhibitions in the first month alone than I thought possible in a year of searching. A few simple Google searches gave me enough reading to blind me. It was humbling. Before I had even started I felt left in the dust.

All I could do was dive in, but quickly my worries evaporated. I realized many of these projects addressed different problems or solved similar problems for a dif-ferent audience. In the evidence from research papers and one particular art piece, David Rokeby’s A Very Nervous System, I found validation that my hunches con-cerning computer vision, motion, and music, were well founded.

After the release of Microsoft’s Kinect there was an explosion of gestural instru-ment projects. I spent the better part of two months a nervous wreck, expecting someone to beat me to my thesis concept. Though there was a lot of amazing work, I never saw anything that approached the problem quite the same way.

INvESTIgaTION

0 3

school of VIsual aRts. MFa in inTERacTiOn dESiGn.24 • INvEstIgatIoN

THE BOdy, THE BRain & MUSic

So far I’ve been collecting a lot of anecdotal research

and snippets from various books and online sources.

Here are a few of the things currently inspiring me.

A passage from Dr. Daniel Levitin’s book This Is Your Brain on Music:

“Pitch is so important that the brain represents it directly; unlike

almost any other musical attribute, we could place electrodes

in the brain and be able to determine what pitches were being

played to a person just by looking at their brain activity”

daniel levitin, THiS iS yOUR BRain On MUSic

he takes it a step further. This brainwave translation of sound happened

so far back in evolution most living creatures have the same reaction to

pitch and sound that we do. he describes an experiment performed on

owls:

“Petr [Janata] placed electrodes in the inferior colliculus of the

barn owl, part of its auditory system. then, he played the owls

a version of strauss’s “the Blue Danube Waltz”… Because the

electrodes put out a small electrical signal with each firing—and

because the firing rate is the same as the frequency of fir-

ing—Petr sent the output of these electrodes to a small ampli-

fier, and played back the sound of the owl’s neurons through a

loudspeaker. What he heard was astonishing; the melody of

“the Blue Danube Waltz” sang clearly from the loudspeakers…”

daniel levitin, THiS iS yOUR BRain On MUSic

I mean, come on! holy shit, right?

conclusion

I guess what I’m saying is… can’t we do both of these things once? create

a total brain/body instrument and composition tool? I’ll leave you with a

thought from David Rokeby’s 1985 essay, Dreams of an Instrument Maker:

“the unique abilities of these instruments seem to propose a

new approach to music, which would use to full advantage their

unique potential. By this I do not mean the invention of new

structural systems, or complex mathematical tunings, but a

renovation of the relationship between music, the composer,

and time itself.”

david rokeBy, dreaMs of an instruMent Maker


{left} Thanks to Ian Curry and Jake Barton of Local Projects for turning me on to David Rokeby’s work. In a lot of ways I’m glad I found out of David now, be-cause his work was, in many ways, what I imagine.

{left} Mick Grierson has created a computer pro-gram that can literally detect and play the note you think. Amazing! Hit the one-minute mark for a quick thought about using such a program to not only de-termine notes, but to use the brain as an orchestral conductor of those notes; a more brain-centric take on the main idea I’m trying to get at through my study of body motion.


Mapping it out

The idea for MOTIV begins to come together as I plan system flows and potential expressive parameters like velocity and note duration. Though I was still searching for other poten-tial directions.


a FlOOd OF RESOURcESUp to this point I’ve been a little

afraid of doing my prior art/litera-

ture research. The nagging fear

in the back of my mind has been,

“Well, what if someone has already

done it?” My Thesis Development

Instructor, Jenn Bove of Kicker

Studio, gave me a swift kick in the

pants by confirming that, yes, in

fact, people have been working in

this space for a long time.

In order to create a project worth anything, I’ll

need to perform a lot more research and figure

out why my thesis would be different or better

than what has come before. So this week I did

just that. So much to show and I’ve only begun

to search.


gestural Music organiZations:

• NIME: New Interfaces for Musical Expression conference

• Gesture controlled Audio Systems Research collective

• Moprim: Search for Upper-Body Motion primitives

• center for computer Research in Music and Acoustics

acadeMic papers:

• Force Feedback controlled physical Modeling Synthesis

• Multimodal Analysis of Expressive Gesture in Music and Dance

• Gesture control of Music

• Slide Guitar Synth with Gestural control

• Instrument Augmentation Using Ancillary Gesture for Sonic Effects

• Audio-Input Drum

• communication of Musical Gesture Using the AES/EBU Audio Standard

• Gesture Sound Experiments

• Symbolic Objects in a Networked Gestural Sound Interface

• Interactive Sonification of Emotionally Expressive Gestures

• Instrumental Gestures and Sonic Textures

• Looking at Movement Gesture: Drumming and percussion

• Gesture and Morphology in Laptop Music performance

• On Development of a System for Gesture control of Spatialization

• Incubator: A Gestural Audio Tool Using Glove Interface for Live perfor-

mance Mixing

• Gesture-controlled physical Modeling Synthesis with Tactile Feedback

• Toward an Affective Gesture Interface for Expressive performance

• Gesture control of Sounds in 3D Space

• A Wii-Based Gestural Interface for computer conducting Systems

• The Sound of One hand: A Wrist-Mounted Bio-Acoustic Fingertip Ges-

ture Interface

• Effect of Latency on playing Accuracy of Two Gesture controlled con-

tinuous Sound Instruments without Tactile Feedback

• Gesture control of Singing Voice, a Musical Instrument

• Soundstudio4D – A VR Interface for Gestural composition

• In Search of the Motion primitives for a communicative Body Language

• On the choice of Gestural controllers for Musical Applications: An Eval-

uation of the Lightning II and the Radio Baton

internet resources

• NIME 2011 | Oslo, Norway

• Gestural Music Sequencer

• Aggregat – Gestural Ableton Sequencer

• The continuum Fingerboard

• Airpiano: Gestural Music controller

• Gestural control of Music Using Vicon 8

• InfoMus Lab, Italy

• hand Gesture controlled Music player

• Mouse & the Billionaire :: Gesture control Exploration

• Keith price Bibliography Music Related Gestures Systems

• Gesture controlled Musical Instruments

• Moprim

• conGAS

• String Thing

• The center for computer Research in Music and Acoustics


{top left} Aggregat, a multi-touch gesture-based digital audio workstation for sound production in ableton live 8. The interface allows up to three people to create music collaboratively in one single set

{top RIGHt} AirPiano is a gestural musical inter-face that controls virtual instruments by the placing of hands above the board. Distance and x-axis place-ment are measured to apply pitch and volume.

{left} Gestural Music Sequencer is a sequencer that uses computer vision to alter the pitch and vol-ume of the musical output according to the x and y position of the tracked object, usually a light source.


nEw inSPiRaTiOn: idMilI found the Input Devices and Music Innovation Lab (ID-

MIL) in Montreal. They are an amazing source of mate-

rials and information. Marcel Wanderley runs the group

at McGill U, and seems to have generated about 80% of

literature on the subject of gestural music interfaces.

In fact, he literally wrote the book on my thesis topic,

New Digital Musical Instruments: Control and Interac-

tion Beyond the Keyboard.

While this example (at right) is slightly more nutty less consumer focused

than I hope my thesis will be, this is the kind of stuff that makes me feel

like I picked the best topic on earth.

B l O g p O S TOcTOBER 2, 2010

{Above} Immortal-machine for méta-instrument from D. Andrew STEWART on Vimeo. Totally nutty and esoteric performance work. But, God, how awesome is that robo-instrument suit?

{left} New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. I bought this book, but never managed to get around to reading it. One day …


Wrong directions

Notes from early in my development process. At this point I was still kicking around potential ideas within the music space and trying to determine the angle of my approach. For a short while I considered creating a swarm intelligence system for music.

school of Visual arts, MFa in Interaction Design


Subject: Holy Shit!

From: Nicholas FeltonTo: Russ Maschmeyer

haha!

I saw that on your twitter.

Still leaves you lots of room to play

From: Russ MaschmeyerTo: Nicholas Felton

It may even enable me to push more real-time melodic generation and sequenc-ing back into the gestural arena instead of confining it to the screen. Lots to think about.

E - M a I lFROM: RUSS MASchMEYERdaTE: OcTOBER 27, 2010TO: NIchOLAS FELTON

{left} ‘Quick Intro to SoundPrism’ on You-Tube.com. This little app blew my mind by re-ducing music making into simple pattern play-ing.

SoundPrism strips away the need for explicit musical knowledge like note names, musical modes or chord construction and simply lets the user play patterns that always seem to sound good.

Brilliant design.


presentation notes

Notes from two early presentations in the semester. I had a long battle trying to figure out who my audience or market was. I could never tell if this thing I was going to build would be best for knowledgeable musicians or the average person. I’m still torn.


nEw RadicalS

new radical person: chris o’shea

B l O g p O S TDEcEMBER 10, 2010

chris O’Shea demonstrates OpenNI after it was open-sourced by

primeSense. OpenNI is the driver that allows skeleton tracking (the white

lines inside the silhouette).

chris O’Shea demonstrates his Air Guitar prototype, using Kinect. I felt as

if I was in a bit of a race against the clock when I saw this video a whole

month before I would even being constructing my prototype.

new radical organization: openni


Motion primitives

Early sketches of potential gestural control systems for mu-sic. Beginning with a theramin-like concept on the far left and migrating toward more high-level control parameters like tempo on the right.


Clive Thompson (@pomeranian99)3/14/11 5:40 PM

Cool @danlevitin study: Variations in timing of a piano perfor-mance more important than loudness in emotional impact: http://bit.ly/htq7Jv

E - M a I lFROM: NIchOLAS FELTON daTE: MARch 14, 2011TO: RUSS MASchMEYER

{left} As if I didn’t have enough respect for this man already, Daniel Levitin published this research in February of 2011, showing the two main performance parameters that impart ex-pression are timing and note velocity.

I had arrived at the same conclusion only a few months earlier, without any scientific evidence to back it up. Levitin’s research vindicated my con-clusions and allowed me to rest easy, knowing I had chosen to put users in control of the right parameters.


a Working Model

By the start of the Thesis Presentation semester, I had ar-rived at the above simplified system model. I proposed cre-ating the gestural engine and the visual feedback display. Every other piece had already been developed by the market.

Russell MaschMeyeR. OncE MORE, wiTH FEElinG. rEsEarch • 37

I love interviews. There’s something really pleasant about sitting down with some-one and knowing what you’d like to talk about or to have them invite you into their world for a moment so you can see how they do things.

Over the course of just a few weeks I conducted three ethnographic interviews and three standard interviews with a mixture of producers and musicians. I asked each of them the following questions:

What interfaces/devices/programs/instruments do you use?

how do you go about creating a new musical idea?

how comfortable are you with leaving things to chance?

What do you find challenging/annoying about current interfaces?

What could be improved?

What do you appreciate about current interfaces?

What do they make effortless?

When synthesizing sounds, what parameters do you find yourself tweaking most often?

Describe the process you went through to learn these interfaces.

A very clear pattern emerged: both musicians and producers love digital instru-ments because they change the ways they approach and think about music. New interfaces or modes of interacting are inspiring. But they also struggle against these tools. They tweak, abuse, and otherwise trick them into producing more ex-pressive, human performances.

rESEarCh

0 4

ChuCk BrOdyProFESSIoNAL ProDUCEr


“I think people are

experimenting in more of

a live feel now, trying to

figure out how to bring

that classic live feel back

into electronic sounds

these days.”

“I like to switch my

process up and use

different sounds and

different techniques

and interfaces because

it takes me in different

directions and makes my

stuff different.”

“I worked with another

producer for a while

who loved to cover

the computer screen

because so many people

just watch the screen as

they work and they look

for things to be wrong

visually.”

“once you’re too focused

on making everything

perfect it gets to be a

little lifeless I think.”

{below} Chuck uses Ableton, a digital audio appli-cation, to compose entire songs. The interface works like a band rehearsal for him by allowing him to put in different phrases which he can turn on or off at any given time.

{below RIGHt} Chuck’s Akai drum machine. Tap-ping on pads to the right plays drum samples from

memory. Expression is limited to the audio samples in the library.

{RIGHt} Chuck’s audio effects rack including com-pressors, equalizers, delays, reverbs and preampli-fiers.

BrIaN CaSSagNOlHoME STUDIo ProDUCEr


“you find yourself doing

funny technique things

to get it to sound more

real and rewriting parts

to sound more real.

you have to write to the

limitations”

“My low budget setup

here is about beating the

computer. technology

makes it so that I can

access things that were

impossible to access

20 years ago, but it

also funnels you into a

computer-y sound.”

“they have these

‘humanizer’ settings on

drum plugins and it’s

like… just being random

doesn’t humanize it. you

play off of the grid in a

very specific way.”

“the interesting thing

to me: the whole

technology boom with

music opened up music

creation on this level as

a possibility to millions

of people who just never

would have had access

or talent or drive enough

to have gotten there.”

{below} Brian works entirely on the computer with digital plug-in effects—software versions of the equipment in Chuck’s effects rack.

{below RIGHt} An audio channel plug-in designed to recreate the sound of a vintage recording console. {RIGHt} Brian’s entire studio toolset consists of a keyboard controller with some drum pads, head-

phones and his computer. Using today’s virtual in-struments, the small keyboard is all Brian needs to perform what would traditionally require an orchestra.

42 • sEctIoN school of VIsual aRts. MFa in inTERacTiOn dESiGn.

Olga BEllELECTroNIC MUSICIAN


“I love that these things

teach you how to think in

an entirely different way

about music.”

“It’s the best feeling in

the world when you can

command something,

a piece of equipment

or program, enough to

make it really do what

you want.”

“you know what I think

would be terrific? If

everything was wireless.

you’d think by now

everything would be

wireless but look at all

these fucking chords!”

“the reason we added

another person in the

band is so that we can

minimize how much we

have to play to a track.

so that we can retain our

integrity as people who

play instruments.”

{below} Olga enjoys using unconventional instru-ments like the keytar, a throwback synthesizer from the 80s that recasts the keyboard in the form factor of a guitar.

{below RIGHt} A sampler, similar to a drum ma-chine. Olga records her own samples and then trig-gers them by tapping on the red buttons.

{RIGHt} Olga during a performance. As both the lead vocalist and keyboardist she’s tied to cumbersome, immobile instruments during performances.


alEx FEdErTrADITIoNAL MUSICIAN


“For me it either grooves

or it doesn’t. It either

makes you bob your head

or it doesn’t make you

bob your head. there’s

an undeniable thing

about that. Part of this

is totally illogical. that’s

why it’s going to be

difficult.”

“teachers never talked

about groove and feel

and time. It was always

like “learn these scales

learn these chords.” No

one talks about this stuff.

When you hit a certain

point you’re like… ‘oh,

this is what matters.’”

“that’s almost your

motto, this focuses not

on what you’re playing,

but how you’re playing

it.”

“It’s easy when you’re

playing to just feel what’s

supposed to happen.

It’s a conversation and

it’s a live conversation

that you’re not thinking

about.”

{below} Alex playing guitar while using his feet to switch on effect units (stompboxes) at his feet. Gui-tarists are some of the most mobile musicians.

{below RIGHt} Alex as lead singer of The XYZ Af-fair, using hand gestures as a means of communica-tive expression.

{RIGHt} Alex (right) on tour as a guitarist for Enrique Iglesias.

Russell MaschMeyeR. OncE MORE, wiTH FEElinG. EvolutIoN • 47

It’s a bit silly to suggest that there was an ‘evolution’ stage in this year-long pro-cess. Every one of those three hundred and sixty-five days brought with it at least one new evolution. Nevertheless, I’ve tried to consolidate the major mood swings my project had during the Thesis Development class. Remarkably, I strayed little from my original hunches concerning pattern, deviation, and the power of gesture to add intuitive control to expressive parameters.

I’ve included diagrams which illustrate my framing of the problem space, the pro-totypes that led me to the final concept as well as a few personal epiphanies here and there.

This collection of materials ends just past the beginning of spring semester, as I change direction one last time, and pick up a second advisor, Robin Bargar, and organize my goals for the execution of my prototype, MOTIV.

EvOluTION

0 5

school of VIsual aRts. MFa in inTERacTiOn dESiGn.48 • EvolutIoN

a MilE MaRkER

problem

current digital music technology has little to no understanding

of motion; therefore, it cannot understand or engender musical-

ity.

opportunity

Gesture recognition technology has created new relationship

opportunities between musicians and digital music interfaces.

these new, exciting relationships allow for a more direct con-

nection between the convenience of computer music creation

and the musical nuances of human pattern recognition and

improvisation.

After some initial research and exploration, I wanted to synthesize some

of my findings. First, I wanted to re-examine my assumptions that a

purely gestural interface could or would trump any previous interface ap-

proaches. The more I thought about why gestures were so great, I also

began discovering why solid or tangible interfaces were great. So I started

making a list (opposite page).

Some thoughts about the current interfaces being used for computer

music began to stick out. I realized there were three predominate inter-

faces: keyboards, samplers, and sequencers. If you plot them (subjec-

tively) across two matrices, ease of entry and expressivity, you’ll notice a

linear progression occurring.

The keyboard, modelled after an analog instrument, is the most expres-

sive, but also incredibly difficult to master. Take anyone who has mastered

a keyboard; chances are they first mastered a piano. Drum machines are

easier to pick up, though still difficult to master.

The easiest interface to pick up belongs to the sequencer and it’s various

iOS & other touch screen variants. You simply need to move objects or flip

switches and a pattern of samples begins to play regardless of whether

you touch anything else or not. Tempo and pattern length are easily se-

lectable. This provides people a simply entry point to begin playing around.

Even non-musicians can generate something interesting after only a few

minutes of tinkering. The problem, however, is that these interfaces are

extremely metronomic and inhuman sounding. The samples are always

played on a grid, and adjusting envelope or volume dynamics on the fly is

very difficult if not impossible.

“Why not combine the simplicity and learnability of sequencers with the dy-namic power of physical gesture?”

This gave me an idea. Gesture is potentially great at setting dynamics

for both time, envelope and volume. Why not combine the simplicity and

learnability of sequencers with the dynamic power of physical gesture?

This would provide a solution that is both highly learnable and expressive.

Moreover, having a compact interface like an ipad as a base, users can

sketch ideas from anywhere and bring them into the gestural environ-

ment—which could be a studio or a live performance stage—and add the

dynamics layer on top of the base system.

B l O g p O S TOcTOBER 25, 2010

{left} The first iteration of my in-strument landscape, mapping ex-pressivity against the ease of learn-ing the instrument.


picking sides

At first I had grand illusions about gesture being the solution to all music interface ills, but after thinking and exploring a little bit I discovered that tactile & touch interface have their own strengths as well.


In October, two presentations into Thesis Development,

I realized I had a problem. Not everyone in the classroom

had a musical background. Yet, here I was, presenting

my potential directions and ideas as if everyone knew

the difference between a chord and key. It left many of

the students and faculty scratching their heads and

turned Q&A sessions into music theory 101. Anti-good.

I knew I had to find some way to communicate my ideas with-out losing the audience on the ramparts of musical jargon. So I set about making a short set of organizing definitions, using plain language that I would slip into my next presenta-tion.

I started at square one by creating a simple organizing prin-ciple:

“Music is made of patterns.”

I played the class a clip of a sequenced drum beat. It was a simple, repeating rock beat. The kind you’ve heard in a thou-sand rock songs. It sounded like music, but it also sounded a bit dull.

Then I elaborated:

“Interesting music is made of changing patterns.”

I played another beat. This one started out the same as the last, the pattern changed partway through. People perked up, and it was pleasing to see the recognition on my audi-ence’s face that it was suddenly a more interesting beat when a change was introduced.

It wasn’t enough to discuss music at a high level. I had grand plans of making an instrument, so I needed to say a few

a FaIlurE

TO COMMUNICATEMUSIc IS MADE OF pATTERNS

INTErESTING MUSIc IS MADE OF CHANGING pATTERNS

words about the craft of playing an in-strument: musicianship.

The way I see it there are two sides to musicianship. To be a great musician, you’ve got to embody both.

The firsT side To musicianship is Tech-

nique. Technique is all the stuff you learn in early music lessons. It’s both a physical understanding of how to pro-duce sound from an instrument as well as a basic understanding of musical rules. It’s knowing what a scale is and how to play it. It’s knowing that if you string together a particular pattern of notes you get Mary had a little lamb

while another string of notes gives you Pachabel’s canon in D. It’s being able to play those notes quickly enough so that someone listening could pick out the tune. Technique is knowing what to play.

Computers are masters of technique. You tell a computer what order and how fast to play a set of notes and it will perform perfectly every single time.

On the other hand, people struggle with technique. It takes most of us months or years before we stop making major mistakes regularly when trying to play. The practice required to achieve good


technique is a major hurdle for people who are interested in playing music.

The second side To musicianship is ex-

pression. If technique is knowing what to play, expression is knowing how to play it. Great musicians add subtle variations in loudness and timing while performing a piece. These subtle ex-pressive variations are a vehicle for the performer’s emotions, crafted in the moment of performance. They’re improvisations, and the emotions they convey are readily deciphered by even the least musically knowledgeable among us.

Computers cannot generate expres-sion in music because they have no understanding of the emotional signif-icance of music. They can add random variations, but they can’t improvise with meaning.

On the other hand, people are fantas-tic at expressive improvisation; even non-musicians. Musical expression is just like dancing. It’s a fundamentally human understanding of how music leads to movement and how move-ment carries emotional meaning.


TwO PROTOTyPE METHOdS

This week I set out to draw up a prototyping plan and

coordinate some further interview research for my the-

sis. The prototyping plan outlines a few specific areas

of investigation I’d like to explore and some practical

methods for exploring them. There are two levels of in-

teraction that I’m looking at: the sequencing interface

and the gestural expression layer.

Each layer requires different prototype approaches. The sequencing in-

terface, arguably the most well-trodden layer, is logic-based. There’s a

task at hand and likely a common process for accomplishing it. It has a

tactile interface; there’s something to look at and touch. But there may

be a lot to learn about the sequencing approach from people who haven’t

dealt much with sequencing before. paper prototyping will definitely be

the best candidate there.

The gestural layer may be less adherent to a strict process or logic. I’m

hoping it has much stronger links to reaction and improvisation. call and

response. What paul pangaro would call a “conversation.” There is no tac-

tile interface. This makes prototyping a little trickier. My working theory

is that I can use master musicians to simulate the gestural system. By

having them react to participant movements (speed up, play harder, etc.)

I can roughly suss out how people might want to interact with a musical

system that adds expression dynamics based on movement.

paper prototype Questions

• Where do users want to start?

• Do users know how to start?

• how do users envision construction taking place?

• how can the sequence grow in complexity?

• Is there a common understanding of what can be constructed?

• how much do people know about beat basics?

gestural prototype Questions

• What elements do people feel add expressivity?

• What are the things that people feel that want to change via gesture?

• how do people want to signal tempo change?

• … groove change?

• … legato/glissando vs. new note?

• … harmony?

• … vibrato/tremolo?

• … note value?

B l O g p O S TNOVEMBER 1, 2010


As part of my third presentation I

refined my instrument chart to il-

lustrate a simple point: There’s an

opportunity for digital instruments

to become expressive if they utilize

gesture the way traditional instru-

ments do.

Typically, Instruments fall into one of two cat-

egories: digital or traditional. Traditional in-

struments are acoustic on some level. Even an

electric guitar relies on vibrating strings to pro-

duce a tone. Digital instruments rely entirely on

integrated circuits and digital to analog con-

version to produce sound. There’s little to no

gestural input.

The technique required to play a digital instru-

ment is about as low as it can be. If you know

how to flick a light switch or turn a knob, you’re

golden. On the other hand, traditional instru-

ments require far more technical knowledge to

operate. Imagine picking up a guitar or an oboe

for the first time. It’s not entirely evident how

you’re supposed to play it. There’s almost al-

ways some coordination that has to take place

between making a vibration (plucking a string,

buzzing a reed) and shaping that vibration

(holding the string against a fret or pressing

certain keys).

Digital instruments have little expressive po-

tential primarily because their interfaces are so

boolean. Notes are either turned on or off. This

leads to a “set it and forget it” mentality which

Exp

rES

SIv

E p

OTE

NTI

al

TEChNIquE rEquIrEMENTS

Violin/Viola/Cello

PianoGuitar

Saxophone

Flute

Drum Set

Digital Keyboard

Sampler

Reactable

Step Sequencer

Tenori-On

iPhone / iPad Apps

Bass Guitar

Drum Machine / Sequencer

Recorder

Digital Interfaces Meet Gesture recognition

Traditional

Digital

results in a lack of expressive variation. Tradi-

tional instruments measure high in expressive

potential because the shaping of the vibration

is changed every moment. The guitarist won’t

pluck the string the exact same way every time.

Great musicians emphasize this inherent varia-

tion and use it to their advantage, creating in-

credibly expressive performances by highlight-

ing differences.

Looking at the chart, a pattern begins to

emerge. As instruments garner greater expres-

sive potential, they saddled with higher and

higher technical requirements.

My slow hunch told me that giving digital in-

struments gesture recognition would create

a third instrument type: an expression instru-

ment. This expression instrument would utilize

the composing strengths of digital interfaces

ThE INSTruMENT

while employing gesture to imbue those com-

positions with variation and expressive mean-

ing through intuitive movement.

LANdsCApE


as concentric wheels. The Trike concept builds off of the common rotary

sequencing model: A playhead, much like a clock hand, spins around a

wheel to the assigned tempo, as it crosses paths with strategically placed

note objects it triggers the associated note or sample. A good example is

the iphone app Spoke.

Trike builds on apps like Spoke by creating multiple levels of intensity. If

your movement is minimal (figure A) it plays only the rhythmic samples

in yellow; a simple basic rhythm. As your movement becomes more full

body (figure B) the wheel’s playhead stretches out in direct proportion to

play the next tier of samples, increasing the complexity and intensity of

the beat. And finally, as your whole body flails rhythmically (figure C) you

reach the outer echelon of intensity and trigger more rhythmic elements

such as cymbal crashes or synthesizer hits.

Because we can measure movement (the total portion of the body mov-

ing), the velocity (average speed of body parts just before sample strike),

and the overall tempo of the movement (periodicity between troughs in

movement) we can map these three variables to intensity (number of

rings playing), note velocity (individual volume of each sample played),

and macro-level tempo respectively. That’s actually a pretty robust and

responsive system already.

A couple other features include the potential to sample shift based on

intensity level by drawing connecting lines between the original sample

and a displaced sample, as well as the ability to create different rhythmic

sections, each with three levels of intensity. So imagine creating one Trike

section that you could use for a distinct intro, first verse, and subsequent

verses based solely on of how intensely you dance to each one, and then

a second one for similar but distinct chorus sections. pretty flexible with

minimal setup.

The greatest part, I think, is that it does nothing until you start dancing.The Trike concept involves three concen-tric rings which build on rhythmic inten-sity.

If I can muster three concepts like this one, I’ll be set. It’s a rhythmic in-

strument based on a tiered system of intensity. These tiers are embodied

a FiRST PROTOTyPE

This week we were asked to come up with three con-

cepts for what form our thesis could take. Then we were

asked to use prototyping to take one of those concepts

a step further. Since I have a pretty solid idea about

“what” my thesis is I decided to focus my brainstorm-

ing and prototyping muscle on the interface relation-

ships between the screen system and the gestural sys-

tem. I got some interesting results.

B l O g p O S TNOVEMBER 15, 2010


figure A figure B figure C

far left & Above: The Trike concept paper and physical prototype. A sequencer with three tiers of intensity (yellow, blue, red). As a user’s gestures become more intense, the sequencer adds tiers to it’s playback, making the rhythms more complex.

This concept was the first to include a param-eter I called “intensity,” a measure of composi-tional layering.


I started the concept presentation with a slight modification of my previous trike prototype and thought through the system a bit more concretely.

hoop is a rhythmic sequencer with three levels of intensity. The performer drags and drops drum hits onto the

prOTOTypE a: hOOp

wheel, spacing them so they create rhythms.

As the performer dances, the vision system measures the degree of move-ment and applies it as intensity. If the performer uses small, slow move-ments, the intensity will be low and the

sequencer will only play the hits in the innermost circle. If the dancer moves wildly the sequencer plays across all three intensities, stacking rhythmic el-ements on top of one another.

I began by sketching, and worked to-ward a paper prototype

{Above And left} Sketches and paper prototypes for Hoop, a tiered gestural sequence performance in-strument.

{RIGHt} Instrument system diagram and gesture map along with an interface wireframe.

Russell MaschMeyeR. OncE MORE, wiTH FEElinG. sEctIoN • 57

SySTEM dIagraM

The performer dances in front of the camera which translates the world into pixels. Computer vision algorithms use those pixels to decipher musical pa-rameters like envelope, panning/vol-ume, vibrato/bend, tempo, and slide. Those parameters are then paired with sequenced notes in the sequencer and ouput as expressive music, which the performer reacts to and the whole loop begins again.

IntensityVelocityTempo

pixels

Input

output

computeR

SeQuenceR

amplitudesfrequencies

note sequencegESTurE

Magnitude

speed

Periodicity

MuSIC

Intensity

Velocity

tempo


The second prototype was a linear sequencer concept, better suited for making melodies. The linear array of notes allows you to see the “shape” of your tune.

The concept involved stacking various sequenced parts together into a multi-part song. Tap to turn notes on and off and tap different sections to edit or to

begin composing for that section. It was a pretty standard set of features for this kind of sequencer. I hoped the gestures set it apart.

Gesturally, Aqwire utilizes conducting gestures to add expressivity. I imag-ined a user actually holding a baton while using both hands to sculpt musi-cal expression.

Whereas Hoop was aimed more at live performance, Aqwire attempts to solve a composition or audio engineer-ing problem. The user could do mul-tiple “takes” of gesture recording. The takes are represented at right by the wavy colored lines. The user could then fine tune those gestural performances through touch gestures and create a keenly honed performance.

Aqwire attempts to make good on some of my original concepts which cast the user as a sort of mixing en-gineer, but failed as a direction on the whole.

prOTOTypE B: aqWIrE

{Above And left} Sketches and paper prototypes for Aqwire, a linear sequencer for melodies that made use of conducting gestures to shape expression.

{RIGHt} Instrument system diagram and gesture map along with an interface wireframe.


SySTEM dIagraM

The performer conducts in front of the camera which translates the world into pixels. Computer vision algorithms use those pixels to decipher musical pa-rameters like envelope, panning/vol-ume, vibrato/bend, tempo, and slide. Those parameters are then paired with sequenced notes in the sequencer and ouput as expressive music, which the performer reacts to and the whole loop begins again.

gESTurE

speed

hand Position

Baton Position

Periodicity

Posture

MuSIC

envelope

Panning/Volume

Vibrato/Bend

tempo

slide

envelopepanning/volumevibrato/bendtemposlide

pixels

Input

output

computeR

SeQuenceR

amplitudesfrequencies

note sequence


nEw idEaS & STRaTEGy

We’ve spent these last five weeks generating tangible

ideas (key elements/actors/functions) as well as con-

sidering business model contexts (customers/part-

ners/value proposition/key activities/competencies). A

lot to consider, and I wish the whole semester had been

dedicated to what we’ve done these past five weeks.

I managed to eke out two concepts: Hoop & Aqwire (discussed in the pre-

vious post). I was pretty happy with hoop, but less so with Aqwire. caught

between Turkey Day, freelance work, and other school work I didn’t have

the brainpower I needed to dive in and start ideating. I went into my pre-

sentation feeling lukewarm about my results. Though I came out of it

happy that my educational preamble about music was well received.

Skip ahead to Thursday, December 2nd, Design Management. Our in-

class assignment:

“You’ve started a company. Create the first draft of a design brief for your imag-inary design team. The project is your thesis. You have two hours.”

In two hours I had to compose a project overview, an audience analysis,

a competitive review, a project scope and a set of business goals to ex-

plain my thesis idea to a team (albeit make-believe). This imaginary team

would have to create my thesis based solely off the brief I provide. Design

thinking lightning round. For whatever reason, this framing finally did the

trick.


three advancements came out of it:

01 I realized that I want to make an instrument focused on perfor-

mance, not composition.

02 I realized there was a hole in the tablet instrument marketplace

for professional instruments. There are a lot of ipad instruments

out there, but they either feel like novelties or powerful applica-

tions ill-suited for the environment. I wanted to make an ipad app

designed to be used on a stage for expressive, intense perfor-

mance.

03 communication between musicians is a killer feature for live per-

formance. The ability to pass performance and expression data

back and forth between performers could make things incredibly

interesting.

The following morning I had a great conversation with Nicholas Felton

about all this. At that point I was still imagining multiple instrument ap-

plications, each of which communicate with the others. It seemed daunt-

ing as a single thesis project. he pointed out that it was entirely possible

to make a single instrument that could be used in a variety of ways (for

rhythmic playing or for melodic playing).

So, what am I making? I’m making an ipad-based, sequencer-style, multi-

purpose instrument that communicates performance data wirelessly with

other ipads running the application to enhance communication and per-

formances between multiple musicians.

All said and done, I’m pretty happy with where I ended up.



THESiS cOncEPT wRaP-UP

The day has finally come. It’s the end of the semester

and today I deliver the fruit of my research and ideation

work: my final concept.

After my last presentation, paul pangaro asked a simple question that

didn’t have a simple answer: “What is music?”

I was speechless. I spent the last couple weeks in a back-and-forth with

paul about that question, formulating an answer. he further asked the

questions: “What’s the purpose of music? Whom does it serve? What is

good music? Why should we care?”

And after a little deliberation between paul and myself, here’s the answer:

MUSic iS an aRT inTRicaTEly cOnnEcTEd TO EMOTiOn.Great music is a tool for empathy, for a musician’s emotion to

resonate with our own. Great musicians possess greater control

of expression. Increasing an musician’s capacity for employing

that expression means better music.

FOR a MUSician TO incREaSE caPaciTy FOR ExPRESSiOn: he/she must practice for years on traditional instruments or

utilize a device that gives more direct control over expression.

design principles

I came up with a few design principles over the past few weeks. Some

ideas that would help focus my final concept and really make it something

I was proud of.

• Standalone interface, that’s augmented by gesture

• Built for live performance

• communication between instruments

• One interface, multiple instrument roles


e l e v a T o r p i T c h :

I’m making a digital instrument that uses gestural input to enhance expressive control, enabling computer musicians to perform emotively.


Texture

Tempo

Intensity

Articulation

melodic chordal

rhYThmic

Velocity

Vibrato

& Bend

Okay, so maybe it’s a Venn Diagram of Expression, but that doesn’t really have the same panache. The idea was to create a mapping of expression in music.

There are three basic roles for instru-ments in music: Rhythm, Melody, and

Accompaniment. That’s an oversim-plification, but a useful one. Each of these roles uses different elements of expression to imbue a performance with emotion. Rhythm instruments use tempo, note velocity, and inten-sity to signal emotion. Melodic instru-ments use articulation (the length of

the note), note velocity, vibrato & note bending. Chordal instruments utilize articulation, intensity, and sound tex-ture or “color” to convey emotion. Con-necting these essential elements to gesture will provide a means of direct control to digital musicians.

ThE pErIOdIC TaBlE

OF ExprEssION{below} Early iterations on the diagram ex-ploring a means to communicate the dimen-sions of expression as they might relate to a particular instrument’s role in a composition.


MakINg CONNECTIONS

some early concept planning for Trinity. three distinct in-strument roles in one app could communicate, ipad to ipad, expressive values of their respective performers. By sharing expressive values between performers, i thought it might be possible to create uniquely convergent qualities in a multi-user performance.


Trinity for iPad is a new kind of in-strument that combines the ease of a sequencer with the expressiveness of gesture. It’s great for composing music, but it’s built for expressive live performance. You can make rhythms in Beat mode, play melodies in Tune mode, or accompany a singer or other musicians in Chord mode.

After you’ve chosen Beat, Tune, or

Chord you can create a sequence of beats, notes, or chords on the wheel. Create multiple sequence wheels to create different sections of a song. Hit the play button to test your sequence and rearrange, recompose or make any other changes you’d like.

When you’re ready to perform, connect wirelessly to a Kinect and laptop run-ning Trinity’s computer vision applica-

tion. The camera captures your move-ments and while the sequence plays, applies your gestures as the tempo, note velocity, intensity, articulation, texture, vibrato & note bend. The ex-pressive values are tailored to the par-ticular instrument you’re playing.

Trinity is built for group performance and wirelessly shares expressive data between performers in real-time, fur-

ther enhancing the performance ex-perience. Beat likes to share timing data, Tune likes to share tonal data, and Chord likes to share sound textural data.The experience of Trinity will revolu-tionize the expressivity of digital mu-sic, providing a means for masters and novices alike to construct musical se-quences and then impart the beauty of musical expression through gesture.

{left} Trinity can be used as a melodic, chordal, or rhythmic instrument. Each one communicates it’s expressive values to other performers, allowing for a real-time sharing of expression.

TrINITy FOr ipad

melodic chordal

rhYThmic

{RIGHt} The performer’s gestures are picked up by the Kinect and interpretted by the Trinity desktop application, which is linked via blue-tooth to the Trinity app for iPad.

The iPad app provides the note sequence while the dektop app provides the expressive nuance.

EvolutIoN • 65Russell MaschMeyeR. OncE MORE, wiTH FEElinG.


leila formed the ultrasounds just over a year ago after bonding with Barry

and Patrick over their love of lcD soundsystem and Daft Punk. they want

to make digital sounding dance music with a great stage performance.

leila a talented singer and has been performing for three years. she can

play the keyboard well enough to write great songs, but she has never

mastered any instruments.

Barry’s talent is programming great dance beats using drum machines

and samplers. he likes to play extra drums during live shows to add some

life to his beats.

Patrick has played cello & violin in classical music groups before, but

joined the band because he was interested in trying out some different

styles of music.

To illustrate a use-case for the Trinity concept I created a story about a band. I hoped to position Trinity as an instrument for those who want to make popular music, but also enjoy thinking about performance and sound-creation in non-traditional ways.

The ulTrasounds


the crowd always looks pretty bored.

they’ve written some great songs, but they’ve been frustrated with their

stage performances. Most of the on-stage action involves them pushing

buttons to trigger sequences.

Barry peruses the new releases on the app store and discovers a new

instrument app called trinity. It uses the Microsoft Kinect and his iPad in

to create expressive digital music.

Being curious about new digital instruments, he downloads trinity to his

iPad and opens it up. he finds its really easy to compose interesting beats.

he decides to try the gestural control and follows the in app instructions

for pairing the Kinect with his iPad.


he tells Patrick about it the next day before rehearsal starts. since it’s

only $14.99 (cheaper than a pack of bass strings!) Patrick downloads

trinity to his iPad as well.

he does a quick tutorial and he’s amazed! the beat has come alive! the

rhythms are responding to his dancing, getting louder and softer, moving

faster and slower, even growing more complex as he increases the

intensity of his movement.

he chooses the “tune” instrument and starts playing around with the

synthesized violin sound. he composes a musical sequence to match

Barry’s beat.

Patrick syncs with Barry’s Kinect as well. the instruments share the

composition data and synchronize the tempo and key.

Just then, leila walks in to the rehearsal space…


they tell leila about trinity and agree she should buy an iPad just to get

this killer instrument app! she likes that she can take her iPad with her to

capture musical inspiration.

… and sees Barry dancing up a storm to a beat that seems to move with

him. It looks like Patrick is conducting. the music swells and moves with

his gestures. It sounds really emotive, and it is mesmerizing to watch.

a few weeks later they debut a new song composed and performed on

three iPads running trinity. the ultrasounds feel more expressive than

ever …

… and it shows in the reaction of the crowd. the audience is dancing up a

storm and is mesmerized by the stage performance.

they’ll certainly be composing more songs using trinity in the future, and

may even bring fans up on stage to help them perform the songs!


gONE TO FlOrIda.dON’T Call.Our second concept presentation signalled the end of Thesis Development and the start of the winter break. I spent the better part of two weeks at my parent’s home in Florida, reflecting on the semester and eating my fair share of home cooked meals. Never underestimate the power of home cooking to clear your head.

I returned to New York after the new year, feeling uneasy about my direction. Paul Pangaro took charge of our Thesis Presentation class and prepared us quickly for a semester of clarifying our ideas and preparing our prototypes.

Clearing my head proved incredibly valuable. In the first week of Spring semester I changed di-rection and rediscovered my project priorities.


a cHanGE OF diREcTiOn

Coming out of last semester, I felt great about where

my thesis was, conceptually. When I thought about the

possibilities generated by an instrument that could

play expression instead of notes I got excited. I still get

excited. What was still cloudy and getting even cloudier

was how exactly I would prototype something like this.

My naïve plan was to start in iOS, building a rudimentary app using the

Stanford MoMu instrument making library. The goal was to end up with an

iOS app I could (hopefully) put into the App Store. Somehow this goal took

precedence over getting as far as possible with the prototype concept

in general. It was suddenly all-important that it live on an ipad. So much

so, in fact, that in the back of my mind I had actually begun considering

forgoing the use of Kinect and computer vision altogether.

This week I had a “road to Damascus” moment, realizing how ridiculous

that course was. It’s not in the least bit important that the prototype I

reveal at the end of this semester live on an ipad. Even in the best of cir-

cumstances it might not even make sense on an ipad. Think about it…

here you are waving your arms and moving about and then in the middle

of a performance you’ve got to stop and manipulate an interface on an

ipad? The best chance of creating a robust system for musical expression

lies in measuring the movements of the whole body or parts of the body in

particular, something that computer vision provides a great solution for.

The accelerometer in a single iOS device… doesn’t.

So I’ve changed my prototyping course. Instead of beginning to work in

iOS creating an app, I’ll begin by utilizing a much more flexible set of en-

vironments that will get me up and running with the many pieces of this

project faster: OpenFrameworks, Max/MSp, and Kinect.

prototyping checklist

• Get Kinect up and running (check!)

• Get OpenNI up and running for skeleton tracking (check!)

• Getting OpenNI Sending OSc values into Max/MSp (Nearly!)

• Experiment with x, y, & z positions, velocities, vectors, accelerations

and movement periodicity to determine best mappings for adding ex-

pression to sequenced pieces within Max/MSp (That’s a big one)

• concretize and fine-tune expressive mappings (Scrubbing it clean)

Of course that leaves out the actual sequencing interface itself. What will

the application look like? In the previous plan it had become the center

of the prototype. Don’t get me wrong, I believe a stunning visual design

is important. But in terms of demonstrating the concept, I’d much rather

have a prototype that someone could actually play—even if it’s ugly—than

show them a beautiful app that does almost nothing.

That being said, I do plan on creating a full set of visual designs and user-

flows for the future consolidated application. Static designs however will

get done far faster once I know what I’m working with functionally.

B l O g p O S TJANUARY 19, 2010



a cOllEcTiOn OF GOalS

Paul Pangaro put me in touch with

robin Bargar, who I’ll be meeting

with later this week to discuss my

thesis. I’m really excited for the op-

portunity as robin seems uniquely

qualified to tell me if I’m completely

crazy or not.

In preparation, Robin asked me to put together

a summary of my goals, both primary and sec-

ondary, as well as an articulation of how I plan

to accomplish them, how I will know when I’ve

accomplished them, and any related questions.

primer:

Music is an art intricately connected to emo-

tion. Great music is a tool for empathy, for a

musician’s emotion to resonate with our own.

Great musicians possess greater control of

expression. Increasing a musician’s capacity

for employing that expression means better

music.

hypothesis:

For a musician to increase their capacity to

control expression, he/she must practice for

years on traditional instruments or utilize

a device that gives more direct control over

expression

primary goal

create a digital instrument that provides

an intuitive enhancement of expressive

control.

plAn: Use computer vision to allow for gestural

control of the expressive elements of a pre-

sequenced musical piece.

SucceSS: Through expressive gestural control,

one should be able to generate multiple emo-

tional variations from a single pre-sequenced

piece.

QueStIonS: What are the primary musical vari-

ables that create expression (tempo, velocity,

timing, articulation)? Which gestural mappings

to expressive variables are the most intuitive?

secondary goal

Build an instrument for live performance

plAn: Building a system that is portable, that

works in a variety of stage environments, is

quick to set up and doesn’t crash or otherwise

fail to operate mid-performance. Building a

system that feels immediately responsive to a

performer’s actions. create a visual feedback

system for the musician, displaying the control

parameters and the musician’s real-time input.

SucceSS: can be carried by a single person.

Takes < 10 minutes to set up. has system fail-

safes in place to make sure glitches go as un-

noticed as possible. Must have a response time

<= 1/16 note at approximately 120bpm. Visual

feedback informs the performance and allows

for a better understanding of the correlation

between movements and expressive outcome.


QueStIonS: What fail-safes can be put in place

to ensure a continuous performance? Latency

between movement and its capture and inter-

pretation already seems to be somewhat of a

problem. how can I either: lessen the latency

as much as possible and/or create gestural

mappings where latency is less of an issue? Is

it important that the performer see their body

position reflected in the visual feedback? Or is

it more useful if feedback is abstracted into the

pertinent variables?

secondary goal

create a single interface that can be ap-

plied to multiple instrument roles

plAn: create an interface whose elements (grid

structure, note selection, visual feedback) can

be applied easily to melodic, rhythmic, chordal/

accompaniment constructions.

SucceSS: A musician would be able to com-

pose all the musical parts for a composition

having only learned a single interface

QueStIonS: Is melody, rhythm and chordal/

accompaniment a robust enough list of in-

strument roles? Are the expressive elements

different for each role? If so, how do/do they

overlap?

secondary goal

afford constructive communication Be-

tween Instruments

plAn: Determine which expressive qualities

can/should be shared between multiple instru-

mentalists. create a system for visual feedback

that makes clear the distinctions between what

is being generated through my gestural input

and what is being shared and used from your

performance.

SucceSS: Two or more musicians could en-

hance a total performance by applying the

output from one performer’s instrument to

the expression of a second, third or fourth in-

strument. For example, I could share my note

velocity data with you, so that our sequenced

melodies stress their notes in the same way.

You could share your articulation data so that

my notes are staccato and legato when yours

are.

QueStIonS: This seems trivial to me on the

surface, but is the simplicity of this the key? Is

it more useful to keep it simple and allow the

way it’s used by musicians to define its worth?

Or should I be thinking more deeply about what

elements are communicated or how they’re

communicated? I.E. is it smart to have a 1:1

pairing? Or could one instrument have a com-

plimentary reaction to incoming shared data

rather than ape it?


MEETinGS and adjUSTMEnTS

This past week I had some great conversations and

some important realizations.

requirements

On Thursday I made a requirements document, a simple list describing

each element that would have to be accomplished for me to complete

my thesis. Going into the exercise I imagined I would create a two-part

system: A sequencing system and what I was calling the “expression en-

gine”—no relation to the cMS system—which would handle the gestural

input. After listing out the requirements for the sequencing system and

realizing it was three to four times longer than the expression engine, it

suddenly occurred to me that it might not be practical.

After a conversation with Nicholas Felton (my advisor) on Friday, I spoke

about my misgivings. he encouraged me to make a hard decision and

drop it. I thought about it for a second and realized that deciding instead

to focus on making the expression engine opens up more interesting pos-

sibilities. If it’s not tied to a particular sequencer then perhaps it could

be incorporated into the workflows of other sequencers. perhaps it could

become an easily adoptable add-on or augmentation of the devices that

musicians are already using.

Thinking of the expression engine as a discrete instrument makes it a

more portable, compelling idea, and without a pretty sequencer interface

to distract the audience on presentation day, it might make the actually

significant part of what I’m developing clearer.

Josh davison

On Saturday, I had my second Skype chat with Josh, a chicago native

and computer musician. I caught him up on my thinking and got some

great feedback. he helped me realize that a lot of the expressive values

I’ve been mulling over (velocity, tempo, texture, intensity, articulation) are

more applicable to all instrument roles than I had previously assumed.

For instance, I had originally assumed “intensi-

ty” was a value best left to chordal and rhythm

instruments, but intensity could also be applied

to melodic instruments in a few ways.

complimentary notes could added into the

melody that match the key and current explicit

notes, instruments of different timbres could

be added playing in harmony with the current

notes, overtones could be added to the explicit

notes, making the basic sound richer in texture.

So I’m re-examined my siloing of expressive

values. perhaps I’ll just give musicians a list of

values and they can turn their control of them

on and off as they please. It’s perhaps less el-

egant, but also perhaps more powerful.

robin Bargar

This very morning I met with Robin Bargar, one

of the early pioneers in the field of virtual real-

ity, a musician, and Dean of Technology & De-

sign at city Tech in Brooklyn. he’s a very sharp

man.

he encouraged me to be very up front about

who and what I’m making this expressive en-

gine for. The field of music and digital music

control is large and has a pretty rich history.

There’s a lot of scientific research within the

field, he pointed out, and that it would be im-

portant for me to be explicit about my assump-

tions around what kinds of music or approach-

es to music this engine is aimed at. In other

words, clearly define the constraints (musical

or otherwise) under which I’ll test my system to

determine success or failure.

he also gave me some insight into the word he

was doing in the early 90’s re: virtual reality. he


prescribed a technique he has employed then

to separate the testing of the two parts of the

system

01 The control schema that actually ef-

fects the expressiveness of the music

◊ What elements comprise expression

(velocity, tempo, articulation)?

◊ What values comprise those ele-

ments (velocity is comprised of a

single numeric value between 0 and

127, but articulation may be com-

prised on multiple values which vary

depending on the instrument, the

same could be said for intensity)?

02 The movement mapping that creates

those values

◊ Does the average velocity of my

arms at any given moment deter-

mine the velocity value? Or should it

be the magnitude of movement in all

of my joints?

All comes down to which mapping feels more

appropriate in testing.

Robin encouraged me to get part one figured

out in a week or so using a simple sequencer

set up and direct (we’re talking knobs here)

control over value setting. This would help me

figure out which elements good candidate for

creating expressive outcomes and what values

they require. The next—and much harder—part

is figuring out how gestural input can yield a

sufficient amount of those values to yield ex-

pression.

B l O g p O S TFEBRUARY 22, 2010


wHaT’S in a naME?I’ve probably lost a few of you along this long and winding road. I’m sorry.

Today, I aim to clarify. Below, you’ll find as clear a description as I can

make at this point as well as the new (still impermanent) name for my

project. Think of it as the next level up from an elevator pitch:


What is MOTiv?

New digital music making interfaces lack affordances for musically expressive control, depriving musicians of their innate ability to emote through performance.

MotIV gives digital musicians expressive control by interpreting their physical gestures in real-time, on stage, during the playback of a composition. using an adaptive computer vision system, MotIV puts musicians in control of the tempo, intensity, note velocities, articulations, pitch bends and vibratos in the moment, giving way to a musical conversation with surprising and expressive results.

For those of you who are curious, MOTiv stands for “Musically Oriented

Translation of Independent Vectors”

Russell MaschMeyeR. OncE MORE, wiTH FEElinG. ExEcutIoN • 77

This was the moment I had been waiting for all these long months. I finally had the green light to dive into the code and start building something for real. I couldn’t have been more excited and I couldn’t have been more nervous.

I was embarking on an entirely new coding language, C++, which I had never used before. I had to navigate the perils of installing openFrameworks as well as find some way to hack the Kinect. I knew a little bit about object oriented programming, having played around in Processing the previous year, as well as having a bit more in depth knowledge of actionscript from all my years in advertising. But I had no idea if my meager knowledge would carry me where I needed to go.

Thanks to the efforts of the open-source community, the Kinect was hacked days after its release in November of 2010, and the code was made available almost immediately for the hacker community to play with. I owe a huge debt of gratitude to the open-source community and the brilliant coders like Zac Lieberman and Theo Watson who work to make things easier for the greener makers among them.

With a kick in the right direction from Robin Bargar at just the right time, it turned out to be a mightily smooth ride.

ExECuTION

0 6

school of VIsual aRts. MFa in inTERacTiOn dESiGn.78 • ExEcutIoN

OH, GlORiOUS dayToday I got up and running with Kinect. Tomorrow, who knows? I’m meet-

ing with my technical advisor tomorrow. We’ll discuss options for world

domination and fill in the holes in the following equation:


{Above} A split screen view of the first Kinect hack I tried. The left pane is the color coded depth map seen by Kinects infrared camera. The pane on the right is the output from the RGB camera.

pWn kinect [something] world domination



a SkElETOn TO call My OwnAfter about a week of what seemed like slamming my head into an impenetrable wall of code I finally

got Roxlu’s ofxOpenNI addon to run! I had another version of OpenNI running in the processing envi-

ronment, but running it in openFrameworks gives me A LOT of room to grow.

next steps include:

• Get ofxOSc addon up and running

• Stream OSc data to Max/MSp using ofxOSc

• Obtain real-time coordinates of skeleton joints (x, y, z space)

• calculate vectors, velocities, and accelerations of joints

• create a Max/MSp sequencer patch that receives movement values

• Experiment with mappings of movement values to musical values


{Above} Here’s Jessica’s silhouette and digital skeleton. I installed many versions of openFrameworks before finally finding one that worked with Roxlu’s Kinect hack addon.


My first kinect hack

ofxopenni, Max/Msp expression test

FiRST ExPRESSivE TESTS

I’ve made some great coding progress so far utilizing openFrameworks

and their ofxOpenNI and ofxOSc addons with a Max patch using the

cNMAT objects. It took a week or so to get OpenNI running, but once I did

the final demo came together in a single day!

can’t wait to start devising some expression experiments.


{Above} The first mapping of skeletal joints to expressive parameters. Here my friend, Clint, changes note velocities using the y-position of his hand.

{top RIGHt} My first attempt at hacking the Kinect using OSCeleton

{bottom RIGHt} My first test run of open-Frameworks with ofxOpenNI which takes the Ki-nect output and creates a digital skeleton.



Max/MSP + wacOM HijinkS

In follow up to a meeting I had with robin Bargar on

Monday, I spent this week constructing a suitable con-

trol environment utilizing MAX/MSP and my Wacom

tablet.

Essentially, I’m using my Wacom pen and the parameters it outputs (x

pos, y pos, x tilt, y tilt, pressure, z-axis, etc) as real-time control knobs to

manipulate different expressive parameters (tempo and bend to start).

The thought goes, if I can construct a suitable format for controlling ex-

pression with this level of control, I’ll have a much better chance of con-

trolling expression using bodily gesture with the Kinect.

Wacom & Basic Midi

Wacom & robust virtual instruments


{left} Using Max/MSP with my Wacom tablet I was able to manipu-late MIDI, changing a song’s tempo and pitch bend in real time.

{left} An extension of the above video. This time the instruments are in Logic and I can control multiple instruments at once.


ExPRESSivE cOnTROl: vElOciTy, BEnd & viBRaTO

After a frustrating week trying to get some real-time control over a MIDI

file’s tempo during playback I decided it was time to move on, try to keep

to my schedule, and begin control work on note velocity, pitch bend, and

vibrato.

Luckily, I did that all in one night. I have until Monday or so to gain control over articulation. hope-

fully I can finish that quickly and get back around to finishing tempo. here’s the video proof from

last night’s work.


{Above} With my Wacom prototype I gain con-trol over velocity, bend & vibrato in Max/MSP. Things start to feel expressive.



BaSic cOnTROl PROTOTyPE cOMPlETE!

By jove I’ve done it! With the final element of expressive control under my

belt, I’m looking forward to getting the gestural control experiments under

way. This is when it starts to get exciting, people!

I’ve also hatched a plan to begin working with an electronic musician, or perhaps a few, to create

compositions with my tool in mind to test how they’d like to interact with the application and see if

anything interesting comes out of it. I’ve got a few folks in mind. More on that when I have it.

For now, enjoy the show!


{RIGHt} I present the complete Wacom prototype with expressive control over articulation and intensity. I also put to-gether a better continuous algorithm for tempo control during playback.


wE aRE GO FOR laUncHA few things happened this week. Jack Schulz and Matt Jones of BERG stopped by the SVA studio

to lead some students in a week-long workshop. It was amazing. We made nonsensical product

drawings, re-imagined the average household thermostat as something you might see in a book

about cooky Japanese inventions by making paper prototypes, and did a bit of technological mate-

rial research to feed our craft. This was all amazing, and well worth me losing a bit of time to focus

on thesis.

Nonetheless, I still managed to hit a major milestone in my thesis prototype. I was able to calculate

the three-dimensional velocity a user’s right hand (chosen somewhat arbitrarily… I’ll eventually be

performing these calculations on all the virtual joints). I wired that velocity directly to the note veloc-

ity parameter of my Max/MSp patch et voila!


{Above} I begin connecting my Max/MSP con-trol algorithms to gestural control. Here we con-trol the velocity of the notes played in the MIDI file by mapping them to the velocity of your right hand.

Nicholas Felton guest stars


Also, after speaking with the fine gentlemen of BErG about my thesis, they

gave me a lot of great feedback and encouraged me to consider how I could

get this idea across in a video, without having to explain it. i.e. How could I

do something like this:

singing sock puppet

Now, admittedly, that’s a simpler device than what I’m constructing. But you get it almost in the first

instant. You get why it’s great by smile on her face and the way she starts adding her own performa-

tive gestures, which really making this gadget sing (forgive me). I don’t need to know how it works.

I just see that it does.

how can I do that?

{RIGHt} Irvine Brown’s project “Singing Sock Puppets.” A brilliant use of docu-mentation techniques to present an idea simply and effectively.


THiS iS GETTinG inTEnSE

At the encouragement of the gentlemen of BErG I be-

gan thinking about how I might communicate this con-

cept without having to explain it. No epiphanies on that

yet, but it did lead me to thinking about the system’s

visual feedback.

I think the system’s visuals could do a lot to communicate what’s happen-

ing without explanation. I started by sketching. There are already some

great ideas in there. I’ll be creating a lot more of those little sketches be-

fore I’m ready to actually make anything happen though.

Working in openFrameworks on Monday, I successfully determined the

velocities of all the virtual Kinect joints, and derived the magnitude of the

body’s movement from that information. I also did a bit of visual graph-

ing to understand the patterns of my own movement. Then I managed

to actually map the magnitude of my movement to this thing I’m calling

“Intensity.” Intensity in my system really just means how densely layered

the composition is. The more “intense” a piece becomes, the more layers

of instrumentation are added.

Enjoy! I’m off to walk around Austin and steel myself for long nights and

lots of cool people meeting.


{Above} I present some rudimentary visual feed-back so that users can see the effect of their move-ments within the system. I also add control over the magnitude of movement and map it to intensity.

ExEcutIoN • 87Russell MaschMeyeR. OncE MORE, wiTH FEElinG.

interface sketches

A few of the post-it note sketches I used to quickly ideate some options for the visual feedback system. I wanted to chart the effects of the gestures in a way that directly tied to the body. What would it look like to visualize musical super-powers.


callinG all cOllaBORaTORS

Today I had an epiphany, and it all started with a lie.

This morning I attended what I thought was a presentation on community

building at SXSW. It turned out to be less of a presentation, and more of

a sharing and discussion group about the communities we were building.

Uh oh. I suddenly realized how far I had sat down from the exit doors. I

hadn’t come prepared to talk about the community I was building. I wasn’t

even building a community! So when it came time for me to share… I told

a little lie.

Well I’m building a new kind of instrument, so I’m trying to build a commu-

nity of musicians to play with it and help build it, together, into something

really valuable.

It was a little white lie—until I heard myself say it.

Then I realized how ridiculous it was that it wasn’t the truth. It immedi-

ately dawned on me how transformative turning full-force to the music &

developer community could be, especially right now. To date I’ve held back

from opening up to others. Until recently I was still forming my vision for

the project. Without a clear vision, opening up to a community—any com-

munity—would have probably led to confusion and frustration. It might

have been exciting in some regards, but would have lacked productive

direction. Ultimately it might have killed what I believe could be a funda-

mentally transformative concept in digital music performance. I needed

some time to figure out what it was that I believed in.


Now—having developed some of the rudimentary principles as a proto-

type, and having a clear vision for MOTIV, I think it’s the perfect time to

invite others to play, develop, and otherwise shape the experience. com-

munity is the driving force behind the development of Wordpress, open-

Frameworks, and the Arduino physical computing platform (not to men-

tion all the Kinect hacking happening as well). Each one of these projects

relies on the collective shaping power of the community and a directing

vision of a core group in communication with that community.

So, today, I’m Issuing an open call on meetup.com to musicians and de-

velopers alike. Let’s get together, play, and develop MOTIV into an intui-

tive, powerful, and expressive musical tool. Let’s collaborate! If you have

friends who fall into one of those categories, spread the word!


{Above} The offi cial MOTIV Musicians & Developers meetup.com page, with over 30 members only a few weeks after launch.

Russell MaschMeyeR. OncE MORE, wiTH FEElinG. rEsult • 91

I was giddy when I first connected gesture to the expression control system in Max. It was a major milestone. At the end of that week I videotaped Nicholas Felton controlling the velocity of the notes in a song with just the speed of his arm move-ments. Within seconds he had the feel of it and was using it expressively. He was improvising. There he was, someone who wasn’t a musician, expressing himself through music. He had a big smile on his face.

I would see that same enormous grin over and over as I tested MOTIV out on vari-ous audiences. There’s this amazing moment when you see people ‘get it’ and their eyes get wide and they start experimenting with their gestures. It’s brilliant to watch.

As I brought the two other core parameters of expression, intensity and tempo, the experience became even more immersive. Performers were concentrating and conversing with the music. Something I only hoped for a few months before was happening with every new user.

rESulT

0 7

92 • rEsult school of VIsual aRts. MFa in inTERacTiOn dESiGn.

MOVE THE MUSIC.

rEsult • 93Russell MaschMeyeR. OncE MORE, wiTH FEElinG.

school of VIsual aRts. MFa in inTERacTiOn dESiGn.94 • rEsult

tempoVelocityattackIntensity

MySONg.MId

compose music on the digital device you already love—a keyboard, sequencer or a drum machine. Then save it as a digital sequence, like a .midi file.

Open MOTIV and load your digital sequence, then select the expressive parameters you want to con-trol during the performance.


connect your Microsoft Kinect to your computer and start moving. MOTIV will track your gestures and turn them into musical expression.

MOTIV weds your expression to the sequence in re-al-time, sending it to your virtual instrument while visualizing your input for the perfect performance.

96 • rEsult school of VIsual aRts. MFa in inTERacTiOn dESiGn.

Motivators

Two users, Jessica and Dave, experiencing MOTIV for the first time. These early test runs were held at the Interaction Design MFA studio. There are some really beautiful moments in both videos when you see the experience come into focus.

rEsult • 97Russell MaschMeyeR. OncE MORE, wiTH FEElinG.

school of VIsual aRts. MFa in inTERacTiOn dESiGn.98 • rEsult

if (rLimb.end_joint == 15){ velocityX = pos[1].X * 2.25; velocityY = pos[1].Y * 1.875;};

if (rLimb.end_joint == 3){ intensityX = pos[1].X * 2.25; intensityY = pos[1].Y * 1.875;};

if (rLimb.oldPos.X != 0 && rLimb.oldPos.Y != 0 && rLimb.oldPos.Z != 0) { XnPoint3D newPos = b.position; float x_diff = fabs(newPos.X-rLimb.oldPos.X); float y_diff = fabs(newPos.Y-rLimb.oldPos.Y); float z_diff = fabs(newPos.Z-rLimb.oldPos.Z); if (x_diff != 0 && y_diff != 0 && z_diff != 0) { double distance3D = sqrt(x_diff*x_diff + y_diff*y_diff + z_diff*z_diff); rLimb.endClock = clock(); double clockDif = (rLimb.endClock-rLimb.startClock)/CLOCKS_PER_SEC*1000; rLimb.velocitySon = distance3D/clockDif; rLimb.velocityAvg = (rLimb.velocitySon+rLimb.velocityDad+rLimb.velocityGrandad)/3; velocities[rLimb.end_joint] = rLimb.velocityAvg; magnitude = 0; for (int k=0;k<25;k++){ magnitude+=velocities[k]; }; rLimb.velocityGrandad = rLimb.velocityDad; rLimb.velocityDad = rLimb.velocitySon; rLimb.startClock = rLimb.endClock; };};rLimb.oldPos = b.position;

01 Set the coordinates for the hand velocity bubble

02 Set the coordinates for the intensity valences

03 If we’ve got a baseline, calculate velocities

04 Set the current joint position / point

05 Calculate the distance between old/new points

06 if there was a change in one of the points...

07 Find the distance between the old and new point

08 End the timer

09 Calculate the time it took

10 Calculate the velocity

11 Store avg velocity in the array of velocities

12 Sum the velocities array

13 Turn the new velocity into the old one

14 Start a new timer

15 Turn the new position into the old one

TraCkINg vElOCITIES


WhaT MakES IT gO

openFrameworks is a library of C++ code that makes it easy create a lot of fun stuff. I dug deep into ofxOpenNI, an ad-don for openFrameworks which creates a digital skeleton, to figure out how to find the position of the user’s hand in every frame. Once I could track the position over time I was able to determine the direction and velocity. Then I performed the same set of calculations for every joint to determine the overall intensity of movement.

Using simple openFrameworks functions I created a visual feedback system to give the user a higher degree of control. I mapped the changing expressive values onto the perform-er’s body.

This environment is the control center for the MOTIV ex-perience, collecting and visualizing the gestural input from the user.

Russell MaschMeyeR. OncE MORE, wiTH FEElinG.

ThE

dr

IvIN

g E

Ng

INE

You’

re lo

okin

g at

the

mai

n ne

rve

cent

er f

or

MOT

IV. I

f op

enFr

amew

orks

is t

he g

as p

edal

an

d da

shbo

ard,

thi

s is

the

eng

ine.

Var

iabl

es

like

tem

po,

velo

city

and

int

ensi

ty c

ome

in-

from

ope

nFra

mew

orks

get

pro

cess

ed a

nd

mat

ched

up

with

the

MID

I seq

uenc

e he

re, i

n M

ax/M

SP. I

t m

ay lo

ok c

ompl

ex, b

ut it

bre

aks

dow

n in

to a

few

sim

ple

part

s.

01 Fi

rst

up i

s th

e co

ntro

l pa

nel

whe

re t

he u

ser

can

turn

on

or o

ff t

he p

aram

eter

s of

con

trol

as w

ell a

s sp

ecify

whe

re th

ey’d

like

the

expr

es-

sive

MID

I inf

orm

atio

n se

nt o

ut. T

hey

can

rout

e

it do

wn

any

MID

I cha

nnel

they

like

. The

y se

t the

orig

inal

tem

po o

f th

e so

ng a

nd c

lick

to s

tart

the

song

pla

ying

.

02 Th

is i

s th

e in

stru

men

t se

ctio

n. T

his

is w

here

the

user

cho

oses

wha

t M

IDI fi

les

to p

lay.

Thi

s

sect

ion

take

s in

the

raw

, un

expr

essi

ve M

IDI

and

pairs

it w

ith t

he e

xpre

ssiv

e va

lues

in r

eal

time,

cre

atin

g ex

pres

sive

MID

I.

03 Th

is is

an

exam

ple

of a

con

vers

ion

algo

rithm

, a

patc

h th

at t

akes

the

val

ues

from

ope

nFra

me-

wor

ks a

nd r

emap

s th

em t

o va

lues

MID

I ca

n

unde

rsta

nd,

usua

lly 0

-127

. Ea

ch o

f th

e ex

-

pres

sive

par

amet

ers

has

a un

ique

pat

ch l

ike

this

one

.

04 Th

is is

the

tem

po c

alcu

lato

r. It

take

s th

e te

m-

po v

alue

bei

ng s

ent

from

ope

nFra

mew

orks

and

adju

sts

the

MID

I pla

ybac

k cl

ock

so th

at th

e

MID

I not

es a

re p

laye

d at

the

spec

ifi ed

tem

po.

05 La

st i

s th

e in

tens

ity p

atch

. Th

ink

of i

t lik

e a

dam

’s s

pillw

ay.

If th

e in

tens

ity v

alue

s re

ach

a hi

gh l

evel

, it

open

s th

e fi r

st c

ham

ber

and

star

ts le

ttin

g th

e M

IDI o

ut a

new

cha

nnel

. As

the

inte

nsity

clim

bs h

ighe

r and

hig

her i

t ope

ns

up t

he o

ther

tw

o sp

illw

ays,

whi

ch r

elea

se M

IDI

to n

ew v

irtua

l ins

trum

ents

.

Russell MaschMeyeR. OncE MORE, wiTH FEElinG. coNclusIoN • 103

MOTIV has been an incredible project to develop. If nothing else, I’ve spent a year creating a project that combines my love of music, process and technology. The way musicians and non-musicians alike have taken to the MOTIV prototype is an incredible validation of my research and the hunches I’ve been nursing to life over the past year, maybe longer.

This is the end of the first stage in the life of MOTIV. As it stands, it’s not yet a product, but a platform for discovery. There’s so much to explore as this new op-portunity space develops and I’m able to test what works and what doesn’t.

The next step is consolidating the code into a single application that can be played with pre-sequenced songs or with instruments like Monome, Tenori-on or iPad via real-time MIDI communication. I hope to open-source the project while continu-ing to build the adventurous community of musicians and developers that have already gathered around MOTIV.

You can always find out what’s new by visiting musicwithmotiv.com.

CONCluSION

0 8

Russell MaschMeyeR. OncE MORE, wiTH FEElinG. thaNk You • 105

Jessica for putting up with grad school.Nicholas Felton for being an amazing collaborator.Robin Bargar for knowing how to proceed.Liz Danzico for pushing in all the right directions.Paul Pangaro for insisting on clarity.Jennifer Bove for insisting on speed.Musicians and producers for sharing your thoughts and time.Larry Legend for the hearing a geek out.OpenFrameworks, Eric St. Onge, & Yang Yang for easing my foray into C++.Everyone who helped in any large or small way.

ThaNk yOu

0 9

Russell MaschMeyeR. OncE MORE, wiTH FEElinG. rEfErENcEs • 107

Steven Johnson, “The Slow Hunch,” in Where Good Ideas Come From (New York: Riverhead Books, 2010), 65.

Bruce Sterling, Shaping Things (Cambridge, MA: Mediawork, 2005), 133.

Adam Greenfield, “Thesis 43,” in Everyware: The Dawning Age of Ubiquitous Computing (Berkley, CA: New Riders, 2006), 148.

Marshal McLuhan, Understanding Media: The Extensions of Man (New York: McGraw-Hill, 1964).

Wikipedia. “Marshall McLuhan.” Accessed April 16, 2010. http://en.wikipedia.org/wiki/Marshall_McLuhan.

Daniel Levitin, This Is Your Brain on Music: The Science of a Human obsession (New York: Penguin, 2006).

Patrick Haggard and Matthew R. Longo, “You Are What You Touch: How Tool Use Changes the Brain’s Representations of the Body” on ScientificAmerican.com. September 7, 2010. http://www.scientificamerican.com/article.cfm?id=you-are-what-you-touch.

Malcolm Gladwell, “The 10,000-Hour Rule,” in outliers (New York: Little, Brown and Company, 2008), 35.

David Rokeby, “A Very Nervous System.” Accessed September 19, 2010. http://homepage.mac.com/davidrokeby/vns.html.

Finn Peters. “Music of the Mind.” March 23, 2010. http://www.youtube.com/watch?v=epT16fbf4RM&feature=player_embedded.

Antonio De Luca. “Aggregat.” July 30, 2010. http://vimeo.com/13759610.

Omeryosha. “AirPiano - Controlling Ableton LIVE.” June 2, 2008. http://www.youtube.com/watch?v=9K10XB1ycT4&feature=player_embedded.

Unearthed Music. “Gesture Music Sequencer.” June 20, 2009. http://vimeo.com/5247458.

D. Andrew Stewart. “Immortal-machine for méta-instrument.” May 30, 2010. http://vimeo.com/12157933.

rEFErENCES

1 0

school of VIsual aRts. MFa in inTERacTiOn dESiGn.108 • rEfErENcEs

Marcel Wanderley, New Digital Musical Instruments: Control & Interaction Beyond the Keyboard (Middleton, WI: A-R Editions, Inc., 2006).

Audanika. “Quick Intro to SoundPrism.” August 11, 2010. http://www.youtube.com/watch?v=385CymvTecU&feature=player_embedded.

Chris O’Shea, “Air Guitar Prototype with Kinect.” December 10, 2010. http://vimeo.com/17669981.

Chris O’Shea, “Testing OpenNI & Kinect.” Decmber 9, 2010. http://vimeo.com/17640133.

McGill University. “Measuring Musical Pleasure.” March 14, 2011. http://www.mcgill.ca/newsroom/news/item/?item_id=172676.

Adafruit Industries. “WE HAVE A WINNER – Open Kinect driver(s) released – Winner will use $3k for more hacking – PLUS an additional $2k goes to the EFF!” November 11, 2010. http://www.adafruit.com/blog/2010/11/10/we-have-a-winner-open-kinect-drivers-released-winner-will-use-3k-for-more-hacking-plus-an-additional-2k-goes-to-the-eff/.

Github. “roxlu/ofxOpenNI.” Last updated January 7, 2011. https://github.com/roxlu/ofxOpenNI.

University of California at Berkley, Center for New Music & Audio Technologies. “Downloads.” Last updated April 7, 2011. http://cnmat.berkeley.edu/downloads.

Irvine Brown, “Singing Sock Puppets.” Accessed March 3, 2011. http://www.irvinebrown.com/?p=15.

(CONT’d)

once more, with feeling

Documents

movement music

i n gthoughts evidence

love of music

processes of music

w i t h f e e

music pattern deviation

traditional instruments

new digital instruments