An Augmented Snare DrumELEC3580 LEVEL 3 PROJECT
Robin Gray Simon Lindsell Rachael Minster
IntroductionAims of the project
1) Use a camera to analyse the expressive patterns created on the drum skin during a drum brush performance.
2) Use a range of electrical sensors to enhance and improve the accuracy when detecting drum hits. This data will be used in conjunction with the visual data.
3) To emulate drum brush techniques used on a traditional acoustic drum controlling either audio samples or synthesis. At present, the use of drum brushes on commercial electronic drum kits is very restricted.
4) To use sensor data to control and manipulate parameters of a visual feed. This will be in real-time as an accompaniment to a performance.
System Overview
Electrical Sensors and
Camera
Data Processing
and Manipulation
Audio and Video Output
Modules
Parameter Mapping
Initial System Design
Camera(Capture brush movement
and position)
Piezoelectric Transducer(capture drum hits and
velocity)
Flex Sensor(capture pressure
appliedto the brush)
Flex Sensor(capture pressure
appliedto the brush)
PIC Microcontroller(A/D convertion
serial transmition)
Processing in
MAX/MSP and JITTER
Serial Connection
USB Connection
Visual/Audio output
Footswitches
Electrical Sensors and
Camera
Data Processing
and Manipulation
Audio and Video Output
Modules
Parameter Mapping
Initial testing - Camera
Initial testing discovered that our idea was viable, and that lighting the camera from above would help define the brushes.
This included:
• Clear and frosted drum skins• Different webcams and handheld camcorders• Various lighting conditions
Lens choice
A wide angled lens would allow the camera to be placed closer to the drum, this was initially tested with a door viewer
The final system consisted of three 38mm lenses mounted in a plastic tube
Initial Testing - Subtraction
Initial Image Processing
•Thresholding could be used to help differentiate the drum brushes.
•Unused areas of the video could be masked, removing further noise and un-needed background
•Necessary to differentiate between the two brushes.
Image Processing - Segmentation• Separates each frame into labelled objects
• This can be used to identify brushes on the skin
• Problems – Apparent ‘jumping’ of brushes between frames
– The merging of brushes when they touch
Motion Tracking 1 - Distance
•If the object travels greater than a set distance the distance to the other object can be measured.
•If this is smaller we can presume that the image has ‘jumped’ and can then re-label the image
Motion Tracking 2 – Colour Tracking
Requires using two different coloured brushes. A simple multiband thresholding method was used to detect objects within a colour range, differentiating between the two brushes.
Motion Tracking 3 - Orientation
120°
65°
It was noted that orientation of brushes remained constant throughout and could be used to track brushes.
The orientation can then be extracted and used to detect the left and right brushes.
Data Acquisition Overview
Sampler
Half rectifcation and smoothing
ADC
Encoder and serial transmitter
Data decoding and manipulation in
MAX/MSP
Webcam
Signal processing
Signal processing
Comparator
Piezo
Flex-sensor in left brush
Flex-sensor in right brush
PIC16F877A
PC
interrupt
Footswitches
Sensors – Piezo
+
-
Inve rte r
~
15V
-15V
10kΩ
200kΩ10kΩ
1µF10kΩ
Re ctifcation and Smoothing
Inve rting Amplife r
+
-
15V
-15V
10kΩ
Comparitor
+
-
15V
0V
15V
100kΩ
50Ω
• Mounted on the skin using tape and later a fabric pocket
• Hit used to trigger the PIC’s interrupt
• Signal processing involved amplification, smoothing and thresholding.
• A monostable circuit was used to prevent multiple triggering
•The piezo’s signal was also processed using a separate amplifying and smoothing circuit to give a hit’s velocity
Sensors – Flex Sensors
+
-
15V
-15V
15V
1kΩ
1kΩ
Diffe re ntial Amplife r
+
-7kΩ
10kΩ+
-
+
-
1kΩ
1kΩ
47kΩ
fex sensor
(11V to 8V depending on fex sensor)
4k7Ω
5k6Ω
(8V fxed)15V
-15V
15V
15V
-15V
15V
-15V
• Flex sensors were embedded inside the brushes to measure the force applied by the drummer.
• Signal Processing is used to convert the variable voltage from the flex sensor into a voltage between 0V and 5V.
Sensor Circuits – Footswitches and PCB
10kΩ
resetfootswitch 1
from piezo velocity circuitfrom flex sensor 1 circuitfrom flex sensor 2 circuit
from monostable
serial data out
footswitch 2
4MHz
• Two footswitches were used to provide the performer with additional control of the output system
• These signals were input into the digital inputs of the PIC
• The other sensor signals were input into the analogue inputs of the PIC
PIC Program Overview
Read ADC for analogue input 1 (fex-sensor1)
START
Break data into 2 nibbles and convert into ASCII equivalent
Transmit header ‘m’ and 2 nibbles
Read ADC for analogue input 2 (fex-sensor2)
Break data into 2 nibbles and convert into ASCII equivalent
Transmit header ‘n’ and 2 nibbles
Is RB1 high (footswitch 1)?
Break piezo data into 2 nibbles and convert into ASCII equivalent
Transmit header ‘p’ and 2 nibbles
Yes
No
Transmit ‘S’
Read ADC for analogue input 0 (piezo velocity)
Is RB2 high (footswitch 2)?
Yes
NoTransmit ‘T’
Check Flags
START ISR
Yes
NoIs timer0 flag set?
ERROR
Output ‘X’
Reset timer0 to FF
Reset flags
END ISR
PIC data encoding
START
Read frst 4-bits of data
Is high nibble <
10?
+0x30
+0x37
Read second 4-bits of data
Is low nibble <
10?
+0x30
+0x37
END
Conve rts to as cii characte rs e quivale nt of he x value
(nibl = data % 16)
(nibh = data / 16)
Serial data transmission
109 53 88 53 110 65 55 112 56 69 109 53 53 110 65 55 112 58 52 m 5 X 5 n A 7 p 8 E m 5 5 n A 7 p 9 4
109 53 53 110 65 56 112 56 55 83 109 53 53 110 65 65 112 52 66 83 m 5 5 n A 8 p 8 7 S m 5 5 n A A p 4 B S
Characters transmitted before sensor data to identify sensor.
Data stream can be searched for identifiers and sensor data extracted
Two nibbles need to be combined to give the 8 bit sensor value
Example Datastream:
PIC data decoding
Mapping Overview
Flex Sensor 1
Flex Sensor 2
Piezo Camera
Brush area
Brush x position
Brush y position
Flex value
Flex value
Velocity Hit
Drum Synthesise
r
PC – data processing in Max/MSP and J itter
Audio Looper
Drum Sample
Triggering
Abstract Synthesise
r
Video Manipulation
inputs
outputs
Foot-switche
s
The Audio Interface 1 - Drum Sample Playback
•Numerous audio samples from a snare drum were recorded to create a large bank of samples.
•When the piezo detects a hit, a sample of a similar velocity is selected at random
•A change in flex sensor data triggers a looping sweep sample.
The Audio Interface 2 - Drum Synthesiser
•The harmonic content of snare drum samples was analysed and the results used to determine the components of the synthesiser.
•The interrupt data from the piezo is used to trigger the “hit” sound, whilst the velocity data controls its amplitude.
•Motion tracking data used to control the overall volume of sweeps.
The Audio Interface 3 – Abstract Synthesiser
•The user can also play the drum as a melodic instrument, using the sensor data to control pitch and timbre.
•The piezo velocity is used to control the frequency and volume of the hit.
•For the sweep sounds, the x co-ordinates of the brushes control the pitch of the sound heard, whilst the y co-ordinates control the volume.
Audio Interface 4 – Record and Playback Module
•The first foot switch allows the performer to record samples in real-time, which are stored in a buffer.
•Recordings can be made either internally, or from an external source.
•The second footswitch can be used to set the playback speed
•The user can play over the top of the recorded sample by switching on another of the audio modules.
•The patch bay allows the user control over data mapping
Video Modules
•Provides a graphical output to enhance the performance using images from the webcam and an external source.
•Maps the sensor data to video processing effects such as hue, brightness and saturation.
Latency Testing
• Average latency of 193ms
• Jitter of 120ms
Validation
A questionnaire was given to a sample of drummers and non-drummers to evaluate the success of the project.
Results comparing drummers and non-drummers
Overall Mean Result
Conclusions
• Sensors were successfully used to capture data from a drum performance, including:
• the velocity of hits, • the flex of the brushes• the patterns created on the drum skin• footswitches
• Data was mapped in Max/MSP, allowing for the manipulation of audio and video parameters.
• The final system provides the drummer with control over the output through the following modules
• drum sample playback• realistic synthesiser• abstract synthesiser• record and playback of samples• video manipulation
• Validations were carried out to evaluate the functionality of the Augmented Drum.
• Results reflected the success of the video output and the drum as a novel surface.
Further Developments
•System latency improvements
•The refinement of the realistic synthesiser and video module.
•The addition of alternative sensors including inclinometers and accelerometers
•Adaptation to allow for the detection of hand gestures as a control