boxes in a hurry - openvibeopenvibe.inria.fr/.../uploads/2014/10/hacking_boxes.pdfj.t. lindgren:...
TRANSCRIPT
1
J. T. Lindgren / OpenViBE 1
Jussi T. Lindgren, PhD [email protected]
HYBRID / Inria Rennes
HACKING
BOXES
IN A HURRY
2
J. T. Lindgren / OpenViBE 2 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 2
Boxes?
Making a prototype box
C++ boxes, anyone?
Inside your C++ box
Everything is not a box
Contents
3
J. T. Lindgren / OpenViBE 3 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 3
Boxes?
4
J. T. Lindgren / OpenViBE 4 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 4
Recall: signal processing chains --
It’s made of boxes
Signal processing boxes
offered by OpenViBE...
...potentially
allow you to do
1001 things already
OpenViBE Designer
5
J. T. Lindgren / OpenViBE 5 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 5
• Try to use existing boxes in a clever way
• Make a wish
• Hire an engineer
• Custom order from a company...
• Do a new box yourself (this talk)
But you want to do thing #1002...
6
J. T. Lindgren / OpenViBE 6 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 6
Input(s)
Output(s)
What is a box
Box Streams have types
(signal, stimuli, etc...)
Box::process()
{
// get input
// do things
// put output
}
Signal processing happens here
(in C++)
7
J. T. Lindgren / OpenViBE 7 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 7
• 1. Prototype box with Matlab or Python wrapper
Or, in C++
• 2. Modify an existing box
• 3. Use Skeleton Generator to get a bare-bones box
Three ways to a box
8
J. T. Lindgren / OpenViBE 8 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 8
Making a prototype box
9
J. T. Lindgren / OpenViBE 9 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 9
• Use existing wrapper boxes
• Pass data between OpenViBE Matlab or Python
• Actual processing by custom .py or .m script
• Box::process() calls script for each chunk of data
• Pros & cons • + fast to prototype, no compilation or restart needed
• + no need for compiler
• - depends on matlab or python
• - potential issues if matlab/python versions change
Prototyping with Scripting Boxes
10
J. T. Lindgren / OpenViBE 10 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 10
• Make three scripts: initialize.m, process.m, uninitialize.m
• Init/uninit called at the start/end of the playback
• Example of process.m:
• box_in is data from Designer, box_out goes to Designer
Prototyping: Matlab scripting box
function box_out = process(box_in)
disp(’process called.')
% do things to box_in
box_out = box_in;
end
11
J. T. Lindgren / OpenViBE 11 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 11
Prototyping: Matlab scripting box, contd.
% do things to box_in
% we iterate over the pending chunks on input 1 (here SIGNAL)
for i = 1: OV_getNbPendingInputChunk(box_in,1)
% pop the first chunk to be processed, box_in used as output
[box_in, start_time, end_time, matrix_data] =
OV_popInputBuffer(box_in,1);
% signal is in matrix_data, [channels X samples] orientation
% …
% replicate header
box_in.outputs{1}.header = box_in.inputs{1}.header;
% The chunk dates are the same as the input chunk
box_in = OV_addOutputBuffer(box_in,1,start_time,end_time,matrix_data);
end
box_out = box_in;
12
J. T. Lindgren / OpenViBE 12 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 12
• Examples & tutorials: • box-tutorials/python/
• box-tutorials/matlab/
• http://openvibe.inria.fr/tutorial-using-python-with-openvibe/
• http://openvibe.inria.fr/tutorial-using-matlab-with-openvibe/
Prototyping: more info
13
J. T. Lindgren / OpenViBE 13 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 13
C++ boxes, anyone?
14
J. T. Lindgren / OpenViBE 14 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 14
• Pros & cons • + C++ is the implementation language of OpenViBE
• + no external dependencies needed
• + code can be as fast as you can make it
• + no added python/matlab compatibility issues
• - needs a compiler (free ones are ok)
• - modifications require compilation and restart
• - C++ prototyping can be slow
• Getting started with C++
http://openvibe.inria.fr/build-instructions
C++ boxes
15
J. T. Lindgren / OpenViBE 15 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 15
Simple:
• Decide an old box that is close to the idea of your new box
• Locate its source code from the source tree (’plugins/’)
• Hack away
Your starting point:
• Has some ’interface glue’ already in place and works
• May not have everything you need
• May have things you don’t need
A. Modifying an existing C++ box
16
J. T. Lindgren / OpenViBE 16 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 16
- A box needs 30+ lines of ’glue code’
- More for boxes having parameters, inputs, outputs, etc
- Skeleton Generator can make the basic box code
- You get a .cpp/.h pair
- Add your custom processing code to - ::initialize() { // ... }
- ::process() { // ... }
- ::uninitialize() { // ...}
B. Skeleton Generator tool
17
J. T. Lindgren / OpenViBE 17 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 17
Tabs to specify
various properties
of your box
When done:
Check, then
Generate!
18
J. T. Lindgren / OpenViBE 18 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 18
• Step-by-step tutorial
http://openvibe.inria.fr/tutorial-1-implementing-a-signal-
processing-box
Time to hack into the box...?
B. Skeleton Generator tool
19
J. T. Lindgren / OpenViBE 19 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 19
Inside your C++ box
20
J. T. Lindgren / OpenViBE 20 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 20
• Each box needs to be described with e.g.
• Box identifier (random 64bit number)
• Box name
• Author
• Inputs & outputs
• Box parameters
• ...
• These are in the box .h file
• The header may also define a ’listener’ : code that reacts to box
parameter changes when made by the user
Box description
21
J. T. Lindgren / OpenViBE 21 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 21
Box settings
Settings registered in the .h
get a GUI automatically
Save & load to scenario xml
also automatic
Getting value of an uint64 param at slot 3 (index from 0):
uint64 order
= FSettingValueAutoCast(*this->getBoxAlgorithmContext(),2);
22
J. T. Lindgren / OpenViBE 22 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 22
- Data needs to be encoded to travel between boxes
- Codecs capsulate the data in box input/output
- Each stream needs its own codec
Codecs
Box Encoders
Decoders
In from kernel
Out to kernel
23
J. T. Lindgren / OpenViBE 23 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 23
Codec example: initializing
myBox::initialize() {
// TSignalDecoder< myBox > m_signalDecoder; // in .h
// TSignalEncoder< myBox > m_signalEncoder; // in .h
m_signalDecoder.initialize(*this);
m_signalEncoder.initialize(*this);
// make decoder and encoder share buffer & sampling rate
m_signalEncoder.getInputMatrix().
setReferenceTarget(m_signalDecoder.getOutputMatrix());
m_signalEncoder.getInputSamplingRate().
setReferenceTarget(
m_signalDecoder.getOutputSamplingRate());
}
24
J. T. Lindgren / OpenViBE 24 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 24
Codec example: processing
myBox::process() {
for(uint32 i=0; i<nChunks; i++) {
// decode the chunk i on input 0
m_signalDecoder.decode(0,i);
if(m_signalDecoder.isBufferReceived()) {
IMatrix* l_pMatrix = m_signalDecoder.getOutputMatrix();
// do stuff with the data in l_pMatrix, [chns x samples]
// ...
// send out
m_signalEncoder.encodeBuffer(0);
l_rDynamicBoxContext.markOutputAsReadyToSend(0,
l_rDynamicBoxContext.getInputChunkStartTime(0, i),
l_rDynamicBoxContext.getInputChunkEndTime(0, i));
}
}
25
J. T. Lindgren / OpenViBE 25 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 25
Everything is not a box
26
J. T. Lindgren / OpenViBE 26 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 26
• Device driver is not a box • Put it in Acquisition Server instead
• Algorithm is not a box • If a routine is shared by several boxes, consider it an ’Algorithm’
• Stimulators may not want to be boxes • Should be external apps outside Designer
• Send their triggers to Acquisition Server
• For timing reasons, avoid making stimulators as boxes
27
J. T. Lindgren / OpenViBE 27 J.T. Lindgren: Hacking OpenViBE boxes Sept 2014 27
Here we have skipped a lot of details
Box coding is best learned by doing
More about at
http://openvibe.inria.fr
That’s it!