sounds in bada

Copyright © 2010 Samsung Electronics Co., Ltd. All rights reserved.

Sounds in bada

1


Contents

Introduction

Common Sound Classes– AudioRecorder/Player– AudioIn/AudioOut

Simple Mixing

Media Management– Image– Video– Streaming

Summary2

*This material is based on bada SDK 1.0.0b3


Media Introduction

Audio/ video/ image processing functions

Key features

3

• Playback• Record• Streaming• PCM processing

Audio

• Playback• Record• Streaming• Camera capture• Camera preview

Video

• Load• Save• Format conversion

Image


Supported FormatsType Format Encode Decode Streaming*Video H.263, MPEG-4 (.3gp, mp4) O O O

H.264 (.3gp, mp4) O OVC-1 (.wmv, .asf) O

Audio PCM (.wav) O OAMR-NB (.amr, .3ga, .m4a) O O OAAC, AAC+, EAAC+ (.aac, .3ga, .m4a)

O O

MIDI (.mid, .spm, .imy, .mmf, .xmf) OMP3 (.mp3), WMA ( .wma, .asf) O

Image JPEG, PNG, BMP O OGIF, TIFF, WBMP O

*Streaming Protocol: RTSP4


Header & Library

Header file: – #include <FMedia.h>

Library – Simulator: FMedia – Target: FOsp

Privilege groups– RECORDING– IMAGE– CAMERA

5


Common Sound Classes

6


Basic method:General purpose

Complex method:Direct access to device

Audio Recorder & Player

*PCM: Pulse Code Modulation 7

Recording– AudioRecorder : Records

audio data from input source to storage

Playing– Player : Open audio file

and play

Recording– AudioIn : Read raw PCM

data from input srouce

Playing– AudioOut : Play sound with

raw PCM data


Demo

Playback– Player– AudioOut

8


Basic method

9


AudioRecorder class

Records audio data from input source to storage– Recording format

• WAV, AMR

Event– IAudioRecorderEventListener

• OnAudioRecorderPaused• OnAudioRecorderStarted• OnAudioRecorderStopped• OnAudioRecorderClosed• OnAudioRecorderCanceled• …

"RECORDING" privilege is needed10


Using the AudioRecorder class

Create AudioRecorder instance

Recording

Stop recording & close

__pAR = new AudioRecorder;__pAR->Construct(*pListener);

// IAudioRecorderEventListener

__pAR->CreateAudioFile( __path, true);__pAR->SetFormat(AUDIORECORDING_FORMAT_WAVE);__pAR->Record();

__pAR->Stop();…void EventListener::OnAudioRecorderStopped( result r){

__pAR->Close();}

11


Player class

Can play audio & video– Multiple audio playback is supported(max 10)

• Only 1 compressed audio files– Can play audio/video by streaming

• Streaming Protocol: Real-Time Streaming Protocol (RTSP)

Event– IPlayerEventListener

• OnPlayerBuffering, OnPlayerInterrupted• OnPlayerEndOfClip, OnPlayerErrorOccurred• OnPlayerOpened, OnPlayerReleased• OnPlayerSeekCompleted

12


Using the Player class

Create Player instance

Load file & play

Stop

__pPlayer = new Player;__pPlayer->Construct(*pListener);

// IPlayerEventListener

result r = __pPlayer->OpenFile(__path);if (IsFailed(r)) return;__pPlayer->Play();

__pPlayer->Stop();__pPlayer->Close();

13


Complex method

14


Record PCM data

AudioIn class– Record raw uncompressed PCM data from

microphone– Double buffering is possible

Event Listener– IAudioInEventListener

• OnAudioInBufferIsFilled• OnAudioInInterrupted• OnAudioInReleased

15


Using the AudioIn class

Create AudioIn instance

Feed buffers and start recording

Feed next buffers

__pAudioIn = new AudioIn();__pAudioIn->Construct(*__pListener);__pAudioIn-> Prepare (AUDIO_INPUT_DEVICE_MIC, AUDIO_TYPE_PCM_U8, AUDIO_CHANNEL_TYPE_STEREO,8000);

__pAudioIn->AddBuffer(__pByteBuffer1);__pAudioIn->Start();

void Listener::OnAudioInBufferIsFilled( Osp::Base::ByteBuffer* pData){ // save result.

pData->Clear();__pAudioIn->AddBuffer(pData);

16


Play PCM data

AudioOut– Play raw uncompressed PCM data– Double buffering is possible

Event listener– IAudioOutEventListener

• OnAudioOutBufferEndReached• OnAudioOutInterrupted• OnAudioOutReleased

17


Using the AudioOut class

Create AudioOut instance

Feed PCM data

Play sound

__pAO = new AudioOut();__pAO->Construct(*pListener);// IAudioOutEventListener__pAO->Prepare(AUDIO_TYPE_PCM_S16_LE,

AUDIO_CHANNEL_TYPE_MONO, SAMPLERATE);__pOutBuffer = new ByteBuffer();__pOutBuffer->Construct(BUFSIZE);

// Fill buffer__pAO->WriteBuffer(*__pOutBuffer);

__pAO->Start();

18


Using the AudioOut class ctd.

Process events

Stop sound__pAO->Stop(); //or Reset()

void EventListener::OnAudioOutBufferEndReached(Osp::Media::AudioOut &src){ //Feed Next PCM data }

19


Double buffering in AudioIn

20

AudioIn DeviceStep 1. Initialize AudioIn Recording

application



21

AudioIn DeviceStep 1. Initialize AudioIn

Step 2. Add two buffers

Buffer1

Buffer2

Recordingapplication



22



Step 3. Start recording

Step 4. Buffer1 is filled

Buffer1

Buffer2




23





Step 5. Event is fired Buffer2 is used by AudioIn

Buffer1

Buffer2


Event: Buffer is filled



24






Step 6. App. reads Buffer1and AudioIn writes to Buffer2

Buffer1

Buffer2




25







Buffer1

Buffer2


Event: Buffer is filled

Step 7. Event is firedAudioIn switches to Buffer1



26







Buffer1

Buffer2


Step 7. Event is firedAudioIn switches to Buffer1Step 8. Read Buffer2


Double buffering in AudioOut

27

AudioOut DeviceStep 1. Initialize AudioOut Player

application



28

AudioOut DeviceStep 1. Initialize AudioOut


Buffer1

Buffer2

Playerapplication



29



Step 3. Start playback

Step 4. Buffer1 is read

Buffer1

Buffer2

Playerapplication



30





Step 5. Event is fired Buffer2 is used by AudioOut

Buffer1

Buffer2

Event: Buffer end is reached

Playerapplication



31






Step 6. AudioOut reads Buffer2 and app writes to Buffer1

Buffer1

Buffer2

Playerapplication



32







Buffer1

Buffer2

Event: Buffer end is reached

Step 7. Event is firedAudioOut switches to Buffer1

Playerapplication



33







Buffer1

Buffer2

Step 7. Event is firedAudioOut switches to Buffer1Step 8. Write to Buffer2

Playerapplication


Simple Mixing

34


Why mixing?

Play many sounds simultaneously– Limited number of player instances– System resource limited

Performance issue– Consume less computing power

Flexibility– Finer control of PCM data is possible

35


.wav file

Important .wav header fields

36

Field Name

Chunk IDChuck Size

FormatSubchunk1 ID

Subchunk1 SizeAudio Format

Num ChannelsSample Rate

Byte RateBlock Align

Bits Per SampleSubchunk2 ID

Subchunk2 Size

Data

4

RIFFHeader444

WavFormatInfo

42244224

ChnkHeader4

Subc

hunk

2siz

e

Field Offset(bytes)

Field Size

(bytes)

Field ValueChannels mono/stereoSample rate 8/22/44.1KhzBit per sample 8bit /16bitSize raw data sizePCM data raw data

0

4

8

12

16

20

22

24

28

32

34

36

40

44


Loading .wav file

Open file

Read header

Read PCM data

File f;f.Construct(/*File Path*/, String(L"r"));

f.Read(&__RIFFHeader, sizeof(RIFFHEader));f.Read(&__WavFormat, sizeof(WavFormatInfo));

ChnkHeader tmp;f.Read((void *)&tmp, sizeof(ChnkHeader));__pData = new Osp::Base::ByteBuffer;__pData->Construct(tmp.size);f.Read(*__pData);__pData->Rewind();

37


Simple mixing

Step 1: Create buffers

Step 2: Add the buffers

__pTmpBuf = new ByteBuffer();__pTmpBuf->Construct(size*4); //int type

int* pTmp = static_cast<int *> (__pTmpBuf->GetPointer());

short* pData1 =static_cast<short *> (__pData1->GetPointer());

short* pData2 =static_cast<short *> (__pData2->GetPointer());

for (int i = 0; i < size; i++)pTmp[i] = pData1[i] + pData2[i] ;

38


Simple mixing

Step 3: Clip the buffers

Step 4: Write the buffer to AudioOut

short * pOut = static_cast<short *> (__pOutBuf->GetPointer());

for( int i=0;i<size;i++){

if (pTmp[i] >= SHRT_MAX) pOut[i] = SHRT_MAX*;else if ( pTmp[i] <= SHRT_MIN) pOut[i] = SHRT_MIN*;else pOut[i] = pTmp[i];

}

__pAO->WriteBuffer(*__pOutBuf);

39

*SHRT_MAX, SHRT_MIN: maximum and minimum values of short type


Image & Video management• Image

• Video

• Streaming

40


Image

Can encode, decode, convert images

Supported pixel format– RGB565/ARGB8888/R8G8B8A8

Decoding– BMP, GIF, PNG, TIFF, WBMP

Encoding– BMP, JPEG, PNG

"IMAGE" privilege is needed

41


Image class decode/encode

Load image

Save image

Bitmap* LoadBitmapN(const String& name){

Bitmap* pBitmap = null;Image* pImage = new Image();pImage->Construct();pBitmap = pImage->DecodeN(name,

BITMAP_PIXEL_FORMAT_ARGB8888);delete pImage;return pBitmap;

}

pImage->EncodeToFile(*pBitmap, IMG_FORMAT_PNG, name, true);

42


Video playing

Can play video using Player class– Using OverlayPanel to display video

Code snippet– Create player

– Open file

– Play/stop/pause/close

pPlayer = new Player();pPlayer->Construct(*pListener, &bufferInfo);

pPlayer->OpenFile(String(L"/Res/badaBI.mp4"));

pPlayer->Play(); //Pause() or Stop() or Close()

43


OverlayPanel

Primarily used for the playback of camera previews or video– Foreground panel

• Overlay graphics and controls– Background buffer

• Support H/W accelerated rendering

44

Foreground panel

Input bufferMasking color

Background buffer


OverlayPanel ctd.

Create a panel and get bufferInfo

Use the bufferInfo in the Player class

45

pPanel= new OverlayPanel();pPanel->Construct(rect);AddControl(*pPanel);pPanel->GetBackgroundBufferInfo(bufferInfo);

pPlayer->Construct(*pListener, &bufferInfo);


Streaming

Can play audio/video streaming content– Supported protocol : RTSP – Supported video codecs : MPEG4, H.263,

H.264 – Supported audio codecs : AAC, HE-AAC,

Enhanced-AAC+, AMR-NB

pPlayer->OpenURL(String(L“RTSP://IP address/content"));

46


Demo

MediaPlayer– Video Playback– OverlayPanel

47


Tips for Performance

Preffered PCM format to reduce internal operations

Double buffering to reduce latency

Sound mixing can reduce internal resources and event processing

Field ValuesChannels stereoSample rate 44.1KhzBit per sample 16bit

48


Tips for Programming

Handled automatically by platformHeadset • Plug and unplug events are not fired

• Platform changes the sound path automaticallyIncoming call

• Player, AudioIn, and AudioOut are stopped by platform• Resuming should be done by application

(Osp::App::Application::OnForeground)

49


Tips for Programming

50

Handled by applicationSilent mode • Event is not fired

• Application should check silent mode (Osp::System::SettingInfo )

Side volume key

• Application should deal with volume controls• Application handles key event

(Osp::Ui::IKeyEventListener)Foreground/ background

• Player consumes too much power• On background, stop playing audio and video


Summary

51


What we have learned

Basic way to record and play audio– AudioRecorder / Player classes

Playing and recording PCM data– AudioIn / AudioOut classes– Wav file format / Simple mixing

Image and Video management– Image encode/decode – OverlayPanel for video– Streaming both audio and video

52


Find out more

Tutorial– bada Tutorial.Media.pdf

Samples – MediaPlayer– VoiceRecorder

53


Dive intohttp://www.goprodiver.com54

sounds in bada

Education

samsung electronics

play pcm dataaudioout

raw pcm datacopyright

new player

streaming streaming

new audioout

recording audiorecorder

recording audioin