multimedia programming 26: playing with video departments of digital contents sang il park
TRANSCRIPT
Multimedia Programming 26:Playing with Video
Departments of Digital Contents
Sang Il Park
Outline• Playing with Video
OpenCV Video I/O• OpenCV 의 Video Reading Functions
– CvCapture– cvCaptureFromFile– cvCaptureFromCAM– cvReleaseCapture– cvQueryFrame– cvGetCaptureProperty– cvSetCaptureProperty
OpenCV Video I/O• OpenCV 의 Video Writing Functions
– cvCreateVideoWriter– cvReleaseVideoWriter– cvWriteFrame
CvCapture• 비디오를 저장하는 데 사용하는 구조체• Image 경우의 IplImage 와 비슷
cvCaptureFromFile• 동영상 파일을 열어서 CvCapture 구조체에 저장
• 동영상 열기에 실패시 NULL 을 반환• 이미지 경우의 cvLoadImage 와 비슷 • Example)
CvCapture * cvCaptureFromFile(const chat * filename)CvCapture * cvCaptureFromFile(const chat * filename)
CvCapture * video = cvCaptureFromFile(“c:\\video.avi”);if(video == NULL){
printf(“fail in loading video!”);return -1;
}
CvCapture * video = cvCaptureFromFile(“c:\\video.avi”);if(video == NULL){
printf(“fail in loading video!”);return -1;
}
cvCaptureFromCAM• 캡코더 ( 웹캠 ) 으로부터의 동영상 스트리밍을
CvCapture 구조체에 저장
• index 는 캠코더가 여러 개 있을 경우의 번호일반적으로 -1
• Example)
CvCapture * cvCaptureFromCAM(int index)CvCapture * cvCaptureFromCAM(int index)
CvCapture * video = cvCaptureFromCAM(-1);CvCapture * video = cvCaptureFromCAM(-1);
cvReleaseCapture• CvCapture 의 메모리를 해제
• 이미지 경우의 cvReleaseImage 와 비슷
• Example)
void cvReleaseCapture (CvCapture ** capture)void cvReleaseCapture (CvCapture ** capture)
CvCapture * video = cvCaptureFromFile(“c:\\video.avi”);
….
cvReleaseCapture(&video);
CvCapture * video = cvCaptureFromFile(“c:\\video.avi”);
….
cvReleaseCapture(&video);
cvQueryFrame• 현재 장면 (frame) 을 나타내는 이미지의 위치
반환하고 다음 장면으로 이동
– CvCapture 내에 내부적으로 저장된 위치값을 나타내며 지우거나 변화를 주면 안된다 !
– 편집을 원할시는 이미지를 복사하여 편집한다
• Examples)
IplImage* cvQueryFrame(CvCapture * capture)IplImage* cvQueryFrame(CvCapture * capture)
IplImage * frame = cvQueryFrame(video);IplImage * image = cvCreateImage(cvGetSize(frame),8,3);
cvCopy(frame, image);
IplImage * frame = cvQueryFrame(video);IplImage * image = cvCreateImage(cvGetSize(frame),8,3);
cvCopy(frame, image);
cvGetCaptureProperty• 동영상 ( 캡춰 ) 의 여러 특징 값들을 읽어옴
• ID 는 아래의 값들 중 하나
double cvGetCaptureProperty(CvCapture *, int ID)double cvGetCaptureProperty(CvCapture *, int ID)
CV_CAP_PROP_POS_MSEC 영상의 현재 시간적 위치 ( 밀리 세컨드 단위 )
CV_CAP_PROP_POS_FRAMES 현재 영상의 시간적 위치 ( 프래임 수 )
CV_CAP_PROP_POS_AVI_RATIO 현재 영상의 상대적 시간 위치 (0= 시작 , 1= 끝 )
CV_CAP_PROP_FRAME_WIDTH 영상의 넓이 (width)
CV_CAP_PROP_FRAME_HEIGHT 영상의 높이 (height)
CV_CAP_PROP_FPS 영상의 프레임 레이트 ( 초당 프레임수 )
CV_CAP_PROP_FOURCC 코덱의 이름 (four character codec name)
CV_CAP_PROP_FRAME_COUNT 전체 프레임 수CV_CAP_PROP_BRIGHTNESS 카메라 밝기 (only for cameras) CV_CAP_PROP_CONTRAST 카메라의 대비 (only for cameras) CV_CAP_PROP_SATURATION 카메라의 saturation(only for cameras) CV_CAP_PROP_HUE 카메라의 Hue (only for cameras)
cvSetCaptureProperty• 동영상 ( 캡춰 ) 의 여러 특징 값들을 설정 ( 변경 ) 함
• ID 는 다음 값들 중 하나
cvSetCaptureProperty(CvCapture *, int ID, double value)cvSetCaptureProperty(CvCapture *, int ID, double value)
CV_CAP_PROP_POS_MSEC 영상의 시간적 위치 변경 ( 밀리세컨드 단위 ) (only for video files)
CV_CAP_PROP_POS_FRAMES 영상의 시간적 위치 변경 (only for video files)
CV_CAP_PROP_POS_AVI_RATIO 영상의 상대적 시간 위치 변경 (0= 시작 , 1= 끝 ) (only for video files)
CV_CAP_PROP_FRAME_WIDTH 영상의 넓이 변경 (only for cameras) CV_CAP_PROP_FRAME_HEIGHT 영상의 높이 변경 (only for cameras) CV_CAP_PROP_FPS 영상의 프레임레이트 변경 (only for cameras) CV_CAP_PROP_FOURCC 영상의 코덱 변경 (only for cameras) CV_CAP_PROP_BRIGHTNESS 영상의 밝기 변경 (only for cameras) CV_CAP_PROP_CONTRAST 영상의 대비 변경 (only for cameras) CV_CAP_PROP_SATURATION 영상의 saturation 변경 (only for cameras) CV_CAP_PROP_HUE 영상의 Hue 변경 (only for cameras)
실습• 동영상 한 프래임 읽기
• 동영상 재생
• 2 개의 동영상을 한꺼번에 재생
• 동영상 간의 부드러운 트랜지션 (cross dissolve)
CvVideoWriter• 비디오를 저장할 때 사용하는 구조체• Image 경우의 IplImage 와 비슷
cvCreateVideoWriter• 비디오를 저장할 공간을 마련함
• Image 경우의 cvCreateImage 와 비슷
• fourcc: four character codec name ( 뒷페이지 참조 )
• size: 화면 크기• fps: 초당 프래임수 ( 보통 25 또는 30)• is_color: 칼라로 저장인지 아닌지 정한다 .
(1= color, 0=gray)• 코덱이 잘못된 경우 NULL 을 반환
CvVideoWriter * cvCreateVideoWriter(char * filename, int fourcc, double fps, CvSize size, int is_color)CvVideoWriter * cvCreateVideoWriter(char * filename, int fourcc, double fps, CvSize size, int is_color)
Four Character Codec Name• codec 이름을 4 자리 character 로 표현• Examples)
– 0 : 무압축– CV_FOURCC('P','I','M','1') : MPEG-1 코덱– CV_FOURCC('M','J','P','G') : Moving JPEG 코덱– CV_FOURCC(‘I',‘4',‘2',‘0') : Intel Indeo 4 코덱– CV_FOURCC(‘H’,’2’,’6’,’1’) : Intel H-261 코덱– CV_FOURCC(‘H’,’2’,’6’,’3’) : Intel H-261 코덱– CV_FOURCC(‘D',‘I',‘V',‘X') : Divx 코덱
잘 선택되지 않으므로 보통 무압축 선택 후 외부 프로그램에서 압축
cvReleaseVideoWriter• 비디오 쓰는 것을 끝낸다 .
• Image 경우의 cvReleaseImage 와 비슷
void cvReleaseVideoWriter (CvVideoWriter ** writer)void cvReleaseVideoWriter (CvVideoWriter ** writer)
cvWriteFrame• 현재 프레임에 그림 한 장을 추가한다 . 추가 뒤에는
한 프레임이 자동으로 진행된다
int cvWriteFrame (CvVideoWriter * writer, IplImage *img)int cvWriteFrame (CvVideoWriter * writer, IplImage *img)
실습• 동영상 copy
• 두 개의 동영상 간의 부드러운 transition 저장
VirtualDub• 대표적인 공짜 비디오 편집 프로그램• www.virtualdub.org
– 기본적인 비디오 압축 , 변환 등을 손쉽게 수행– 여러가지 작업을 필터들의 조합으로 수행
• 자신만의 필터를 만들 수 있도록 sdk 가 공개되어 있음
VirtualDub 을 이용한 압축• video > compression…
• 여러 코덱 중 선택
VirtualDub 을 이용한 편집• 그림의 축소 , 확대 , 블러링 등의 다양한 효과를
줄 수 있다 .
실습 • 만든 동영상 코덱 변환
그 동안 배운 것들fastforward
What you’ve learned so far:• OpenCV• Image Processing
– Image filtering• Brightness/Contrast• Gamma• Histogram Equalization• Blur Filtering• Unsharp Filter
– Image Warping• Scaling/Rotation/Shearing• Translation• Recovering Transformation
• Image Morphing
HelloCV
#include "stdafx.h"#include <cv.h>#include <cxcore.h>#include <highgui.h>
int _tmain(int argc, _TCHAR* argv[]){
IplImage * img;img = cvLoadImage("d:\\test.jpg");
cvNamedWindow("HelloCV");cvShowImage("HelloCV", img);
cvWaitKey();
cvDestroyWindow("HelloCV");cvReleaseImage(&img);
return 0;}
• IplImage
• cvLoadImage• cvReleaseImage
• cvNamedWindow
• cvShowImage• cvDestroyWindo
w
• cvWaitKey
Image structure
• IplImage (Image Processing Library)• typedef struct _IplImage {
int nSize; /* size of iplImage struct */int ID; /* image header version */int nChannels;int alphaChannel;int depth; /* pixel depth in bits */char colorModel[4];char channelSeq[4];int dataOrder;int origin;int align; /* 4- or 8-byte align */int width;int height;struct _IplROI *roi; /* pointer to ROI if any */struct _IplImage *maskROI; /*pointer to mask ROI if any */void *imageId; /* use of the application */struct _IplTileInfo *tileInfo; /* contains information on tiling*/int imageSize; /* useful size in bytes */char *imageData; /* pointer to aligned image */int widthStep; /* size of aligned line in bytes */int BorderMode[4]; /* the top, bottom, left, and right border mode */int BorderConst[4]; /* constants for the top, bottom,left, and right border */char *imageDataOrigin; /* ptr to full, nonaligned image */
} IplImage;•
Image I/O IplImage* cvLoadImage(image_path, colorness_flag);
loads image from file, converts to color or grayscle, if need, and returns it (or returns NULL).
image format is determined by the file contents.#define CV_LOAD_IMAGE_COLOR 1 #define CV_LOAD_IMAGE_GRAYSCALE 0 #define CV_LOAD_IMAGE_UNCHANGED -1 DEFAULT
cvSaveImage(image_path, image);saves image to file, image format is determined from extension.
cvReleaseImage(image_path, image);releases memory
BMP, JPEG, PNG, TIFF, PPM/PGM formats are supported.IplImage* img = cvLoadImage(“picture.jpeg”,-1);
if( img ) cvSaveImage( “picture.bmp”, img );
Windows
• cvNamedWindow(window_name, fixed_size_flag);creates window accessed by its name. Window handles repaint, resize events. Its position is remembered in registry:cvNamedWindow(“ViewA”);cvMoveWindow(“ViewA”,300,100);cvDestroyWindow(“ViewA”);…
• cvShowImage(window_name, image);copies the image to window buffer, then repaints it when necessary. {8u|16s|32s|32f}{C1|3|4} are supported.
only the whole window contents can be modified. Dynamic updates of parts of the window are done using operations on images, drawing functions etc.
• cvDestroyWindow(window_name);deletes the window with the given name
User Input
• int cvWaitKey( int delay=0 )waits for a pressed key. After waiting for the given delay, it proceeds. Zero delay means waiting forever until user input. – Delay in milliseconds.
Good for animating something
• Example) swapping between two images
Mouse Callback• Two things you have to do:
– Implementation• Define what you are going to do when events happen
– Setting (registration)• Let the OpenCV know which one is the callback function
void cvSetMouseCallback(window_name, yourFunction)
void yourFunction (int event, int x, int y, int flags, void *param);
Draw Line
void cvLine( IplImage, CvPoint pt1, CvPoint pt2, CvScalar color, int thickness=1);
void cvLine( IplImage, CvPoint pt1, CvPoint pt2, CvScalar color, int thickness=1);
• Draw Line from pt1 to pt2 with the given thickness
• CvPoint: a structure for storing 2D position
– Use cvPoint(x,y) for a quick usage.
struct CvPoint{
int x; // x-coordinateint y; // y-coordinate
}
struct CvPoint{
int x; // x-coordinateint y; // y-coordinate
}
Draw Rectanglevoid cvRectangle( IplImage, CvPoint pt1, CvPoint pt2, CvScalar color, int thickness=1 )
void cvRectangle( IplImage, CvPoint pt1, CvPoint pt2, CvScalar color, int thickness=1 )
• Draw a rectangle with two opposite corners pt1 and pt2
pt1
pt2
What happens if thickness is -1?
Draw Circlevoid cvCircle( IplImage, CvPoint center, int radius, CvScalar color, int thickness=1 )
void cvCircle( IplImage, CvPoint center, int radius, CvScalar color, int thickness=1 )
• Draw a circle with given center and radius
centerradius
Image Processing
• image filtering: change range of image• g(x) = h(f(x))
• image warping: change domain of
image• g(x) = f(h(x))
hf g
hf g
Alexei Efros
Image Filtering 1
• Brightness/contrast
input
ou
tpu
t
g = Af + B
g = input color valuef = output color value
A = contrast value ( 초기값 = 1)B = brightness value ( 초기값 = 0)
• Gamma
Image Filtering 2
• Histogram Equalization:Modify the image to have a well-distributed histogram
Image Filtering 3
Image Filtering 3
• Histogram Equalization Procedure:For each color channel (R, G, B)– 1. Compute the histogram
– 2. Compute the cumulative histogram
– 3. Set the maximum value as 255
– 4. Using the cumulative histogram as a mapping function
255
192
128
64
0
Image Filtering 4:
Cross-correlation filtering:
or it can be written as:
H is called the “filter,” “kernel,” or “mask.”
Image Filtering 4-1
• Blur filter: 3x3 Mean kernel
1/9 1/9 1/9
1/9 1/9 1/9
1/9 1/9 1/9
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
u -1 0 1v
-1
0
1
Image Filtering 4-2
• Blur filter: 3x3 Gaussian kernel
1 2 1
2 4 2
1 2 1
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Image Filtering 4-3
• Median Filter: selecting the median intensity in the window.
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0
0 90 0
0 0 0
1 2 3 4 5 6 7 8 90 0 0 0 0 0 0 0 90
Median( 중간값 )
Image Filtering 5:
• Unsharp Masking
=+
blurred difference original
200 400 600 800
100
200
300
400
500
- =
original blurred difference
Image Processing
• image filtering: change range of image• g(x) = h(f(x))
• image warping: change domain of
image• g(x) = f(h(x))
hf g
hf g
Alexei Efros
Image Warping
•Homogeneous coordinates– represent coordinates in 2 dimensions with a 3-
vector
1
coords shomogeneou y
x
y
x
( 동차좌표 )
• Basic 2D transformations as 3x3 matrices
1100
0cossin
0sincos
1
'
'
y
x
y
x
1100
10
01
1
'
'
y
x
t
t
y
x
y
x
Translate
Rotate
1100
00
00
1
'
'
y
x
s
s
y
x
y
x
Scale
Image Warping
• Affine transformations: – Combinations of …
• Linear transformations, and• Translations
– Properties of affine transformations:• Origin does not necessarily map to origin• Lines map to lines• Ratios are preserved• Parallel lines remain parallel• Closed under composition• Models change of basis
wyx
fedcba
wyx
100''
( 유사변환 )
Image Warping
• Projective transformations: – Combinations of..
• Affine transformations, and• Projective warps
– Properties of projective transformations:• Origin does not necessarily map to origin• Lines map to lines• Ratios are not preserved• Parallel lines do not necessarily remain parallel• Closed under composition• Models change of basis
( 사영변환 )
Image Warping
wyx
ihgfedcba
wyx
'''
• Matrix Composition
wyx
sysx
tytx
wyx
1000000
1000cossin0sincos
1001001
'''
p’ = T(tx,ty) R() S(sx,sy) p
Sequence of composition1. First, Scaling2. Next, Rotation3. Finally, Translation
Image Warping
• Keyframe animation:Interpolation of state vector:
)1( 10 vvv
1-α α
0v1v
α
Image Warping
Image Morphing: Cross-Dissolve
Interpolate whole images:Imagehalfway = (1-t)*Image1 + t*image2
This is called cross-dissolve in film industry
But what is the images are not aligned?
Image Morphing : cross-disolve2
Align first, then cross-dissolve– Alignment using global warp – picture still valid
General Image MorphingProcedure
1. Create an intermediate shape (by interpolation)2. Warp both images towards it3. Cross-dissolve the colors in the newly warped
images
Compositing Procedure
1-α α
다중 합성 : Multiple Compositing
1. 스프라이트 추출 ( 예 : 포토샵의 Intelligent Scissors 활용 )
Composite by David Dewey
2. 올바른순서로 합성한다 . ( 배경부터 차래 대로 )
매팅 문제 : 수학적 정의
이미지의 모든 픽셀에 대해 주어진 것 : - 배경색 Ck = [Rk Gk Bk] - 합성된 색 C = [R G B] 계산할 것 : - 전경 ( 물체 ) 색
Co = [Ro Go Bo αo]
- 주어진 조건
매팅 방정식 만족 :C = αoCo + (1-αo)Ck
삼각화 매팅 (Smith & Blinn)
매팅 방정식 :
C = αoCo + (1-αo)Ck
구해야 할 수의 개수를 줄이는 대신방정식의 수를 늘리면 어떨까 ?
물체는 같지만 배경의 색을 달리하여 여러 번측정 한다면 ?
생각해 봐야 할점 : 얼마나 많은 방정식 ( 배경색 ) 이 필요할까 ? 배경은 단색이어야 할까 ?
스프라이트 애니메이션 :시간에 따라 스프라이트를 선택하여 랜더링
주어지는 것 !
sprite.jpg spritea.jpg
background.jpg
OpenCV Video I/O• OpenCV 의 Video Reading Functions
– CvCapture– cvCaptureFromFile– cvCaptureFromCAM– cvReleaseCapture– cvQueryFrame– cvGetCaptureProperty– cvSetCaptureProperty
OpenCV Video I/O• OpenCV 의 Video Writing Functions
– cvCreateVideoWriter– cvReleaseVideoWriter– cvWriteFrame
한 학기 간 수고하셨습니다 .