project final 23-06-099

8/6/2019 Project Final 23-06-099

http://slidepdf.com/reader/full/project-final-23-06-099 1/48

CHAPTER-1

INTRODUCTION

A virtual keyboard is where a full-size image of a QWERTY keyboard is projected onto any

surface. Touching the image of a key generates a unique electronic signal corresponding to a

key's image. Using a virtual keyboard eliminates the chance of breakage and infection

transfer. Additionally virtual keyboards require no cleaning and they have no wires, buttons,

or switches. Virtual keyboards are also compatible with many Smart phones and PDAs.

As the demand for computing environments evolves, new human-computer interfaces have

been implemented to provide multiform interactions between users and machines. Also, the

basis for most human-to-computer interactions remains the binomial keyboard/mouse.

Ordinary keyboards however, to be comfortable and effective, must be reasonably sized.

Thus they are cumbersome to carry and often require wiring. To overcome these problems, a

smaller and more mobile touch-typing device has been proposed which does not have

physical support. This device is known as virtual keyboard.

The depth information supplied by the range camera allows developing simpler and more

efficient computer-vision algorithms to estimate the position of fingertips and to locate the

corresponding stricken key. Simplicity and efficiency are key elements to enable real-time or

even portable applications

.

However, there are still some challenges associated with the range camera utilized in this

project. A number of problems, such as light scattering, and “close target” artifacts impact

achievable depth accuracy. Moreover, the image resolution of the camera is relatively low,

thus restricting its use to applications with large view window. As a result the working area

of the keyboard is limited today to a sub-optimal size. The current power consumption and

size of the range camera are also impediments to its use in truly portable applications

.

On-screen keyboards have a limited area due to the limitations of the screen, but they can

have a dynamic key layout and they can be used in darkness because of the back light. Full

1

8/6/2019 Project Final 23-06-099


text keyboards are too large to fit into a compact mobile device without the keys being

uncomfortably small. However, in this case the user gets a known keyboard that functions

exactly like he is used to.

In this project the main focus will be on implementing a traditional full text keyboard

using Augmented Reality, but future work should consider alternative text input methods.

Projection keyboard

A projection keyboard is a virtual keyboard that can be projected and touched on any surface.

The keyboard watches finger movements and translates them into keystrokes in the device.

Most systems can also function as a virtual mouse or even as a virtual piano. A proposed

system called the P-ISM will combine the technology with a small video projector to create a

portable computer the size of a fountain pen.

How a projection keyboard generally works:

1. A laser or beamer projects visible virtual keyboard onto level surface

2. A sensor or camera in the projector picks up finger movements[3]

3. detected co-ordinates determine actions or characters to be generated

Some devices use a second (invisible infrared) beam:

1. An invisible infrared beam is projected above the virtual keyboard

2. Finger makes keystroke on virtual keyboard. This breaks infrared beam and infrared

light is reflected back to projector

3. Reflected infrared beam passes through infrared filter to camera

4. Camera photographs angle of incoming infrared light

5. Sensor chip determines where infrared beam was broken

6. detected co-ordinates determine actions or characters to be generated

An optical virtual keyboard has been invented and patented by IBM engineers in 1992. It

optically detects and analyses human hand and finger motions and interprets them as operations

on a physically non-existent input device like a surface

2

http://en.wikipedia.org/wiki/Infrared

http://en.wikipedia.org/wiki/Infrared

8/6/2019 Project Final 23-06-099


The virtual keyboard consists of webcam (as a media for sensing the key strings),

Prototype keyboard and MATLAB (as a tool for interfacing). In that way it allows to emulate

unlimited types of manually operated input devices (mouse, keyboard, etc.). All mechanical

input units can be replaced by such virtual devices, optimized for the current application and

for the user's physiology maintaining speed, simplicity and unambiguity of manual data

input.

In this project the main focus will be on implementing a traditional full text keyboard

using Augmented Reality, but future work should consider alternative text input methods.

Why we selected the virtual keyboard

Virtual Keyboard is just another example of today’s computer trend of ‘smaller and faster’.

Computing is now not limited to desktops and laptops, it has found its way into mobile

devices like palm tops and even cell phones. But what has not changed for the last 50 or so

odd years is the input device, the good old QWERTY keyboard. The virtual keyboard

technology is the latest development. The virtual keyboard technology uses sensor

technology and artificial intelligence to let users work on any flat surface as if it were a

keyboard. Virtual Keyboards lets you easily create multilingual text content on almost any

existing platform and output it directly to PDAs or even web pages. Virtual Keyboard, being

a small, handy, well-designed and easy to use application, turns into a perfect solution for

cross platform text input. The main features are: platform-independent multilingual support

for keyboard text input, built-in language layouts and settings, copy/paste etc. operations

support just as in a regular text editor, no change in already existing system language settings,

easy and user-friendly interface and design, and small file size.

Problems of QUERTY keyboard

It is now recognized that it is important to be correctly seated while using a computer. A

comfortable working position will help with concentration, quality of work, and reduce the

risk of long-term problems. This is important for all who use computers, and especially so for

those with disabilities.

The increased repetitive motions and awkward postures attributed to the use of computer

keyboards have resulted in a rise in cumulative trauma disorders (CTDs) that are generally

3

8/6/2019 Project Final 23-06-099


considered to be the most costly and severe disorders occurring in the office. Lawsuits for

arm, wrist, and hand injuries have been filed against keyboard manufacturers allege that

keyboarding equipment is defectively designed and manufacturers fail to provide adequate

warnings about proper use to avoid injury.

As early as1926, Klockenberg described how the keyboard layout required the typist to

assume body postures that were unnatural, uncomfortable and fatiguing. For example,

standard keyboard design forces operators to place their hands in a flat, palm down position

called forearm pronation. The compact, linear key arrangement also causes some typists to

place their wrist in a position that is skewed towards the little fingers, called ulnar deviation.

These awkward postures result in static muscle loading, increased muscular energy

expenditure, reduced muscular waste removal, and eventual discomfort or injury. Researchers

also noted that typing on the QWERTY keyboard is poorly distributed between the hands and

fingers, causing the weaker ring and little fingers to be overwork

VIRTUAL KEYBOARD

Introduction

Virtual Keyboard is just another example of today’s computer trend of "smaller and faster".

Computing is now not limited to desktops and laptops, it has found its way into mobile

devices like palm tops and even cell phones. But what has not changed for the last 50 or so

odd years is the input device, the good old QWERTY keyboard. Alternatives came in the

form of handwriting recognition, speech recognition, abcd input (for SMS in cell phones) etc.

But they all lack the accuracy and convenience of a full-blown keyboard. Speech input has an

added issue of privacy. Even folded keyboards for PDAs are yet to catch on. Thus a new

generation of virtual input devices is now being paraded, which could drastically change the

way we type. Virtual Keyboard uses sensor technology and artificial intelligence to let users

work on any surface as if it were a keyboard. Virtual Devices have developed a flashlight-

size gadget that projects an image of a keyboard on any surface and let’s people input data by

typing on the image. The device detects movement when fingers are pressed down. Those

movements are measured and the device accurately determines the intended keystrokes and

translates them into text. The Virtual Keyboard uses light to project a full-sized computer

keyboard onto almost any surface, and disappears when not in use. The translation process

also uses artificial intelligence. Once the keystroke has been decoded, it is sent to the portable

device either by cable or via wireless.

4

8/6/2019 Project Final 23-06-099


The Virtual Keyboard uses light to project a full-sized computer keyboard onto almost any

surface, and disappears when not in use. Used with Smart Phones and PDAs, it provides a

practical way to do email, word processing and spreadsheet tasks, allowing the user to leave

the laptop computer at home. The technology has many applications in various high-tech and

industrial Sectors. These include data entry and control panel applications in hazardous and

harsh environments and medical markets.

Projection key boards or virtual key boards claim to provide the convenience of compactness

with the advantages of a full-blown QWERTY keyboard. An interesting use of such

keyboards would be in sterile environments where silence or low noise is essential like

operation theaters. The advantage of such a system is that you do not need a surface for

typing, you can even type in plain air. The company's Virtual Keyboard is designed for

anyone who's become frustrated with trying to put information into a handheld but doesn't

want to carry a notebook computer around. There is also the provision for a pause function to

avoid translating extraneous hand movements function, so that users can stop to eat, drink etc

It is also a superior desktop computer keyboard featuring dramatically easier to learn touch-

typing and leaving one hand free for mouse or phone. Combination key presses ("chords") of five main and two extra control keys allow users to type at 25-60 words per minute, with

possibly greater speeds achieved through the use of abbreviation expansion software. Most

users, however, will find memorizing the chords easy and fun, with the included typing

tutorial. The scanner can keep up with the fastest typist, scanning the projected area over 50

times a second. The keyboard doesn't demand a lot of force, easing strain on wrists and digits.

Virtual keyboards solve the problem of sore thumbs that can be caused by typing on the tiny

keyboards of various gadgets like PDAs and cell phones. They are meant to meet the needs of

mobile computer users struggling with cumbersome, tiny, or nonexistent keyboards. It might

help to prevent RSI injuries.

The Virtual Keyboard uses an extremely durable material which is extremely easy to clean.

The Virtual Keyboard is not restricted to the QWERTY touch-typing paradigm; adjustments

can be done to the software to fit other touch-typing paradigms as well, such as the

DVORAK keyboard. It will work with all types of Bluetooth enabled devices such as PDAs

and smart phones, as well as wearable computers. Applications include computer/PDA input,

5

8/6/2019 Project Final 23-06-099


gaming control, TV remote control, and musical applications. Thus virtual keyboards will

make typing easier, faster, and almost a pleasure.

CHAPTER-2

WORKING

The virtual keyboard consists of webcam (as a media for sensing the key strings), Prototype

keyboard and MATLAB (as a tool for interfacing). The program continuously monitor for a

pressed key. The pictures/video taken by the webcam will process by the MATLAB for any

pressed key. The key press is detected by comparing the areas with a predetermined

6

8/6/2019 Project Final 23-06-099


threshold. Change in value indicates a key press. The corresponding key will be detected and

printed on a MATLAB window.

As we started working on to our project with a still camera taking photos manually, and

then transferring the digital image obtained from camera on to the working tool MATLAB

and then processing the obtained image we were able to determine the keys.

Fig1: image taken by the webcam

CHAPTER-3

PROJECT SETUP

7

8/6/2019 Project Final 23-06-099


Fig2: Project set up

Keyboard should be placed on a correct background and the light falling on the keyboard

should be uniform throughout. But in real cases this will not happen mainly due to scattering.

To reduce this scattering effect makes the surface of the keyboard flat.

The camera should focus on the keyboard exactly vertical in order to avoid resizing

problems. The unwanted resizing will ultimately affect or reduce the system performance.

The pictures taken from the camera are moved to the MATLB directory by using Bluetooth

or data cable.8

8/6/2019 Project Final 23-06-099


The CMOS camera continuously captures images of the region where the printed

keyboard is supposed to be placed and checks these images for red color data above a

specified threshold. The threshold concept works in this case because the laser shining on a

typical human finger generates saturating values of red color data, which is very easily

distinguishable from its surroundings.

CHAPTER-4

PROJECT OVERVIEW

In this paper, a complete system is presented which mimics a QWERTY keyboard on an

arbitrary (paper) surface. The system consists of a webcam and a WINDOWS real time

MATLAB program for detecting the typing events. We exploit snapshot feature of the

camera and detect the key pressed using a MATLAB. Fingertips are found by analyzing the

hand picture and compare with a reference keyboard. To detect a key press, we analyze the

hand movements through the snapshots and when we press a key the snap before the event

and after the event will be same and map it back to the reference keyboard to find which key

9

8/6/2019 Project Final 23-06-099


was pressed. These steps are fully automated by MATLAB programming and do not require

human intervention. The resulting key is getting printed on to the MATLAB window and

getting stored in a file.

The steps involved

1. Webcam interfacing and video processing

2. image acquisition, frames acquisition

3. image segmentation

4. keystroke detection

5. key detection

CHAPTER-5

WEBCAM INTERFACING AND VIDEO PROCESSING

10

8/6/2019 Project Final 23-06-099


Fig3: webcam Fig4: virtual keyboard set up

We acquire the images from camera depends on lots of software. Creating the inter phase

between computer and the camera can be done using lower level language like C or C++

will give lots of flexibility, but it will also invoke lots of work and background knowledge.

MATLAB’s image acquisition toolbox has a variety of simple functions

For Mat lab to recognize the video camera you have to create it as an object using the

command obj=videoinput(‘winvideo’). Mat lab will automatically find the webcam

connected to your computer. Once it is an object in your workspace you can edit its settings,such as the number of frames per second, to optimize it for your project. preview () and get

frame() are two useful functions for determining if the camera has been positioned properly.

The first allows you to see what the camera sees, without collecting any data from it, and the

second acquires a single snapshot and stores it as in image.

Create an image acquisition object -- This example creates a video input object for a

Webcam image acquisition device. To run this example on your system, use the imaqhwinfo

function to get the object constructor for your image acquisition device and substitute that

syntax for the following code.

vid = videoinput('winvideo',1); ie converting to static state.

preview

Display preview of live video data

Syntaxes :

11

8/6/2019 Project Final 23-06-099


preview(obj)

preview(obj,himage)

himage = preview(...)

Description

preview(obj) creates a Video Preview window that displays live video data for video input

object obj. The window also displays the timestamp and video resolution of each frame, and

the current status of obj. The Video Preview window displays the video data at 100%

magnification (one screen pixel represents one image pixel).

getsnapshot

It immediately return a single image frame

Syntax

frame = getsnapshot(obj)

Description

frame = getsnapshot(obj) immediately returns one single image frame, frame, from the video

input object obj. The frame of data returned is independent of the video input object

FramesPerTrigger property and has no effect on the value of the FramesAvailable or

FramesAcquired property.

The object obj must be a 1-by-1 video input object.

frame is returned as an H-by-W-by-B matrix where

HImage height, as specified in the ROIPosition propertyWImage width, as specified in the

ROIPosition propertyBNumber of bands associated with obj, as specified in the

NumberOfBands property

frame is returned to the MATLAB workspace in its native data type using the color space

specified by the ReturnedColorSpace property.

12

8/6/2019 Project Final 23-06-099


CHAPTER-6

IMAGE ACQUISITION AND FRAME ACQUISITION

For the image acquisition we use the preview() and getsnapshot(). Using this two functions

we get a coloured images of the keyboard and keyboard with hand motions. The each image

is subjected for conversion to black and white in order to reduce the complexity.

Resize the image to enhance the speed. By resizing the picture the number of pixels

decreases so the processor needs to consider only lesser number of pixels. Rotate the image

to properly align the keyboard.

13

8/6/2019 Project Final 23-06-099


Fig5: captured image

Fig6: Resize and conversion to black and white

14

8/6/2019 Project Final 23-06-099


CHAPTER-7

IMAGE SEGMENTATION

Segmentation means dividing the entire area into small section decided by number of

rows and columns.

The total number of segments will be equal to number of rows multiplies number of columns.

To increasing or decreasing the number of keys in keyboard we only need to change the rows

or columns. The resized image size is stored in a matrix variable.

We can use the loop statement given below to address different areas in the segmented

keyboard.

s = size(kb1); %storing the resized value into new variable s

kalfa = 1; %selecting the first selected area and putting it into loop

15

8/6/2019 Project Final 23-06-099


8/6/2019 Project Final 23-06-099


CHAPTER-8

KEYPRESS & KEYSTROKE DETECTION

The main feature of our keyboard is software enabled key-press detection. In ordinary

keyboards special voltages generation or change in circuit parameter values ( eg: resistors)

are used to detect a key-press. Whenever the parameter values corresponding to key is

happen this event calls a subroutine which will return the key that is pressed.

Our method of key detection is entirely different one. It is based on counting the pixel bits.

The input to this section is the black and white image from the section image acquisition and

frame acquisition. The white bit corresponds to 1 and black corresponds to 0. We are

comparing two consecutive images taken by the webcam in MATLAB. If the two

consecutive images are the same or the difference is less than a threshold value. A key-press

is detected. And this calls the key detection section. That outputs the typed key. The

advantage of this method is that if we are keep on moving our hand then the images taken

from the webcam is not processed by the key detection( So a false key will not be detected

when our hand is moving). But the problem with this method is that we should move our

hand at a faster rate than the capture rate of webcam in between two consecutive key presses.

Key stroke detection

Keystroke detection means detecting the depth of key press. That means the time of repeated key press by without taking hand from a particular key. Our programs

17

8/6/2019 Project Final 23-06-099


adjust/increase the key-press detection rate within the two consecutive press of the same key.

Since the program needs to process the same image (it needs lesser time).

while 1 % function to check whether two snaps are the same

kb1 = getsnapshot(obj);

kb1 = filter_kb(kb1);

if diff_checker(kb1) < 18

diff = sum(sum(abs(kb2-kb1)));

if diff < 500 %if the difference between the two snaps is below 500 detect a key press

dc = diff_checker(kb1);

chara = alfa_det(kb1);

end

end

kb2 = kb1;

end

function [a] = diff_checker(kb)

a = bwarea(kb)/ 1000;

return

18

8/6/2019 Project Final 23-06-099


Fig8: Keystroke detection process

CHAPTER-9

KEY DETECTION

Key detection means pressing a key and detecting the corresponding key pressed.

Many methods are there like potential difference method, latching digital keys etc. in our

method when ever we press a virtual key it’s image will taken and corresponding key press is

detected by comparing the area enclosed by our hand. If our hand cover an area more then

the threshold area the corresponding key is detected and printed on the MATLAB command

window.

Fig9: The segmented key image with hand pressed

Key ‘B’ detected which is having a lesser white area

19

8/6/2019 Project Final 23-06-099


Fig10: The black and white image of the pressed key

1 2 3

Fig11(a): the connected elements

20

8/6/2019 Project Final 23-06-099


Fig11(b): the largest connected element

From the above figure to cut the biggest white area-3 from the three areas area-1, area-2

and area-3 we are using our function filter_big().

function [y] = filter_big(bwsg)

sg_labeled = bwlabel(bwsg);

noce = max(max(sg_labeled)); % number of connected elements

if noce == 0

y = ones(100,200);

% disp(noce)

return

end

for k = 1 : noce

sg_tmp = sg_labeled==k;

t = size(find(sg_tmp==1));

s_sg_tmp(k) = t(1);

end

% s_sg_tmp

kl = find(s_sg_tmp==max(s_sg_tmp));

y = sg_labeled==kl;

return

21

8/6/2019 Project Final 23-06-099


To align the keyboard in a horizontal plane we use the size_calc_rect() function. This

function calculate the alignment of the white are(the largest white area enclosing the

keyboard) in different positions and finding the alignment where the are enclosing the white

area is minimum. This is obtained only at horizontal alignment or in vertical alignment.

function [size_sg, sg_raw] = size_calc_rect(sg_raw)

sg_raw = clip(~sg_raw);

% gonna rotate image to make it fit in the smallest matrix, so that

% allighing orthogonally

k = 0;

rot_ang = 45;

eps_rot = 0.1; % epsilon a small number to represent acceptable tolerance

while (rot_ang > eps_rot)

k = k + 1;

size_sg_raw = size(sg_raw);

size_enc_rect = size_sg_raw(1) * size_sg_raw(2);

size_sg_act = size(find(sg_raw==1));

sg_tmp1 = imrotate(sg_raw, rot_ang);

sg_tmp1 = clip(~sg_tmp1);

size_sg_tmp1 = size(sg_tmp1);

size_enc_rect_tmp1 = size_sg_tmp1(1) * size_sg_tmp1(2);

size_sg_act_tmp1 = size(find(sg_tmp1==1));

22

8/6/2019 Project Final 23-06-099


if size_enc_rect_tmp1 <= size_enc_rect

rot_ang = rot_ang / 2;

sg_raw = sg_tmp1;

else

sg_tmp1 = imrotate(sg_raw, -rot_ang);

sg_tmp1 = clip(~sg_tmp1);






sg_raw = sg_tmp1;

else


end

end

end

size_sg = size(sg_raw);

return

23

8/6/2019 Project Final 23-06-099


function imgn=clip(imagen)

%Clip a black letter in word.

%Example:

% imagen=imread('metal.bmp');

% imgn=clip(imagen);

% subplot(2,1,1);imshow(imagen);title('INPUT IMAGE')

% subplot(2,1,2);imshow(~imgn);title('OUTPUT IMAGE')

if ~islogical(imagen)

imagen=im2bw(imagen,0.99);

end

a=~imagen;

[f c]=find(a);

lmaxc=max(c);lminc=min(c);

lmaxf=max(f);lminf=min(f);

imgn=a(lminf:lmaxf,lminc:lmaxc);%Clip image

Fig12 : Initial key board position

24

8/6/2019 Project Final 23-06-099


Fig13(a): horizontal alignment after some rotation

Fig13(b): horizontal alignment after some rotation

25

8/6/2019 Project Final 23-06-099


The above picture is horizontally aligned with an angle of just 0.01 degree. At this position

the area to enclose the keyboard is lesser than in all other cases( see the area of the black

square enclosing the keyboard). After obtaining this picture the image is subjected to image

segmentation

IMAGE SEGMENTATION

Segmentation means dividing the entire area into small section decided by number of

rows and columns.

The total number of segments will be equal to number of rows multiplies number of columns.

To increasing or decreasing the number of keys in keyboard we only need to change the rows

or columns. The resized image size is stored in a matrix variable.

We can use the loop statement given below to address different areas in the segmented

keyboard.

From the previous figure

Fig14: the detected key from fig 7

26

8/6/2019 Project Final 23-06-099


When a key Press is detected the image is scanned from left to write in a row and whenever a

region with minimal white area is detected the key corresponding to that will be get printed

on to a notepad using MATLAB file extension function and also on to the MATLAB

command window. For that we use the alfa_dit()

function [alfa] = alfa_det(kb1)



for m = 1 : 2

for n = 1 : 4

kbl(:,:,kalfa) = kb1( (((m-1) * floor(s(1) / 2)) + 1 ) : ( (m * floor(s(1) / 2))) , (((n-1) *

floor(s(2) / 4)) + 1 ) : ( (n * floor(s(2) / 4))) );

kba(kalfa) = bwarea(kbl(:,:,kalfa));

kalfa = kalfa + 1;

end

end

kba(1:4) = (kba(1:4) / sum(kba(1:4)) ) * 4;

kba(5:8) = (kba(5:8) / sum(kba(5:8))) * 4;

key = 10;

for k = 1 : 8

if kba(k) < 0.9

key = k;

break;

end

end

27

8/6/2019 Project Final 23-06-099


switch key

case 1

alfa = 'a';

case 2

alfa = 'b';

case 3

alfa = 'c';

case 4

alfa = 'd';

case 5

alfa = 'e';

case 6

alfa = 'f';

case 7

alfa = 'g';

case 8

alfa = 'h';

otherwise

alfa = 'no alfa';

end

fprintf(1, '%c' , alfa); % to print the alphabets in one line

28

8/6/2019 Project Final 23-06-099


fprintf(fid, '%c' ,alfa); %to input the characters into the file data.txt

end

29

8/6/2019 Project Final 23-06-099


CHAPTER-10

BLOCK DIAGRAM

30

8/6/2019 Project Final 23-06-099


Fig15 : Block diagram

31

8/6/2019 Project Final 23-06-099


CHAPTER-11

ALGORITHM

Image capturing and preprocessing

• Reading the image.

• Resizing- to increase the processing speed(since less size means less power)

• Fixing the image to a particular colour.

• Adjust a threshold value for white/black.

• The image is converted to black and white. The level above threshold will be

converted to black/white and vice versa.• Adjust the contrast of white.

Image Conversion

1. Choose the largest white spot from the picture.

2. For this purpose we use filtering.

3. Properly align the largest white spot by a rotating algorithm.

4. Crop the picture to a desired offset(Offset level at bottom, above and sides)

5. The picture is converted to different areas according to pixel size.

Keystroke detection

1. the captured image before and after the key press.(ie, we should keep our hand on the

pressed key to take two picture)

2. if the difference in image between two consecutive images will monitor continuously.

3. if the difference between two frames is less than a predetermined threshold the image

is checked for the key which is pressed.( that means the key press is identified)

Detection

1. Scan the entire area for finding out largest area of black (that is smallest white area).

2. Scanning is row vice.

3. Detection of any large black area above a threshold value will detect the key press and

the remaining check is stopped.

Output

1. Assign alphanumeric character in Chronological order with respect to area.

32

8/6/2019 Project Final 23-06-099


2. The character will be printed on the command window.

CHAPTER-12

ADVANTAGES

33

8/6/2019 Project Final 23-06-099


8/6/2019 Project Final 23-06-099


• It cannot allow multiple key presses at a time, like Ctrl or Alt key functions.

• The webcam needs to be in exactly vertical and perpendicular to the keyboard

• Our keyboard prints the keys only in the MATLAB command window since we were

not developed the system software for this keyboard

• The working surface should be flat.

• Lesser performance due to the webcam

• Necessitates the use of black

Covering on the hand

• Light scattering problems

CHAPTER-14

FUTURE DEVELOPMENT

• The speed of typing can be increase by using faster camera this can also remove slant

error problem by mathematical modeling or by using two cameras

• The multiple key identification(alt ctrl) can be possible by using 3D image modeling

• Light scattering problem can be eliminate by using laser cameras

•

Camera can be place any angle by using high power cameras

• System implementation

35

8/6/2019 Project Final 23-06-099


CHAPTER-15

APPLICATIONS

1. Used with Smart phones, PDAs, email, word processing and spreadsheet

tasks.2. As computer/PDA input.

3. Gaming and TV remote control

36

8/6/2019 Project Final 23-06-099


CHAPTER-16

CONCLUSION

Virtual Keyboard uses sensor technology and artificial intelligence to let users

work on any surface as if it were a keyboard. Projection key boards or virtual key boards

claim to provide the convenience of compactness with the advantages of a full-blown

QWERTY keyboard. The company's Virtual Keyboard is designed for anyone who's become

frustrated with trying to put information into a handheld but doesn't want to carry a notebook

computer around.

Canesta appears to be the most advanced in this class of technology. Different types

of virtual keyboards suit different typing styles. Thus virtual keyboards will make typing

easier, faster, and almost a pleasure.

37

8/6/2019 Project Final 23-06-099


CHAPTER-17

IMPORTANT MATLAB FUNCTIONS CREATED

The following are the important MATLAB functions that we created to make the virtual

keyboard with 8 keys. To work with more number of keys or to change the keyboard design

we only wanted to change the function alfa_det.m

1. cam_adjust.m

clear all

close all

obj = videoinput('winvideo', 2);

preview(obj)

2. cam_init.m

% script to initialise device

% initialising the device

clear all

38

8/6/2019 Project Final 23-06-099


close all

obj = videoinput('winvideo', 2);

preview(obj)

figure(2)

subplot(2,1,1)


imshow(kb1)


subplot(2,1,2)

imshow(kb1)

3. kbread2.m

% function [alfa] = kbread

fid = fopen('data1.txt', 'wt');% to open a file data.txt for writing using a variable fid

preview(obj) % to open the hardware of the webcam to fasten the snapshot

kb2 = kb1;

while 1 % function to check whether two snaps are the same



39

8/6/2019 Project Final 23-06-099


if diff_checker(kb1) < 18

diff = sum(sum(abs(kb2-kb1)));

if diff < 500 %if the difference between the two snaps is below 500 detect a key

press

dc = diff_checker(kb1);

chara = alfa_det(kb1);

fprintf(1, '%c' , chara); % to print the alphabets in one line

fprintf(fid, '%c' , chara); %to input the characters into the file data.txt

end

end

kb2 = kb1;

end

4. filter_kb.m

% fuction for filtering biggest element and return back

function [kb1] = filter_kb(kb1)

s = size(kb1); %Finding the size of main image(Original image)

kb1 = imresize(kb1, 200/s(1)); %Resizing the image to enhance the processing speed

% kb1 = kb1(:,:,2); % third layer offers highest contrast with human skin, i.e with the blue

layer in std format for RGB

40

8/6/2019 Project Final 23-06-099


kb1 = im2bw(kb1); % we might have to write a function for setting level value, i.e setting the

threshold

kb1 = imrotate(kb1, 180);

kb1 = ~kb1;

kb1 = filter_big(~kb1); %to filter in the biggest white area as keyboard

[skb1, kb1] = size_calc_rect(kb1); %to horizontally align the max white keyboard area so as

to eliminate the unwanted areas

s = size(kb1); %to assign the new size and put it into variable kb1

kb1 = imresize(kb1, [100 200]); %still we are scaling the figure based on column

5. filter_big.m

function [y] = filter_big(bwsg)

sg_labeled = bwlabel(bwsg);

noce = max(max(sg_labeled)); % number of connected elements

if noce == 0

y = ones(100,200);

return

end

41

8/6/2019 Project Final 23-06-099


for k = 1 : noce

sg_tmp = sg_labeled==k;

t = size(find(sg_tmp==1));

s_sg_tmp(k) = t(1);

end

kl = find(s_sg_tmp==max(s_sg_tmp));

y = sg_labeled==kl;

return

6. size_calc_rect.m

% fuction to calculate size of a rectangular piece

function [size_sg, sg_raw] = size_calc_rect(sg_raw)

% double(sg_raw);

sg_raw = clip(~sg_raw);

% sg_bkup = sg_raw; % for test insertion, delete soon this line

% gonna rotate image to make it fit in the smallest matrix, so that

% allighing orthogonally

k = 0;

42

8/6/2019 Project Final 23-06-099


8/6/2019 Project Final 23-06-099







sg_raw = sg_tmp1;

else


end

end

end

size_sg = size(sg_raw);

return

7. clip.m

Afunction imgn=clip(imagen)

%Clip a black letter in word.

if ~islogical(imagen)

imagen=im2bw(imagen,0.99);

44

8/6/2019 Project Final 23-06-099


end

a=~imagen;

[f c]=find(a);

maxc=max(c);lminc=min(c);

lmaxf=max(f);lminf=min(f);

imgn=a(lminf:lmaxf,lminc:lmaxc);%Clip image

8. diff_checker.m

% function to keep reading keyboard, and calculate the area

function [a] = diff_checker(kb)

a = bwarea(kb)/ 1000;

return

9. alpha_dit.m

function [alfa] = alfa_det(kb1)



for m = 1 : 2

for n = 1 : 4

kbl(:,:,kalfa) = kb1( (((m-1) * floor(s(1) / 2)) + 1 ) : ( (m * floor(s(1) / 2))) , (((n-1) *

floor(s(2) / 4)) + 1 ) : ( (n * floor(s(2) / 4))) );

45

8/6/2019 Project Final 23-06-099


kba(kalfa) = bwarea(kbl(:,:,kalfa));

kalfa = kalfa + 1;

end

end

kba(1:4) = (kba(1:4) / sum(kba(1:4)) ) * 4;

kba(5:8) = (kba(5:8) / sum(kba(5:8))) * 4;

key = 10;

for k = 1 : 8

if kba(k) < 0.9

key = k;

break;

end

end

switch key

case 1

alfa = 'a';

case 2

alfa = 'b';

46

8/6/2019 Project Final 23-06-099


case 3

alfa = 'c';

case 4

alfa = 'd';

case 5

alfa = 'e';

case 6

alfa = 'f';

case 7

alfa = 'g';

case 8

alfa = 'h';

otherwise

alfa = 'no alfa';

end

47

8/6/2019 Project Final 23-06-099


CHAPTER-18

REFERENCES

1. A three state model of graphical input by William A.S Buxton

www.billbuxton.com/3state.html

2. Kölsch, M. and Turk, M. Keyboards without Keyboards: A Survey of Virtual

Keyboards, Workshop on Sensing and Input for Media-centric Systems, Santa Barbara, CA,

June 20-21, 2002.

3. Alessandro Valli, “ Natural interaction”, Touch Panel, Vol. 2, No. 5, July 2007

4. Discover a powerful new virtual on screen keyboard with unlimited users

http://www.virtual-keyboard.com/special_needs/special_needs.html

5. VKB Bluetooth virtual keyboard- Reviewed by Bonnie Cha

http://reviews.cnet.com/pda-accessories/vkb-bluetooth-virtual-keyboard/4852-6460_7-

31293682.html

project final 23-06-099

Documents