Handy Facebook(Hand gestures to manipulate social networking website)

Team Members

Jaskaran Uppal (0419)

Sandeep Mallela (9769)

Darpan Dhamija (0550)

Rahul Perhar (4562)

Project Objective

Identify hand gestures in front of a webcam

Navigate the website depending on the gestures recognized

Tasks to be performed

• Making of gestures in front of the camera

• Gesture detection at a suitable frame rate

• Capturing the gestures and storing them in a .jpg file

• System training to recognize the gestures with a low error rate

• Execution of events upon the successful gesture recognition on the webpage

• Notification to be sent to the user

Gesture Making

Usage of a small set of gestures (fingers).

Every finger raised will perform some predefined navigation of the webpage

System capabilities can be programmed to accommodate other human gestures as well

Error in detection can be reduced by training

Gesture Detection

Gestures are detected at a suitable frame rate.

The camera captures the hand gesture and we apply canny edge detection algorithm to store the gestures in the following format

System Training

System training is done using “Neuroph” an open source Image Recognition tool that takes images as input and produces a neural network.

This Neural network can be trained to recognize the gestures

This can be used with Java Classes to be integrated in our application, using plug-in provided with the tool

Website Navigation

The default page shown to the user

User makes gesture

System recognizes

Website navigates

Facebook profile loaded

Furthermore the user can use other gestures to navigate though additional WebPages

Website Navigation Contd.

After initial gesture recognition, user is navigated to a personal profile page where he is given additional options

The user can make gestures to perform either of the actions

1 Chat

2 Write on wall

3 Like a post

4 Poke a person

Implementation Details

The application is implemented using the following:

OpenCV libraries for gesture recognition code

Using Java to capture the image and convert it into a BufferedImage for easy processing

Neuroph tool is used to train the system

The output from Neuroph is the recognition of Gesture upon which we have actions defined

Results

Home Page

Results

Personal Page of a user

Results

Opening Chat for a user

Results

Writing on the wall of a user

Results

“Like” a user post

Results

“Poke” a user

Limitations

The Limitations to the system includes the following:

The error rate in gesture recognition is persistent

It is a Lo-Fi prototype of what can be done on a larger scale further improvements can be done

Gesture recognition is dependent upon on available light.

Future Additions

Improvement in Hand gesture recognition. Making the system more refined and gestures easily recognizable

We can Integrate this into a number of applications like Google maps to get the address of a particular place.

A lot more different gestures can be used and trained in the system

We can have a real chat window in the future

Credits & References

Prof. Suya You, for all the support and knowledge of various User Interface Designs

Vijayakumar Gopalakrishnan, TA for giving an initial idea and helping us in realizing the project till the completion

Neuroph and related documentation for gesture recognition (http://neuroph.sourceforge.net/documentation.html)

Thank You