texas instrumentsinnovation challenge india design contest ... · texas instrumentsinnovation...
TRANSCRIPT
Texas InstrumentsInnovation Challenge India Design Contest 2015
SELF ASSESSMENT
“We, authors of the report entitled Spatial Augmented Reality- GESTAIR, confirm that this report has not been
submitted to any other forum such as another contest or conference for publication. We hereby state that we will not
submit the same work for any other contest in the future. We understand that Texas Instruments has the right to use this
report in its conferences/publications. We will seek TI’s permission before we submit the report for publication in an
external forum.”
1. Comment on the originality of your idea. Did you derive inspiration from any other work? Provide the appropriate
references.
Ans: The motivation of the project was to help motor people with voice impairments. Assistive technology for speech
disorders could consist of equipment or a device that will supplement a user’s attempt to verbally communicate with
others. It could also be an unaided method of augmenting or even replacing speech (an alternative method of verbal speech).
An example of alternative speech is sign language. The existing systems available require the efforts of other person to learn
sign language .Thus providing a better solution to the society in the form of augmented reality can help them not only
interact with their surrounding better but also prove assistive in their verbal communication through inclusion of speech of
the recognized text .
But the prototype involved utilized two technologies to realize motive: a) Displair
b) digital pen utilizing triaxial accelerometers
2. List any persons who helped you in the course of the project and explain their contribution.
Ans: Our mentor guided us through the project and helped us coming out with a prototype for the proposed solution.
Also our seniors helped us in understanding the relevant circuits that could help accomplish the task. Also in making the
final video and in setting up the arena we took help from our friend and her contribution stands vital in successful
demonstration of the final prototype.
1. Highlight at least two technical challenges you faced and how you overcame them.
Ans: The technical challenges faced:
i. Inspite of intensive study , programming the Launchpad and understanding of syntax of ENERGIA and getting
desired output was a benchmark in itself. But further a more comprehensive study, experimentation and taking
help from seniors provided a hands on experience.
ii. Setting the boundary for the x,y,z coordinates
iii. Developing the fogscreeen module. Since the module was bulky and complex in hardware making the interface
was tough. Also other parameters like air-flow density hindered the project completion.
2. Please highlight at least two non-technical challenges you faced and how you overcame them.
Ans: None
3. Did you use WEBENCH to design power supply, filter etc in your project? If yes, share your experience of using
WEBENCH
Ans: No
4. Explain how the experience of the TI India Analog Design Contest helped you.
Ans: It helped us realize our potentials to develop a prototype of our own . Also in the process ,we learnt more about the
components manufactured by Texas Instruments and helped us analyse the various applications of these components in
realizing endobjectives and building smart and sophisticated systems.
5. List two things that could have added further value to your project.
Ans: A more sophisticated and portable system could have been developed . The idea endorses a new perspective to the
application of augmented reality driven systems to providing assistive technologies to the people in need. Also the project
still lacks composite characters and words all at the same time then passing single glyphs. The addition of the above could
have made the idea more comprehensive and could have facilitated ease of use.
6. Please tick all aspects of your project that you believe are now complete.
Paper design of hardware Algorithm/software design
Hardware implementation on breadboard System-level testing with examples
Hardware implementation on PCB Benchmarking/Performance Analysis
Hardware Testing Short Video on Project
Anshita Agarwal-
Diksha Sharma-
Divyanshi Rastogi
Deshraj Yadav-
Names and signatures of student team members
Mr.Sumitkhandelwal- Name and signature of the mentor
Spatial Augmented Reality-GESTAIR
Divyanshi Rastogi, Anshita Agarwal,Diksha Sharma, Deshraj Yadav SumitKhandelwal
JSS Academy of Technical Education Noida
C-20/1, Sector-62 Noida, U.P.-201301
Email: [email protected]
Abstract—If buttons are a thing of the past and touch screens are the present, what are the screens of the future? It’s not a riddle, but it is a trick question: the screens of the future won't be screens at all but interactive images floating in mid-air. The Digital Pen Technology and fogscreen projection for Virtual Projection is the new generation method for trajectory recognition based input device. The digital pen helps users to use the pen to write digits or make hand gestures, and the accelerations of hand motions measured by the accelerometer are transmitted to a computer for online trajectory recognition. In this technology, the implementation of a new concept of image drawing and mouse controlling is done using image processing along with the speech after recognition of the text that hangs in the mid air. Every day we use our gestures to interact with everyone. Gestures play an important role in our life. Thus Gestairis helpful in using the digital world through our gestures for interaction. It provides a link between the digital world and the real object world. And when this combines with the spatial coordinates, it can allow people to display and interact with information without cluttering the physical environment.Thus augmented reality blurs the line between what is real and what we see; by augmenting what we see, hear and feel. Thereby, iteliminates the need of multiple interface and creates a single universal interface allowing us to perform wide range of tasks. Keywords—Digital pen, accelerometer, fogscreen, text
recognition,augmented reality
I. INTRODUCTION
The main idea behind developing this system is to develop an
advance human computer interaction system using trajectory
recognition algorithm and image processing tools where system
can look for human behavior or action through use of
accelerometer, process it and consequently perform predefined
task or action and display floating images in mid-air.Main aim
of proposed system is to construct a system for online hand
writing character recognition written in air and a fogscreen that
gives us the video feedback in the mid-air. Here accelerometer
gives response for every slight deflection or movement in the
system. Accelerometer is developed by using MEMS
technology. A significant advantage of accelerometer for
general motion sensing is that they can be operated without any
external reference and limitation in working conditions
Thus, the final layout of the system proposed would look
like the following:
Figure 1: Basic functional block diagram
II. TECHNICAL BACKGROUND
The proposed solution is inspired by the digital pen that can
write in a 3-D plane and involves application of basic
handwriting recognition algorithm to identify the glyphs made
by the user. Also the second part of the project derives its
inspiration from an existing technology called Displairwhich is
a 3D interactive raster display technology developed by a
Russian company of the same name. The Displair projects
images onto sheets of water droplets suspended in air, giving
the illusion of a hologram. Our project is an amalgamation of both the
technologies and not only makes the use of hand gestures to
communicate but also aims at displaying the communicated
messages in the air.
Another added feature is the text to speech which reads out the
text aloud. Due to its motivation of helping motor people with
speech impairments, it helps rendering them a virtual voice to
communicate to others and also makes it easier for the person
on the other side to understand without making extra efforts of
learning sign language.
III PROPOSED SOLUTION
The existing technologies like displair use expensive cameras
and detect the hand movements. The prototype thus would be
quite expensive. The solution we have provided uses an
accelerometer as a wearable device that makes the prototype
comparatively cost effective.
Also since it is an assistive technology to the people with voice
impairments it is a boon to them by equipping them with a
virtual voice.
The overall execution flow is as follows:
Figure 2.Top-level view of the proposed solution
We assume that a handwriting recognizer has access to a
handwriting profile based on a large number of samples of a
user’s handwriting. In the case of letters, upper and lower case
must be written separately and in all cases the glyphs must be
formed in the same manner. Also, the main constraint factor
included was that the user could write only one letter at a time
and not a complete string. The training data is scaled to fit
within a fixed dimension set to a constant. Since we are dealing
with pixels of finite size, the variable sized dimension of the
bounding box never takes on a zero value. For recognition, the
user writes an unknown symbol which is then scaled to the
same bounding box as the training data. The recognizer has
access to all of the training data and precomputed statistics
about this data. By comparing aggregate statistical information
from the training data with statistics from the unknown symbol,
and by directly engaging in a pointwise comparison between
the unknown symbol and all known symbols in the training set,
the recognizer identifies what it believes is the best match.
IV IMPLEMENTATION
A. Hardware Implementation
a. Pattern Formation:
Digital pen consist of Tri-axial accelerometer& TI
microcontroller (MSP430G2553).
The MEMS based accelerometer measures the acceleration
signals generated by a user’s hand motions. The
microcontroller converts the analogacceleration signals to
digital ones via the A/D converter.The raw accelerationsignals
of hand motions are generated by the tri-axial accelerometer are
given to microcontroller. Our hand always trembles slightly
while moving due to human nature, which causes certain
amount of noise. The signal pre-processing consists of
calibration, a moving average filter, a high-pass filter, and
normalization. Using moving average filter we collect set of 10
value received from accelerometer & takes average of this
value because of that the signal become smoother and if there is
any sudden change in signal due to hand movement is avoided
with the help ofhigh pass filter.The normalization is to start the
signal from start point.
b.Text-to-speech module: The TTS256 is an 8-bit microprocessor programmed with letter-to-sound rules. This built-in algorithm allows us for the automatic real-time translation of English ASCII characters into allophone addresses compatible with the MagnevationSpeakJet Speech Synthesizer IC. It is then combined with the SpeakJet to build a complete text-to-speech solution.In our proposed system we are using TTS256 IC whichuses predefined speech rules to break up text into various sound components (called allophones) and then translates these into the numbers. The combination of TTS256 and SpeakJet allow us to turn text strings stored in the processor into speech. The circuit usedfor the same is:
Figure 3. Text to speech Module
The SpeakJet is connected to the audio conditioning circuit that
reads out the written text. It consists of a pre-amplifier stage , a
band pass filter employing NE5532A IC’s followed by a
power amplifier circuit using LM833 connected to the speaker
at its output. The circuit is shown below:
Figure 4. Pre-amplifier circuit
Figure 5.Band-pass filter
Fig 6.Power amplifier
c. Fogscreen Projection:
The last part of project is the projector which gets the video feedback after processing of the image that was extracted from the glyphs written by the user.Fogscreen is an exciting new projection technology that allows to project image and video screen of “dry fog” giving a false illusion that the objects are floating in mid-air. It is one type of advanced projecting device that consumes water and electricity to form fogs on which the images are projected. It is formed by ordinary tap water and digital technology like ultrasonic devices(usually a ultrasonic humidifier) to create a thin layer of dry fog which is sandwiched between two air curtains. Fogscreen is suspended fog generating device and has no
chemicals apart from ordinary tap water. It creates a dry fog by
ensuring water droplets are in range of 2-3 microns in size and
are electrostaticallycharged so they move around and away
from other objects.It is thus ahigh-tech-version of cool air
humidifier.
The system we used employed a mini fogger machine that was made using dry ice and tap water and a small fan about 60mm in diameter to direct the fog into the inlet of the fogscreen box. The fogscree box had one set of fans to blow the fog downwards while the other two sandwiches the fog between air curtains so that it becomes a smooth projection screen.
B. Software Implementation
The algorithm for handwriting recognition provides a
multiple character recognition output indication which includes
compressing the velocity indicating output, separately
analyzing information for the x and y velocity components,
analyzing the directions of the velocity, calculating velocity
thresholds, digitizing velocity components, comparing and
matching between a digital dictionary and a digitized velocity
record and using only part of the digitized velocity record,
comparing and matching between a digital dictionary and a few
different representation forms of digitized velocity using a
prioritization procedure which takes place in case of
disagreement between different comparing and matching
results, filtering out short duration segments of the velocity
components, indicating velocity value changes and ignoring
time durations between the changes, performing a merger
operation on velocity segments and binarizing velocity
segments. But all this happens internally within the inbuilt
package of python TESSERACT OCR. Thus the gestures made
by user are recorded and sent to computer for processing that
uses a python code to read, reconstruct and identify the
characters as written by the user.
V. CONCLUSIONS
The project as suggested used an accelerometer to
translate gestures into meaningful images and convey what was
exactly written by the user. The concentration was on the
improvement of the interface that connects the digital world
and the real world and this was successfully achieved by
implementing this gesture based device.
Also the idea aimed at creating a more clutter free
interaction of humans with their physical world and provide an
environment friendly and chemical-free interaction of
technology by making the traditional concept of pen and paper
redundant , but due to technical difficulties and insufficient air
flow , the fogscreen couldn’t be successfully implemented.
A.Future Scope:
The project has many applications: 1.One of the major contributions is to help the people with
speech disorders and impairment. School-aged children who
have special needs relating to speech or communication
disorders may be entitled to assistive technology services in the
classroom. Assistive technology for speech disorders could
consist of equipment or a device that will supplement a child’s
attempt to verbally communicate with others. It could also be
an unaided method of augmenting or even replacing speech (an
alternative method of verbal speech). An example of alternative
speech is sign language. Children who use assistive technology
will profit greatly. They will be able to communicate their
needs and desires and will perform better at school. They may
also see social benefitsand canbe proved to be a great
assistance in interacting and conveying messages to the people
around them thus making the extra efforts of learning sign
language redundant.
2.Also one of the major outcomes of this is the efforts of
making the concept of pen and paper redundant.The use of
writing texts virtually into the mid air and saving it into your
portable devices can greatly contribute to the environment.
3.Another aspect of the proposed system is its ability to take
education to the next level. With its power to be visually
appealing, it could open the doors to new education
opportunities. Students of all ages could be entertained by
interacting with it while also being challenged academically.
You can do anything from drafting documents to building
interactive 3D models.
B.Limitations:
1.It works with only single glyphs.
2.Thefogscreen module employed is bulky and is not portable
for ease of use. But, further development and research can make
device more economic and portable.
ACKNOWLEDGMENTS
We would like to thank our mentor Mr.SumitKhandelwal
for his constant support and guidance. We also thank our
seniors in helping us in coming out with the prototype and
guiding us in understanding the various circuits involved. We
thank our department and our HoD, Mr.SampathKumar.V in
providing us an access to labs ,components and equipments to
complete our project testing and realizing our potential in
developing the prototype.
REFERENCES
. [1] Sung-Do Choi, Lee, A.S., Soo-Young Lee, On-Line
HandwrittenCharacter Recognition with 3D Accelerometer, 2006 IEEE International Conference on Information Acquisition, 20-23 Aug.2006.
[2] Jeen-Shing Wang, Yu-Liang Hsu, Cheng-Ling Chu, Online
Handwriting Recognition Using an Accelerometer-Based Pen Device, 2nd International Conference on Advances in Computer Science and Engineering (CSE 2013)..
[3] http://prosauce.org/blog/2012/6/10/how-to-diy-improved-inexpensive-fog-screen.html
APPENDIX A
# [max size of image be let 900*900]
from PIL import Image
frompytesser.pytesser import *
import sys
import cv2
importos
importnumpy as np
coord = [(1,2,3),(2,3,4),(4,5,6),(6,7,8)]
img = cv2.imread('white.jpg')
fori in coord:
x,y = i[0],i[1]
img.itemset((x,y,0),0)
img.itemset((x,y,1),0)
img.itemset((x,y,2),0)
# plot the given gesture into the image
cv2.imwrite('test.jpg',img)
image_file = 'test.jpg'
im = Image.open(image_file)
text = image_to_string(im)
text = image_file_to_string(image_file)
text=image_file_to_string(image_file, graceful_errors=True)
# the identified character is
print "THE OUTPUT IS"
print text
img1 = cv2.imread('test.jpg', 0)
# pick the character printed from the database
img2 = cv2.imread(text+'.jpg', 0)
#merge the two images
h1, w1 = img1.height,img1.width
h2, w2 = img2.height,img2.width
vis = np.zeros((max(h1, h2), w1+w2), np.uint8)
vis[:h1, :w1] = cv2.GetMat(img1)
vis[:h2, w1:w1+w2] = cv2.GetMat(img2)
vis2=cv2.CreateMat(vis.shape[0],vis.shape[1],cv2.CV_8UC)
cv2.CvtColor(cv2.fromarray(vis),vis2, v2.CV_GRAY2BGR)
cv2.ShowImage("test", vis2)
cv2.waitKey(0)
cv2.destroyAllWindows()
APPENDIX C – BILL OF MATERIALS
Component Manufacturer Cost per
compon
ent
Quant
ity
Total
cost of
compon
ent
TI Supplied/
Purchased
1 ADXL 335 Analog
Devices
4.95
USD
2 9.95
USD
Purchased
2 TTS256C Magnevation 21.95 USD
1 21.95 USD
Purchased
3 SPEAKJET
IC
Magnevation 24.95
USD
1 24.95
USD
Purchased
4 LM833 Texas
Instruments
1.05
USD
1 1.05
USD
TI
supplied
5 LM386 Texas
instruments
0.5
USD
1 0.5
USD
TI
supplied
6 NE5532A Texas
instruments
0.5
USD
2 0.5
USD
Purchased
7 PC Fans Cooler
master
5 USD 10 50
USD
Purchased
8 Fogscreen
module
components
1 50
USD
Purchased
Total Cost of the Project 159
USD
Youtube Link:- https://www.youtube.com/watch?v=HMHPLSf3ZdA
TEAM ID:-1298 TEAM LEADER:-Anshita Agarwal TEAM MENTOR:- Sumit Khandelwal