isightengineering.nyu.edu/mechatronics/projects/me7836/spring...isight reach out and grab it! seoho...
TRANSCRIPT
![Page 1: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/1.jpg)
iSightReach Out and Grab It!
Seo Ho Kang Nicholas MiltonFabio Vulpi
![Page 2: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/2.jpg)
Problem• Visually impaired people have difficulties to detect and retrieve objects in unknown environment.
Status Quo• Orcam• Aira• Personal Assistance
Problem Statement
2
![Page 3: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/3.jpg)
User Requirements• Fashionable• Can pull it out only when you need it• Lightweight• Voice control like siri• distinguish between coke and pepsi
Our Solution
3
Semester Goals • Computer vision navigation• Haptic feedback glove to guide hand• Audio interface for object request
![Page 4: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/4.jpg)
Actuation Hardware
Design
4
rFduino on custom PCB on Glove Bluno Beetle on watch
• Compact and lightweight
• Vibrating Motors protected
• Single wireless article
• Gazelle proprietary communication Protocolc
• Compact and lightweight
• Vibrating Motors in rings
• More Stable camera mount
• Arduino ide support
![Page 5: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/5.jpg)
Design
7/28/20 5
PCB schematic and board made with Eagle software, showing Bluno Beetle, four identical motor control circuits (transistor, resistor, diode for back EMF) voltage regulator, switch and battery support.
![Page 6: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/6.jpg)
Software
Design
6
![Page 7: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/7.jpg)
Design
7
![Page 8: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/8.jpg)
Design
8
![Page 9: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/9.jpg)
Design
9
1-1) Run script with command below.
[Command]python3 demo/object_detection_ver5.py \--model test_data/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite \--label test_data/coco_labels.txt
![Page 10: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/10.jpg)
Design
10
2-1) FPS optimization-used parser to organize script[Snap of object_detection_ver5.py]parser = argparse.ArgumentParser()parser.add_argument(
'--model', help='Path of the detection model.', required=True)parser.add_argument(
'--label', help='Path of the labels file.')parser.add_argument(
'--input', help='File path of the input image.')parser.add_argument(
'--output', help='File path of the output image.')args = parser.parse_args()
![Page 11: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/11.jpg)
Design
11
2-2) FPS optimization-use array inputWe chose to input image array to “mobilenet_ssd_v2” directly after flatten the image. We previously input image but after changing it to input array, FPS increased by 5~10. Flatten step was done because Google TF lite api asked us to put flatten image. Plus, in order to get fast FPS you should follow the way we implemented with pi-camera.
![Page 12: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/12.jpg)
Design
12
3-1) Communication (Android app <-> Raspberry Pi3)We use socket serial communication between app and rpi3. App converts voice to text and send it to raspberry pi3.
![Page 13: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/13.jpg)
Design
13
3-2) Communication (Arduino <-> Raspberry pi3)We use ttyACM0 serial communication.
![Page 14: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/14.jpg)
Design
14
4) SSD Object detection modelInput is [300, 300] image array and we use label, and box (x,y) coordinate output. We use pretrainedmodel.
![Page 15: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/15.jpg)
Demonstration
![Page 16: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/16.jpg)
Object Detection Different Light
TESTING
16lux
![Page 17: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/17.jpg)
Object Detection Different Objects
TESTING
17lux
![Page 18: iSightengineering.nyu.edu/mechatronics/projects/ME7836/spring...iSight Reach Out and Grab It! SeoHo Kang Nicholas Milton Fabio Vulpi Problem •Visually impaired people have difficulties](https://reader036.vdocuments.us/reader036/viewer/2022081621/612878a25fe82a634b2e35ce/html5/thumbnails/18.jpg)
Object Detection Different Senario
TESTING
18lux