AI and ARGodfrey Nolan
01.Experience
Experience
• FieldMap
• Create waypoints
• Rolling shutter vs global shutter
• Automated flight around field
• Tile images on iPad
• Stitch images using OpenCV.
9/5/2019 Counting Sheep 5
9/5/2019 Counting Sheep 6
Experience
• Secure Drones Labs
• No data leakage
• Tamper resistant
• Self destruct button
9/5/2019 Counting Sheep 7
02.Counting Sheep??
Counting Sheep
• Requirements
• Semi-practical example
• Using existing experience
• Move to Drone 2.0
• Answer the age old question‘Do Androids dream ofElectric Sheep’
9/5/2019 Counting Sheep 9
03.Plug and Play
Components
• DJI Mobile SDK
• Google Tensorflow
• Google Cloud Platform
• Docker
9/5/2019 Counting Sheep 11
Components
• DJI Mobile SDK
• Google Tensorflow
9/5/2019 Counting Sheep 12
Components
• DJI Mobile SDK
• Google Tensorflow
9/5/2019 Counting Sheep 13
Components
• DJI Mobile SDK
• Google Tensorflow
9/5/2019 Counting Sheep 14
Components
• DJI Mobile SDK
• Google Tensorflow
9/5/2019 Counting Sheep 15
Components
• DJI Mobile SDK
• Google Tensorflow
9/5/2019 Counting Sheep 16
Components
• DJI Mobile SDK
• Google Tensorflow
9/5/2019 Counting Sheep 17
04.Machine Learning Workflow
Workflow
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 19
Workflow
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 20
Workflow
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 21
Workflow
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 22
Workflow
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 23
Workflow
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 24
Workflow
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 25
05.Implementation
Implementation
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 27
Implementation
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 28
Implementation
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 29
Implementation
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 30
Train and Test
• Prepare the Dataset
• Set Up Google Cloud Account
• Set Up Docker
• Configure Google Cloud
• Set Up Object Detection API
• Train Your Model
• Evaluate Your Model
9/5/2019 Counting Sheep 31
Train and Test
Train and Test
• Prepare the Dataset
• Set Up Google Cloud Account
• Set Up Docker
• Configure Google Cloud
• Set Up Object Detection API
• Train Your Model
• Evaluate Your Model
9/5/2019 Counting Sheep 33
Train and Test
• Prepare the Dataset
• Set Up Google Cloud Account
• Set Up Docker
• Configure Google Cloud
• Set Up Object Detection API
• Train Your Model
• Evaluate Your Model
9/5/2019 Counting Sheep 34
Train and Test
• Prepare the Dataset
• Set Up Google Cloud Account
• Set Up Docker
• Configure Google Cloud
• Set Up Object Detection API
• Train Your Model
• Evaluate Your Model
9/5/2019 Counting Sheep 35
Train and Test
• Prepare the Dataset
• Set Up Google Cloud Account
• Set Up Docker
• Configure Google Cloud
• Set Up Object Detection API
• Train Your Model
• Evaluate Your Model
9/5/2019 Counting Sheep 36
Train and Test
• Prepare the Dataset
• Set Up Google Cloud Account
• Set Up Docker
• Configure Google Cloud
• Set Up Object Detection API
• Train Your Model
• Evaluate Your Model
9/5/2019 Counting Sheep 37
Train and Test
• Prepare the Dataset
• Set Up Google Cloud Account
• Set Up Docker
• Configure Google Cloud
• Set Up Object Detection API
• Train Your Model
• Evaluate Your Model
9/5/2019 Counting Sheep 38
Train and Test
• Prepare the Dataset
• Set Up Google Cloud Account
• Set Up Docker
• Configure Google Cloud
• Set Up Object Detection API
• Train Your Model
• Evaluate Your Model
9/5/2019 Counting Sheep 39
Train and Test
• Prepare the Dataset
• Set Up Google Cloud Account
• Set Up Docker
• Configure Google Cloud
• Set Up Object Detection API
• Train Your Model
• Evaluate Your Model
9/5/2019 Counting Sheep 40
Implementation
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 41
Implementation
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 42
Implementation
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 43
Implementation
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 44
Implementation
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 45
Implementation
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 46
Implementation
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 47
Implementation
• Get Data
• Train & Test
• Adjust
• Deploy
• Test & Validate
9/5/2019 Counting Sheep 48
06.Resources Q&A
Lessons Learned
• Crowdsource earlier
• Choose more practical example
• Create your own images
• Image quality
9/5/2019 Counting Sheep 51
Lessons Learned
• Dyed sheep
• Streamline config. with Docker
• Tensorflow works on iOS
9/5/2019 Counting Sheep 52
Future Plans
• Cattle
• Horses
• Deer
• Polar Bear (thermal)
• Parking Lots
• Plants
• Pallets
9/5/2019 Counting Sheep 53
Future Plans
• Animal Tracking
• AR and AI
• Amazon’s Mechanical Turk
• Automate
9/5/2019 Counting Sheep 54
Resources
• http://developer.dji.com
• https://www.tensorflow.org/install/
• https://github.com/tzutalin/labelImg
• https://github.com/tensorflow/models/tree/master/research/object_detection
• https://becominghuman.ai/
• https://towardsdatascience.com/how-to-train-your-own-object-detector-with-tensorflows-object-detector-api-bec72ecfe1d9
• https://docs.bazel.build
• https://t.co/uUTWuPw1ve (whitepaper) and https://github.com/nickdonnellyriis/sheep_demo (code)
9/5/2019 Counting Sheep 55
Q&A
• @godfreynolan
• riis.com/blog
• slides.com/godfreynolan
9/5/2019 Counting Sheep 56
07.Extras
Prepare the Dataset
• Gather images
• Label the images with LabelImg found at https://github.com/tzutalin/labelImg
• Generate CSV files from your labeled dataset using script found at https://github.com/datitran/raccoon_dataset/blob/master/xml_to_csv.py
• Convert the CSV files into your TFRecord Dataset using script found at https://github.com/datitran/raccoon_dataset/blob/master/generate_tfrecord.py
• Create your label map file.
9/5/2019 Counting Sheep 58
Prepare the Dataset
9/5/2019 Counting Sheep 59
Set Up Google Cloud Account
• Sign up for a Google Cloud Account
• Set up a project named CountingSheep
• Enable ML Engine for your project https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com
• Click “Storage” on the side bar, and then create a new bucket called CountingSheepData
• Create a sub-directory called “data” in your storage bucket
9/5/2019 Counting Sheep 60
Set Up Your Docker Environment
• Download Docker from docker.io
• Download the Dockerfilehttps://github.com/tensorflow/models/blob/master/research/object_detection/dockerfiles/android/Dockerfile
• docker build .
• docker images
• docker run -it IMAGE_ID
9/5/2019 Counting Sheep 61
Configure your local Google Cloud environment
• Login - gcloud auth login
• Set your google cloud project - gcloud config set project YOUR_PROJECT_NAME
• Set your storage bucket - gsutil mb gs://YOUR_UNIQUE_BUCKET_NAME
• Export Google Cloud project/bucket names - export PROJECT="YOUR_PROJECT_ID"
export YOUR_GCS_BUCKET="YOUR_UNIQUE_BUCKET_NAME"
9/5/2019 Counting Sheep 62
Configure your local Google Cloud environment
• Get the name of your service account:-curl -H "Authorization: Bearer $(gcloud auth print-access-token)" https://ml.googleapis.com/v1/projects/${PROJECT}:getConfig
• Using the tpuServiceAccount value from the response. Export the TPU service account with the following command:-
- export TPU_ACCOUNT=your-service-account
• Grant ML service agent to TPU service account:-gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:$TPU_ACCOUNT --role roles/ml.serviceAgent
9/5/2019 Counting Sheep 63
Set up Object Detection API
• Ensure your local copy of object detection API works with following commands:cd /tensorflow/models/researchpython object_detection/builders/model_builder_test.py
9/5/2019 Counting Sheep 64
Set up Object Detection API
• Move your TFRecord files and label map from your host machine into your docker imagedocker cp train.record CONTAINERNAME:/train.recorddocker cp test.record CONTAINERNAME:/test.recorddocker cp sheep_label_map.pbtxt CONTAINERNAME:/ sheep_label_map.pbtxt
• Copy the data over to your bucket (from your docker container)gsutil -m cp -r /train.record gs://${YOUR_GCS_BUCKET}/data/gsutil -m cp -r /test.record gs://${YOUR_GCS_BUCKET}/data/gsutil cp /sheep_label_map.pbtxt gs://${YOUR_GCS_BUCKET}/data/
9/5/2019 Counting Sheep 65
Set up Object Detection API
• Download the object detection model:curl –O http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_0.75_depth_300x300_coco
14_sync_2018_07_03.tar.gztar xzf ssd_mobilenet_v1_0.75_depth_300x300_coco14_sync_2018_07_03.tar.gz
• Upload it to Google cloudgsutil cp /tmp/ssd_mobilenet_v1_0.75_depth_300x300_coco14_sync_2018_07_03/model.ckpt.*
gs://${YOUR_GCS_BUCKET}/data/
9/5/2019 Counting Sheep 66
Set up Object Detection API
• Update the following values in the config file:/tensorflow/models/research/object_detection/samples/configs/ssd_mobilenet_v1_0.75_depth_quantized_300x300_pets_sync.config
Upload file to Google Cloud bucket with the following command:gsutil cp
/tensorflow/models/research/object_detection/samples/configs/ssd_mobilenet_v1_0.75_depth_quantized_300x300_pets_sync.config gs://${YOUR_GCS_BUCKET}/data/pipeline.config
• Package the object detection API with the following commands:bash object_detection/dataset_tools/create_pycocotools_package.sh /tmp/pycocotoolspython setup.py sdist
9/5/2019 Counting Sheep 67
Train your model
• Start your training job on Google Cloud TPU with the following command:cd /tensorflow/models/researchgcloud ml-engine jobs submit training `whoami`_object_detection_`date +%s` --job-dir=gs://${YOUR_GCS_BUCKET}/train --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-
0.1.tar.gz,/tmp/pycocotools/pycocotools-2.0.tar.gz --module-name object_detection.model_tpu_main--runtime-version 1.9 --scale-tier BASIC_TPU --region us-central1--model_dir=gs://${YOUR_GCS_BUCKET}/train --tpu_zone us-central1 --pipeline_config_path=gs://${YOUR_GCS_BUCKET}/data/pipeline.config
•9/5/2019 Counting Sheep 68
Train your model
• Start your training job on Google Cloud TPU with the following command:cd /tensorflow/models/researchgcloud ml-engine jobs submit training `whoami`_object_detection_`date +%s` --job-dir=gs://${YOUR_GCS_BUCKET}/train --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-
0.1.tar.gz,/tmp/pycocotools/pycocotools-2.0.tar.gz --module-name object_detection.model_tpu_main--runtime-version 1.9 --scale-tier BASIC_TPU --region us-central1--model_dir=gs://${YOUR_GCS_BUCKET}/train --tpu_zone us-central1 --pipeline_config_path=gs://${YOUR_GCS_BUCKET}/data/pipeline.config
•9/5/2019 Counting Sheep 69