from human machine interaction to human machine ... · from interaction to integration and...
TRANSCRIPT
![Page 1: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/1.jpg)
From Human Machine Interactionto Human Machine Integration and Intelligence
Le Thanh Ha, Ph.DAssoc. Prof. at University of Engineering and Technology,
Vietnam National [email protected]; [email protected]; +84983 692 592
![Page 2: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/2.jpg)
About myself
• Full name: Le Thanh Ha
• 2005-2010: Ph.D at Korea University, Korea
• 2010-now:
– Assoc. Prof. at University of Engineering and Technology (UET), VNUH
– Head of HMI Laboratory
• Expertise: Computer vision, Image/video processing and analysis, Machine learning
5/14/2018 2Le Thanh Ha, HMI Laboratory
![Page 3: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/3.jpg)
HMI Laboratory
IntelligenceIn
terf
ace
Standardization
Standardization
Standardization
ISO/IEC JTC 1 Information Technology
Interaction
Integration
Intelligence
Inte
rfac
e https://hmiuet.wordpress.com
Computer Vision
Artificial Intelligence
Computer Graphics
Video Compression
![Page 4: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/4.jpg)
Single intelligent humanoid
They look like machines. But there is human soul inside!!!
5/14/2018 Le Thanh Ha, HMI Laboratory 4
NS-5 Eva R2-D2 Terminator
![Page 5: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/5.jpg)
Swamp and crowd inspired from nature
5
Collective transportation Flock of birds School of fishes
Seasonal migrationCollective hunting Celebrating crowd
![Page 6: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/6.jpg)
5/14/2018 6
Human and Machine senses
Le Thanh Ha, HMI Laboratory
![Page 7: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/7.jpg)
Human and Machine collaborations
Interface Interaction
Integration Intelligence
5/14/2018 Le Thanh Ha, HMI Laboratory 7
![Page 8: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/8.jpg)
Human Machine Interface
• How do humans manipulate with machine and how information exchanged between human and machines?
• Answer: Standardization
5/14/2018 Le Thanh Ha, HMI Laboratory 8
![Page 9: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/9.jpg)
Keyboard layout
ISO/IEC 9995-3 Information technology: the Complementary Latin group layout according to ISO/IEC 9995-3:2010
5/14/2018 Le Thanh Ha, HMI Laboratory 9
![Page 10: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/10.jpg)
Video standards
![Page 11: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/11.jpg)
ISO standards
243 Committee in ISOIT Committee publish 3004/21347 (14%) ISO standards
5/14/2018 Le Thanh Ha, Lab of HMI 11
April 2017
April 2018: 3149
![Page 12: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/12.jpg)
Information technology
Committee TitlePublished standards
Standards under development
ISO/IEC JTC 1 INFORMATION TECHNOLOGY 3004 546
ISO/IEC JTC 1/SC 17 Cards and personal identification 114 31
ISO/IEC JTC 1/SC 2 Coded character sets 54 3
ISO/IEC JTC 1/SC 22 Programming languages, their environments and system software interfaces 108 20
ISO/IEC JTC 1/SC 23 Digitally Recorded Media for Information Interchange and Storage 138 2
ISO/IEC JTC 1/SC 24 Computer graphics, image processing and environmental data representation 79 8
ISO/IEC JTC 1/SC 25 Interconnection of information technology equipment 188 27
ISO/IEC JTC 1/SC 27 IT Security techniques 165 63
ISO/IEC JTC 1/SC 28 Office equipment 48 2
ISO/IEC JTC 1/SC 29 Coding of audio, picture, multimedia and hypermedia information 574 (20%) 103
ISO/IEC JTC 1/SC 31 Automatic identification and data capture techniques 117 30
ISO/IEC JTC 1/SC 32 Data management and interchange 77 17
ISO/IEC JTC 1/SC 34 Document description and processing languages 80 11
ISO/IEC JTC 1/SC 35 User interfaces 67 16
ISO/IEC JTC 1/SC 36 Information technology for learning, education and training 38 10
ISO/IEC JTC 1/SC 37 Biometrics 118 28
ISO/IEC JTC 1/SC 38 Cloud Computing and Distributed Platforms 10 5
ISO/IEC JTC 1/SC 39 Sustainability for and by Information Technology 6 12
ISO/IEC JTC 1/SC 40 IT Service Management and IT Governance 19 9
ISO/IEC JTC 1/SC 6 Telecommunications and information exchange between systems 352 38
ISO/IEC JTC 1/SC 7 Software and systems engineering 171 425/14/2018 Le Thanh Ha, Lab of HMI 12
April 2018: 3149
![Page 13: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/13.jpg)
ISO/IEC JTC 1 - Information technology
5/14/2018 Le Thanh Ha, Lab of HMI 13
![Page 14: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/14.jpg)
From Interaction to Integration and Intelligence
5/14/2018 Le Thanh Ha, HMI Laboratory 14
IntelligenceIntegrationInteraction
implies stimulus-response
implies partnerships WORKING together
implies partnerships THINKING together
![Page 15: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/15.jpg)
From Interaction to Integration and Intelligence
Interaction
• Stimulus-response
• Machine simply take task orders from human
• Like client-server architecture
Integration
• Machine and human work as partners
• Work done by exchange sequence of information between Machine and Human
• Engage all kinds of physical sensors: Geometric-Visual, Sound-Auditory, Haptic-Tactile
Intelligence
• Information exchanged just like human and human
• Engage higher abstraction of information exchanged between Machine and Human
• Machine can be able to draw both physical and emotional meaning from human
5/14/2018 Le Thanh Ha, HMI Laboratory 15
![Page 16: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/16.jpg)
From Interaction to Integration
Interaction can be described as stimulus-response
Integration implies partnership WORK together
5/14/2018 Le Thanh Ha, HMI Laboratory 16
Desktop applications: MS Offices.doc, .rtf, .html
Online applications: Google drivers.html version 5.0
![Page 17: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/17.jpg)
From Interaction to Integration
5/14/2018 Le Thanh Ha, HMI Laboratory 17
U. Farooq, et al., Human-Computer Integration, Interaction vol. 23, iss. 6, pp. 26-32, 2016
![Page 18: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/18.jpg)
From Integration to Intelligence
• There is a continuum from Integration to Intelligence. Intelligence extends but does not replace interaction an integration.
• Engage higher abstraction of information exchanged between Machine and Human
• Machine can be able to draw both physical and emotional meaning from human
5/14/2018 Le Thanh Ha, HMI Laboratory 18
Integration implies partnerships WORKING
together
Intelligence implies partnerships THINKING
together
![Page 19: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/19.jpg)
From Integration to Intelligence
5/14/2018 Le Thanh Ha, HMI Laboratory 19
Integration implies partnerships WORK
together
Intelligence implies partnerships THINK
together
Open issue in Computer science:+ How to present level of abstract of information?+ How to autonomous create new thing? (Unsupervised learning)+ Knowledge presentation and discovery?+ …
![Page 20: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/20.jpg)
Abstraction level of visual information
5/14/2018 Le Thanh Ha, HMI Laboratory 20
Physical Abstract
Low level Middle level High level
Light frequenciesBrightness
emotion
Edges
Lines
ContoursColors
Regions
Depth
Pixel Intensity
anomaly events
a facesa pen
a person
attention
quality assessmentan animal
Intelligence degree
a banknote
social awareness
![Page 21: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/21.jpg)
Standardization for AI
5/14/2018 Le Thanh Ha, HMI Laboratory 21
Intelligence
Inte
rfac
e
Standardization
Standardization
Standardization
ISO/IEC JTC 1 Information Technology
ISO/IEC JTC 1/SC 42 - ArtificialIntelligence
Recently created in 2017
Interaction
Integration
Intelligence
![Page 22: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/22.jpg)
SOME WORKS FROM OUR GROUP
Vietnamese banknote speaker
Smart camera for healthcare
Human visual system based Image quality assessment
5/14/2018 Le Thanh Ha, HMI Laboratory 22
![Page 23: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/23.jpg)
Vietnamese banknote speaker
Money speaker helps visual impaired people to identify Vietnamese banknotes (iOS & Android)
Make use of Compact descriptors for Visual Search (CDVS):• A standard proposed by the Moving Picture Experts Group [3] (MPEG) organization• Implemented with SIFT at its core• CDVS operators:
– Extract – extract and encode compact descriptors from a set of images– Make index – generate a tree-structured codebook from descriptors extracted from a set of images– Pairwise matching – compare the descriptors extracted from query and reference images– Retrieval – search and discovery of images contained within a large collection, some of which depict
the same objects or scenes as those depicted by a query image.
![Page 24: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/24.jpg)
CDVS
5/14/2018 Le Thanh Ha, HMI Laboratory 24
Extraction pipeline
Pairwise Matching pipeline
Retrieval pipeline
a SIFT vector represented by 128 bytes
![Page 25: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/25.jpg)
Image database
5/14/2018 Le Thanh Ha, HMI Laboratory 25
![Page 26: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/26.jpg)
Smart camera for healthcare
• Fall detection
• Video source from IP cameras
• Detection accuracy from testing dataset: 90%.
• Speed of detection on Rasberrypi 2: 20fps
• Automatically send sms or notifyVN Patent pending Number: 1-2017-02136
5/14/2018 Le Thanh Ha, HMI Laboratory 26
![Page 27: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/27.jpg)
Detection pipeline
5/14/2018 Le Thanh Ha, HMI Laboratory 27
![Page 28: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/28.jpg)
5/14/2018 Le Thanh Ha, HMI Laboratory 28
![Page 29: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/29.jpg)
5/14/2018 Le Thanh Ha, HMI Laboratory 29
![Page 30: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/30.jpg)
HVS-based image quality assessment• Objective measures do
not meet subjective• All current IAQs grades
images as a whole– Can’t be aware of
subject context– Can’t be applied to
current block based video coding
➢ Needs method assess quality of images locally
5/14/2018 Le Thanh Ha, HMI Laboratory 30
The same amount of noise has been added to these two images, such that their PSNR isidentical. High-frequency noise was inserted into the bottom region of the left image, whereas band-pass filtered noise was inserted into the top region of the right image. The noise is hardly visible in the left image due to our low sensitivity to high-frequency stimuli and the strong masking by highly textured content in the bottom region.
![Page 31: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/31.jpg)
Subjective IQA database
• Since there is now benchmark available
• We need to create that database for our own purposes.
• Adapt ITU-R BT.500-11 – “Methodology for the Subjective Assessment of the Quality of Television Pictures,” 2002.
5/14/2018 Le Thanh Ha, HMI Laboratory 31
![Page 32: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/32.jpg)
Create new tool for subjective testing
5/14/2018 Le Thanh Ha, HMI Laboratory 32
![Page 33: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/33.jpg)
5/14/2018 Le Thanh Ha, HMI Laboratory 33
![Page 34: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/34.jpg)
IQA databases
Simple IQA database• 20 subjects• 20 images• Distortion levels: 5 QPs• Image patch size: 64 × 64.• 2 resolution levels• 3 types of image patches
– Edge– Smooth– Texture (complex)
• Total pairs of image patchs: 600
Huge IQA database• 1480 subjects• 308 images (HD and FHD)• Distortion levels: random• Image patch size: 64 × 64• Types of image patches: Spatially
random• Total pairs of image patches: 61 600
5/14/2018 Le Thanh Ha, HMI Laboratory 34
![Page 35: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/35.jpg)
Experiment results on 1st database
5/14/2018 Le Thanh Ha, HMI Laboratory 35
• Relationship between MOSs and QP values are approximated by a logistic function
where 𝑎1 and 𝑎2 are estimated from data.
![Page 36: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/36.jpg)
Experiment results on 1st database
• Applied on H.264/AVC video coding standard and obtain good results.
• A patent has been filed for this HVS-based video coding method.
5/14/2018 Le Thanh Ha, HMI Laboratory 36
![Page 37: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/37.jpg)
Experiment results on 1st database
5/14/2018 Le Thanh Ha, HMI Laboratory 37
![Page 38: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/38.jpg)
5/14/2018 Le Thanh Ha, HMI Laboratory 38
![Page 39: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/39.jpg)
Works on 2nd database
• Data cleaning to remove noise and outliers
• Deep neuron network architectures for learning how HVS assess image quality (locally)
• Applied to the HVS-based IQA DNN to video coding scheme H.265
5/14/2018 Le Thanh Ha, HMI Laboratory 39
![Page 40: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/40.jpg)
Reference DNNs
5/14/2018 Le Thanh Ha, HMI Laboratory 40
Jongyoo Kim, Sanghoon Lee, "Deep Learning of Human Visual Sensitivity in Image Quality Assessment Framework," CVPR 2017
![Page 41: From Human Machine Interaction to Human Machine ... · From Interaction to Integration and Intelligence Interaction •Stimulus-response •Machine simply take task orders from human](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec911dc61588613f32911a2/html5/thumbnails/41.jpg)
THANK YOU VERY MUCH!
5/14/2018 Le Thanh Ha, HMI Laboratory 41