computer and machine visionecee.colorado.edu/~siewerts/extra/ecen5763/ecen5763_doc/...modern digital...
TRANSCRIPT
January 27, 2014 Sam Siewert
Computer and Machine
Vision
Lecture Week 3
Part-1
Outline of Week 3
Processing Images and Moving Pictures –
High Level View and Computer
Architecture for it
Linux Platforms for Computer/Machine
Vision
I/O, Memory and Processing Challenges
Sam Siewert 2
Old School Moving Picture Media
and Cameras NTSC OTA (1941, 1953 color, 2009 dead)
– Analog, Interlaced, Continuous Broadcast Transmission or CCTV (Closed Circuit TV)
– Coax Cable or Tuner with Immediate CRT Display
– No Buffers, No Routing, No De-mux
– No Compression
Analog Cable
AM/FM OTA
Film Projectors
Sam Siewert 3
Modern Digital Cameras Camera Link – High Frame Rates
– High Data Rates and Resolutions
– Industry Standard for Machine Vision Automation
– E.g. Inspection Systems
– E.g. – Sony, IDT, National Instruments
SD-SDI and HD-SDI – Standard and High Definition Synchronous Digital Interface
– Standard for Studios, Broadcast
Digital Cinema – Red Camera
– 1080p, 2K, 4K Resolutions and Much Higher
– Automated Digital Delivery and Projection
Webcams and Mobile Phone Cameras – Very Low Cost
– Proprietary
– Performance Varies Dramatically
Sam Siewert 4
Differences Analog vs Digital Encoding for Transmission
– Digital Allows for Image Processing
– Adds Latency
– Requires Compression for Packet Switched Networks and Storage
Routed (Diversely), Buffered
Compressed (MPEG, JPEG) to Lower Bit-rates
Multiplexed (Shares Transmission Carrier for Audio, Video, Channels)
Transported by IP (Large Packets)
Continuous Transmission – Analog or Constant Bit-Rate / Frame-Rate
Sam Siewert 5
E.g. UAV Latency and Jitter
Verification of Video Frame Latency Telemetry for UAV Systems Using A Secondary Optical Method, Sam Siewert, Muhammad Ahmad, Kevin Yao
Sam Siewert 6
NTSC (Analog TV)
Sam Siewert 7
AM Video to CRT
FM Audio
Chroma Added Later
Odd/Even Lines (Interlaced)
29.97 FPS (30 before color)
Vertical Blanking (CRT Retrace Time, Closed Captioning)
525 Lines, 262.5 per Field, 60 Fields per Second
http://en.wikipedia.org/wiki/File:Ntsc_channel.svg
Linux in Computer Vision
Embedded Solutions – Texas Instruments OMAP (Beagle xM, Bone)
– Numerous ARM SoCs (NVIDIA, Qualcomm, Broadcomm, …)
Scalable Solutions – Multi-Core (Xeon Phi)
– Vector Processing – CUDA, OpenCL GPU and GP-GPU
Computer and Machine Vision is I/O, Memory and Processing Intensive
Sam Siewert 8
Camera Interfaces
CCD (Charge Coupled Device) or CMOS (Common Metal Oxide Substrate) Detector – Integration Time for Photo-sensitive Elements in Array (to Build
up Charge)
– Read-out Time to Sample Elements in Array
Luminance and Chroma Analog to Digital Conversion
Double Buffer for Read-out + Processing
Frame Capture – http://www.cse.uaa.alaska.edu/~ssiewert/a485_doc/Frame-
Capture-Chips/
– Host Interface over PCI Bus or USB
Sam Siewert 9
Digital Video Transport QoS Latency – To Tune in a Program, Turn-on
– To Deliver a Video Frame or Audio PCM Sample
– To Start, FF, REW, Start-Over, Pause
Bandwidth – Resolution, Lossy/Lossless Compression, High Motion
– Pixel Encoding for Color
– Frame Rate
– Constant Bit-rate Transport?
– Variable Bit-rate Transport and Encoding?
Jitter – Decode and Presentation Rates
– Elasticity in Decode to Presentation Buffering Necessary
Sam Siewert 10
January 27, 2014 Sam Siewert
Linux System Options
(Linux for Image Processing, Camera
Interfacing and Computer Vision)
Sam Siewert 12
Processing Outline
Many-Core Linux Host(s)
– Intel Atom
– ARM
– Xeon
GP-GPU Vector Processing PCI-E Co-Processors
NVIDIA Tesla/Fermi
AMD ATI
NPTL – Native POSIX Threads Library
NPTL Example Code Walkthrough
Sam Siewert 13
Conceptual View of RT Resources Three-Space View of Utilization Requirements
– CPU Margin?
– IO Latency (and Bandwidth) Margin?
– Memory Capacity (and Latency) Margin?
Upper Right Front Corner – Low-Margin
Origin – High-Margin
Mobile – Must Consider Battery Life Too (Power)
CPU-Utility
IO-Utility
Memory-Utility
Processing – Initial Focus Processing and Scaling Frame Transformation, Encode, Decode is Critical
Memory for Buffering (Frame Transformations, CPU Integrated or GPU Offloaded – e.g. Linux VDPAU)
I/O for Networking (Transport)
I/O for Storage (On-Demand, Post, Non-Linear Editing)
Sam Siewert 14
Flynn’s Computer Architecture
Taxonomy Single Instruction Multiple Instruction
Single Data SISD (Traditional Uni-
processor)
MISD (Voting schemes
and active-active
controllers)
Multiple Data SIMD (e.g. SSE 4.2, GP-
GPU, Vector Processing)
MIMD (Distributed
systems (MPMD),
Clusters with MPI/PVM
(SPMD), AMP/SMP)
Sam Siewert 15
GPC has gone MIMD with SIMD Instruction Sets and SIMD Offload
(GP-GPU) NUMA vs. UMA (Trend away from UMA to NUMA or MCH vs. IOH)
SMP with One OS (Shared Memory, CPU-balanced Interrupt Handling,
Process Load Balancing, Mutli-User, Multi-Application, CPU Affinity
Possible)
MIMD - Single Program Multi-Data vs. Multi-Program Multi-Data
Computer and Machine Vision
Treated as a Real-time and/or Interactive
System
– Requires Predictable Response (By Deadline)
– Rate Monotonic
– Earliest Deadline First
– Least Laxity First
Covered in Depth in OS and Digital Media
(Just be Aware for Computer Vision) Sam Siewert 16
Sam Siewert 17
CPU Scheduling Taxonomy Execution Scheduling
Global-MP Local-Uniprocessor
Distributed Asymmetric
(AMP )
Symmetric
(SMP OS)
Preemptive Non-Preemptive
Fixed-Priority
Hybrid
Dynamic-Priority Cooperative
Batch
FCFS SJN
Co-Routine Continuation
Function
Heuristic EDF/LLF RR Timeslice
(desktop)
Multi-Frequency
Executives
Static Dynamic
Rate
Monotonic
Deadline
Monotonic
Dataflow
(Preemptive, Non-Preemptive Subtree
Under Each Global-MP Leaf)
SMT
(Micro-Paralell)
Sam Siewert 18
Response Latency Ci WCET
Input/Output Latency
Interference Time
Event
Sensed Interrupt Dispatch Preemption Dispatch
Interference
Completion
(IO Queued)
Actuation
(IO Completion)
Input-Latency
Dispatch-Latency
Execution Execution
Output-Latency
Time
Response Time = TimeActuation – TimeSensed
(From Release to Response)
Resource Scaling
Processing – Co-Processors – GPU – CUDA, OpenCL
– Many-Core – E.g. Intel Xeon Phi MICA
– FPGA – E.g. Altera Stratix
– Ideally Camera Interface
I/O – High Rate Transport – HD-SDI, Camera Link, GigE/10GE
Memory – SSD, PCIe Nand, NVM – FusionIO, Micron, Intel
– Memristor (Future)
Sam Siewert 19