signal processing in smartphones - 4g perspective

+

Signal Processing in Smartphones – 4G Perspective Ramesh Prasad

+About Me

n  M.E. in Electronics & Telecommunications Engineering

n  15+ Years of Technology Experience

n  Senior Member IEEE

n  Entrepreneur/Startup Enthusiast/Technologist/Evangelist/Author/Speaker

n  United States Patent 7,522,774

n  http://www.rpthegeek.com/

+Agenda

n  Introduction

n  Evolution of Mobile Communications

n  Smartphone Anatomy

n  Smartphone Hardware Architecture

n  Smartphone Software Architecture

n  Signal Processing In Smartphones

n  Communications Signal Processing

n  Speech Processing (VoIP)

+Agenda

n  Camera Image And Video Processing

n  Display Image Processing

n  Audio Post Processing

n  Signal Processing For Multimedia Communication

n  Geolocation Techniques & GPS Signal Processing

n  Signal Processing For Apps n  Voice Enabled Services n  Scanner and BarCode Reader n  Augmented Reality

n  References

+Introduction

+

Scanner & Bar Code Reader

Media Player

Internet

Email

Chat

Navigation

Games

TV

Phone

Credit Card

Apps Personal Assistant

Social Video Conf.

Bluetooth/ WiFi/NFC

Camera

+Technology Enablers

Signal Processing

Communications

VLSI

Mobile Computing

+Modern Communications Theory

+Evolution of Mobile Communications

+Wireless Technology Generations

+Wireless Technology Migration

+Introduction to 4G (LTE)

n  Standard for wireless communication of high-speed data for mobile phones and data terminals

n  Evolution of 3G systems

n  Developed by the 3GPP (3rd Generation Partnership Project)

n  OFDM (Orthogonal Frequency Division Multiplexing) to avoid ISI

n  MIMO (Multiple-Input Multiple-Output) to boost Data Rates

n  All-IP flat architecture supporting QoS

+Drivers For 4G

n  Creation and development of new services for mobile devices

n  Advancement of the Signal Processing and Communications technologies for mobile systems

n  Advancement of the VLSI technology

n  Competition between mobile operators

n  Challenges from other mobile technologies

n  New regulation of spectrum use

+Features Of 4G n  Provides a global ecosystem with inherent mobility

n  Offers easier access and use with greater security and privacy

n  Dramatically improves speed and latency

n  Delivers enhanced real-time video and multimedia for a better overall experience

n  Enables high-performance mobile computing

n  Supports real-time applications due to its low latency

n  Creates a platform upon which to build and deploy the products and services of today and those of tomorrow

n  Reduces cost per bit through improved spectral efficiency

+Benefits Of 4G

n  High peak speeds:

n  100 Mbps downlink (20 MHz, 2x2 MIMO)—both indoors and outdoors

n  50 Mbps uplink (20 MHz, 1x2)

n  At least 200 active voice users in every 5 MHz (i.e., can support up to 200 active phone calls)

n  Low latency:

n  < 5 ms user plane latency for small IP packets

n  < 100 ms camped to active

n  < 50 ms dormant to active

+Benefits Of 4G

n  Scalable bandwidth:

n  The 4G channel offers four times more bandwidth than current 3G systems and is scalable. So, while 20 MHz channels may not be available everywhere, 4G systems will offer channel sizes down to 5 MHz, in increments of 1.5 MHz.

n  Improved spectrum efficiency:

n  Spectrum efficiency refers to how limited bandwidth is used by the access layer of a wireless network. Improved spectrum efficiency allows more information to be transmitted in a given bandwidth, while increasing the number of users and services the network can support.

n  Two to four times more information can be transmitted versus the previous benchmark, HSPA Release 6.

+Benefits Of 4G n  Improved cell edge data rates:

n  Not only does spectral efficiency improve near cell towers, it also improves at the coverage area or cell edge.

n  Data rates improve two to three times at the cell edge over the previous benchmark, HSPA Release 6.

n  Packet domain only

n  Enhanced support for end-to-end quality of service: n  Reducing handover latency and packet loss is key to delivering a

quality service. n  This reduction is considerably more challenging with mobile

broadband than with fixed-line broadband. n  The time variability and unpredictability of the channel become more

acute. n  Additional complications arise from the need to hand over sessions

from one cell to another as users cross coverage boundaries. These handover sessions require seamless coordination of radio resources across multiple cells.

+Technical Specifications & Attributes

+3G Limitations

n  The maximum bit rates still are factor of 20 and more behind the current state of the systems like 802.11n and 802.16e/m.

n  The latency of user plane traffic (UMTS: >30 ms) and of resource assignment procedures (UMTS: >100 ms) is too big to handle traffic with high bit rate variance efficiently.

n  The terminal complexity for WCDMA or MC‐CDMA systems is quite high, making equipment expensive, resulting in poor performing implementations of receivers and inhibiting the implementation of other performance enhancements.

+3G vs 4G

+Challenges in 4G Mobile System Design


n  The goal of 4G has been made possible by sophisticated signal processing algorithms

n  Signal processing for 4G communications- n  Antenna

n  RF

n  Modulation

n  Baseband

n  Source coding

n  Channel coding

n  Signal Processing for Services and Apps

+Signal Processing Challenges in 4G

Performance

Cost Power


n  Data rate - Unfortunately, the growth of data rates is not matched by advanced in semiconductor structures, terminal power consumption improvements or battery lifetime improvements. New processing architectures and algorithms are needed to cope with these data rates.

n  High performance MIMO receivers such as maximum-likelihood-like receivers, sphere decoders or interference rejection combiners offer substantial performance gains but impose an implementation challenge, especially when the high peak data rates are targeted.


n  LTE utilizes precoding, which requires accurate channel estimation. Advanced methods like iterative decision directed channel estimation offer performance improvements, but pose again a complexity challenge.

n  LTE has a large “toolkit” of MIMO schemes and adaptive methods. The selection and combination of the right method in a cell with heterogeneous devices, channel conditions and bursty data services is a challenge.


n  LTE roll-out will be gradual in most cases– interworking with other standards such as GSM or HSPA will be required for a long time. This imposes not only a cost and complexity issue. One of the reasons many early 3G terminals had poor power consumption was the need for 2G cell search and handover in addition to normal 3G operation. Reduced talk-time for dual-mode devices is not acceptable.

n  “Dirty RF” challenges. OFDM systems cause problems with power amplifier nonlinearity, and are sensitive to frequency errors and phase noise. Digital compensation techniques are proposed for the transmitter and receiver, but innovation is needed to make them reality in low cost devices.

+Smartphone Anatomy

+Smartphone Hardware Architecture

+Hardware Architecture

+Single Chip Solution - SoC

+SoC – Functional Blocks

+Smartphone Software Architecture

+Software Architecture

+Software Architecture - Telephony

RF Driver

RF Modules

Protocol Stack Interface

Protocol Stack

Telephony Middleware

Telephony Driver

Telephony Apps

Telephony Middleware Interface

+Application Stack - Android

+Challenges & Solutions For Wireless Communications

+Challenges in Wireless Communications

n  Signal Propagation

n  Signal Attenuation and Path Loss

n  Multipath Effect

n  Signal Spread

n  Interference

n  Noise

+Signal Propagation Effect

+Signal Attenuation and Path Loss

+Multipath Effect

+Delay & Doppler Spread

+Interference

+Noise

n  Thermal Noise

n  White Noise

n  Flicker Noise

n  Phase Noise

n  Burst Noise

n  Shot Noise

n  Avalanche Noise

+Overcoming Challenges in Wireless Communications

n  Diversity – Time, Frequency, Space

n  Higher Order Modulation

n  Channel Coding

n  Interleaving

n  Channel Estimation & Equalization

n  Multi Carrier Transmission

+Fundamental Limits on Data Rates

+Fundamental Limits on Data Rates

n  To provide high data rates as efficiently as possible, the transmission bandwidth should be at least of the same order as the data rates to be provided

n  For provisioning of higher data rates with good coverage, wider transmission bandwidth is required

n  Challenges n  Spectrum is a scarce and expensive resource n  The use of wider transmission and reception bandwidths has an

impact on the complexity of the radio equipment, both at the base station and at the terminal.

n  increased corruption of the transmitted signal due to time dispersion on the radio channel

+Multi-Carrier Transmission

M symbols transmitted in parallel, so rate R becomes R/N

+Principles of OFDM

+OFDM Transmitter Receiver

+Simplified Realization of OFDM

+OFDMA

+SC - FDMA

+SC-FDMA Transmitter Receiver

+Signal Processing For Communications

+Protocol Stack in 4G Handset

+Protocol Stack - PHY

n  To support the high data rate, exceptionally large amounts of processing power are needed, particularly in the baseband, where all the error handling and signal processing occurs.

n  The LTE PHY employs orthogonal frequency division multiplexing (OFDM) and multiple input multiple output (MIMO) data transmission.

n  The LTE PHY uses orthogonal frequency division multiple access (OFDMA) on the downlink (DL) and single carrier – frequency division multiple access (SC-FDMA) on the uplink (UL).

+Signal Processing in PHY

Downlink Uplink

+PHY Layer in 4G Handset

+Speech Processing (VoIP)

+Speech Coder Attributes

n  Bitrate

n  Quality

n  Complexity

n  Delay

n  Robustness

+VoIP Technology Barriers

n  End-to-End Delay and Jitter

n  Packet Loss

n  Throughput

n  Internet Availability and Reliability

n  Security and Confidentiality

+VoIP Over Wireless

n  Channel Quality and Adaptive Operation

n  Background Noise

n  Tandeming

n  Voice Activity Detection

n  Unequal Error Protection

n  Frame Erasures

+AMR Wideband

n  Adaptive Multi-Rate Wideband (AMR-WB) is a patented wideband speech coding standard developed based on Adaptive Multi-Rate encoding, using similar methodology as Algebraic Code Excited Linear Prediction (ACELP).

n  AMR-WB provides improved speech quality due to a wider speech bandwidth of 50–7000 Hz compared to narrowband speech coders which in general are optimized for POTS wireline quality of 300–3400 Hz.

n  AMR-WB codec has the following parameter:

n  Bit Rate – 16 Kbps n  Delay frame size: 20 ms n  Look ahead: 5ms n  Complexity: 38 WMOPS, RAM 5.3KWords n  Voice activity detection, Discontinuous Transmission, Comfort Noise Generator n  Fixed point: Bit-exact C

+AMR Wideband Encoder

+AMR Wideband Decoder

+Camera Image And Video Processing

+Smartphone Camera Features

n  Video Camcorder

n  Still Image Capture

n  Auto Focus & Metering

n  Low Light Photography

n  Zero Shutter Lag Image Capture & Video Snapshot

n  Face Detection & Face Tracking

n  Imaging Responsiveness

n  Video Editing

n  Image Post Processing

+Image Processing Pipeline

+Black-Level Adjustment

n  This stage in the pipeline adjusts for dark current from the sensor and for lens flare, which can lead to the whitening of an image’s darker regions. In other words, sensor black is not the same as image black. The most common method for calculating this adjustment is to take a picture of a completely black field (typically accomplished by leaving the lens cap on), resulting in three base offsets to be subtracted from the raw sensor data. Failure to adjust the black level will result in an undesirable loss of contrast.

+Noise Filtering

n  There are numerous sources of noise that can distort image data – optical, electrical, digital and power

n  The actual noise level present in an image, however, plays a critical role in determining how strong the noise filter must be since the use of a strong filter on a clean image will actually distort and blur the image rather than clear it up.

n  Noise reduction is achieved by averaging similar neighboring pixels. Through the use of an Optical Electrical Conversion Function (OECF) chart and a uniform lighting source, the noise level can be characterized for different intensities.

+Noise Filtering

n  If the noise level is high for a particular intensity, then more weight is given to the average pixel value of similar neighbors.

n  On the other hand, if the noise level is low, more weight is given to the original pixel value.

n  The OECF chart is comprised of 12 uniform gray patches and produces 12 corresponding power levels based on the noise standard deviation at the mean value for each intensity/luminance level. These 12 power levels are then used to reduce noise across an image using either a linear or square-root model, depending on the sensor and gain (or ISO) level.

+Noise Filtering

+White Balance

n  Different types of lighting – such as incandescent, fluorescent, natural light sources, LED flash – have a pronounced effect on color.

n  The most difficult to tune is in mixed-light conditions.

n  White balance automatically compensates for color differences based on lighting so white actually appears white.

+White Balance

+White Balance

n  Fine tuning white balance begins by measuring the average RGB values across the six gray patches on a ColorChecker chart

+White Balance

n  Using mean square error minimization, the appropriate gains for each color can be calculated.

The resulting gains are applied to each image pixel:

+CFA Interpolation

n  Typically, digital cameras employ only a single sensor to capture an image, so the camera can only obtain a single color component for each pixel even though three components are necessary to represent RGB color.

n  CFA interpolation is the process of interpolating two missing color components for each pixel based on the available component and neighboring pixels.

n  CFA interpolation is primarily a transform function that does not vary based on sensor or lighting conditions, and therefore no tuning of this image-processing pipeline stage is required.

n  However, it is still one of the most complex algorithms in the image-processing pipeline

+CFA Interpolation

+RGB Blending

n  Different sensors produce different RGB values for the same color.

n  Tuning this pipeline stage involves creating a blending matrix to convert the sensor RGB color space to a standard RGB color space such as the Rec709 RGB color space.

+RGB Blending

n  The blending matrix is calculated by starting with a ColorChecker chart and obtaining average RGB values for 18 different color patches (the top three rows of the chart) that have already been white balanced.

n  Next, inverse Gamma correction is applied to the reference RGB values. The blending matrix is then constructed using constrained minimization.

+RGB Blending

n  The blending matrix is applied as follows:

The final result is consistent color between cameras using different sensors.

+Gamma Correction

n  Gamma correction compensates for the nonlinearity of relative intensity as the frame buffer value changes in output displays.

n  Typically, displays are calibrated using a standard gamma correction such as Rec709 or SMPTE240M.

n  Calibrating the image-processing pipeline to the same standards ensures optimal image quality across the majority of displays.

+RGB to YCC Conversion

n  Images also need to be adjusted for the human eye, which is more sensitive to luminance (Y) than color (Cb, Cr) information. This pipeline stage separates luminance from color for different processing using different precisions.

+Edge Enhancement

n  Edge enhancement affects the sharpness of an image.

n  In low-light conditions where images have a lower signal-to-noise ratio (SNR), edge enhancement will boost noise making it more visible to users.

+Contrast Enhancement

n  Contrast enhancement is comprised of two parameters: contrast and brightness.

n  Since the optimal contrast and brightness vary based on the particular lighting conditions, as well as upon user preference, these parameters are often implemented so that they can be dynamically selected by the user.

+Contrast Enhancement

+False Chroma Suppression

n  The final stage in the image-processing pipeline corrects various color artifacts.

n  Chroma suppression is a color correction that neutralizes the green or blue colored light that often bounces off a green screen or blue screen background and tints the edges of a subject during a shoot.

+Display Image Processing

+Audio Post Processing

+Signal Processing For Multimedia Communication

+Applications

n  Video-on-demand.

n  Distance learning and training.

n  Interactive gaming.

n  Remote shopping.

n  Online media services, such as news reports.

n  Videotelephony.

n  Videoconferencing.

n  Telemedicine for remote consultation and diagnosis.

n  Telesurveillance.

n  Remote consultation or scene-of-crime work.

n  Collaborative working and telepresence.

+Challenges

n  Higher coding efficiency

n  Reduced computational complexity

n  Improved error resilience

+MultiMedia Communications

+Challenges

n  Bandwidth limitations of communication channels.

n  Real-time processing requirements.

n  Inter-media synchronization.

n  Intra-media continuity.

n  End-to-end delays and delay jitters.

n  Multimedia indexing and retrieval.

+Audio/Video Streaming

n Increase in demand for fast and location independent multimedia access

n Driving force behind growth of Mobile Internet

n “Killer App” needed for success of 3G/4G systems

n Broad range of Applications – from Entertainment to Telemedicine

+What is Streaming

n  Is it File download?

n  Real Time consumption of data

n  Advantages n  Low initial delay

n  Media protection

n  Lower Memory usage

+Challenges in Video Streaming

n  Bit-rate

n  Packet Errors/Losses

n  Buffer Management

n  Low Delay

n  Low Jitter

n  No Re-transmission

+What if Network is Wireless ?

n  Higher Attenuation

n  Shadowing

n  Fading

n  Multi-user interference

n  Mobility/Handoff

n  Low Processing power and Memory

+Solution

n  Low Bit-rate Codec

n  Error Resiliency

n  Error Concealment

n  Prevent Spatio-Temporal Error Propagation

n  Flexibility of Bit-stream

n  Feedback from Transport layer

+Network Architecture

+Protocol Architecture

RTSP RTCP RTP

TCP UDP

IP

SESSION SETUP & CONTROL QUALITY FEEDBACK

MEDIA DELIVERY(AUDIO & VIDEO)

+RTCP-Real Time Control Protocol

n Adaptive encoders and streaming servers can utilize the feedback information for adjusting the stream to match the current transport quality

n Feedback is delivered in RTCP sender and receiver reports

n periodic transmission of control packets to all participants in the session

+Video Coding – H.264

n  Massive Quality, Minimal Files

n  Scalable from 3G to HD and Beyond

n  The New Industry Standard

n  Latest Innovations in Video Technology

n  Outperforms all preceding standards

+H.264 - Innovations

n multi-frame motion-compensated prediction

n  adaptive block size for motion compensation

n generalized B-Pictures

n quarter-pel motion accuracy

n intra coding utilizing prediction in the spatial domain

n in-loop de-blocking filters

n efficient entropy coding methods

+H.264 – Block Diagram

+H.264 - Performance

Use Scenario Resolution & Frame Rate Example Data Rates

Mobile Content 176x144, 10-15 fps 50-60 Kbps

Internet/Standard

Definition 640x480, 24 fps 1-2 Mbps

High Definition 1280x720, 24fps 5-6 Mbps

Full High Definition 1920x1080, 24fps 7-8 Mbps

+H.264 – Bandwidth Adaptation

n Send one of several pre-encoded versions of the same content, based on the current channel bit-rate

n Frame dropping of non-reference frames, if the channel rate fluctuates only in a small range

n Instantaneous decoder refresh (IDR) pictures to compensate large scale variations of the channel rate

+H.264 – Error Probability Reduction

n Slice-structured coding – Slices are independently coded

n Short slice/packets reduce the amount of lost information

n Probability of a bit-error hitting a short packet is generally lower than for large packets

n Short packets reduce the amount of lost information and, hence, the error is limited

+H.264 – Error Resilience

n  Flexible MB Ordering (FMO)

n  Data partitioning - unequal error protection

+H.264 – Error concealment

n  Intra coded MBs

n  Multiple Reference Frames

n  Redundant Slices

+ H.264 – Intra Frame Error Concealment

+ H.264 – Inter Frame Error Concealment

+Geolocation Techniques & GPS Signal Processing

+Geolocation Techniques

n  For wireless communication networks, an inherently suitable approach for wireless geolocation is known as radiolocation, in which the parameters that are used for location estimation are obtained fromradio signal measurements between a target and one or more fixed stations.

n  GPS is often inoperable in areas where satellites are blocked, such as in buildings and built-up urban areas.

n  Further, the time-to-first-fix (TTFF) for a conventional GPS receiver from a “cold” start can take several minutes.

n  Additionally, adding GPS functionality to a handset can be costly, bulky, and drain battery power at an unacceptable rate

+Geolocation Methods

n  The parameters that are often measured and used for location include- n  angles of arrival (AOAs),

n  signal strength, times of arrival (TOAs), and

n  time differences of arrival (TDOAs).

n  The fundamental methods of radiolocation that use these parameters can be grouped into three categories: n  direction finding,

n  ranging, and

n  range differencing.

+Direction Finding

+Ranging

+Range Differencing

+Algorithms For Geolocation

n  Geometrical Techniques

n  When there are errors in the measured AOAs, ranges, or range differences, statistical solutions are more justifiable in the presence of measurement errors

n  Least Squares Estimation - estimating parameters by minimizing the squared discrepancies between observed data, on the one hand, and their expected values on the other

+GPS Signals

+GPS Receiver

Adaptive Space Time Array Receiver

+Signal Processing For Apps

+Voice Enabled Services

n  Google Voice Search Video

+Distributed Speech Recognition

+Scanner & Bar Code Reader

+2D Bar Code

n  Defining of threshold as pre-processing,

n  Detecting a black bar using spiral search method, and

n  Finding the sampled scanning line which is perpendicular to the detected bar in phase

+QR Code n  Pre-processing The gray level histogram calculation is

adopted,

n  Corner marks detection Three marked corners are detected using the finder pattern,

n  Fourth corner estimation The fourth corner is detected using the special algorithm,

n  Inverse perspective transformation Inverse transformation is adopted based on the obtained corner geometry positions to normalize the size of the code, and

n  Scanning of code Sample the inside of code and output the normalized bi-level code data to host CPU.

+Visiting Card Scanner

+Augmented Reality

n  Augmented Reality Video

+Image Matching

Query Image

Prefetched Data

Database Images

+System Architecture

Geo-Tagged Images

Group Images by Loxel

Extract Features

Camera Image

Extract Features

Compute Feature Matches

Geometric Consistency Check

Match Images

Geometric Consistency Check

Display Info for Top Ranked Image

Device Location

Feature Cache

ANN

Loxel-Based Feature Store

Server

Net

wor

k

Cluster Features

Prune Features

Compress Descriptors

+References

+References

n  Mobile Handset Design – Sajal Das

n  Signal Processing for Mobile Communications Handbook –Mohamed Ibnkahla

n  4G LTE/LTE-Advanced for Mobile Broadband - Erik Dahlman