gs2k ov720p hd full duplex audio video adk application note · fault-tolerant and are not designed,...

GS2K OV1080p HD Full-Duplex Audio Video ADK Application Note

80560NT11734A Rev. 0 – 2019-01-22


80560NT11734A Rev. 0 of 65 2019-01-22

NOTICE

COPYRIGHTS

COMPUTER SOFTWARE COPYRIGHTS

SPECIFICATIONS ARE SUBJECT TO CHANGE WITHOUT NOTICE

While reasonable efforts have been made to assure the accuracy of this document, Telit assumes no liability resulting from any inaccuracies or omissions in this document, or from use of the information obtained herein. The information in this document has been carefully checked and is believed to be reliable. However, no responsibility is assumed for inaccuracies or omissions. Telit reserves the right to make changes to any products described herein and reserves the right to revise this document and to make changes from time to time in content hereof with no obligation to notify any person of revisions or changes. Telit does not assume any liability arising out of the application or use of any product, software, or circuit described herein; neither does it convey license under its patent rights or the rights of others. It is possible that this publication may contain references to, or information about Telit products (machines and programs), programming, or services that are not announced in your country. Such references or information must not be construed to mean that Telit intends to announce such Telit products, programming, or services in your country.

This instruction manual and the Telit products described in this instruction manual may be, include or describe copyrighted Telit material, such as computer programs stored in semiconductor memories or other media. Laws in the Italy and other countries preserve for Telit and its licensors certain exclusive rights for copyrighted material, including the exclusive right to copy, reproduce in any form, distribute and make derivative works of the copyrighted material. Accordingly, any copyrighted material of Telit and its licensors contained herein or in the Telit products described in this instruction manual may not be copied, reproduced, distributed, merged or modified in any manner without the express written permission of Telit. Furthermore, the purchase of Telit products shall not be deemed to grant either directly or by implication, estoppel, or otherwise, any license under the copyrights, patents or patent applications of Telit, as arises by operation of law in the sale of a product.

The Telit and 3rd Party supplied Software (SW) products described in this instruction manual may include copyrighted Telit and other 3rd Party supplied computer programs stored in semiconductor memories or other media. Laws in the Italy and other countries preserve for Telit and other 3rd Party supplied SW certain exclusive rights for copyrighted computer programs, including the exclusive right to copy or reproduce in any form the copyrighted computer program. Accordingly, any copyrighted Telit or other 3rd Party supplied SW computer programs contained in the Telit products described in this instruction manual may not be copied (reverse engineered) or reproduced in any manner without the express written permission of Telit or the 3rd Party SW supplier. Furthermore, the purchase of Telit products shall not be deemed to grant either directly or by implication, estoppel, or otherwise, any license under the copyrights, patents or patent applications of Telit or other 3rd Party supplied SW, except for the normal non-exclusive, royalty free license to use that arises by operation of law in the sale of a product.


80560NT11734A Rev. 0 of 65 2019-01-22

USAGE AND DISCLOSURE RESTRICTIONS

I. License Agreements

II. Copyrighted Materials

III. High Risk Materials

IV. Trademarks

V. Third Party Rights

The software described in this document is the property of Telit and its licensors. It is furnished by express license agreement only and may be used only in accordance with the terms of such an agreement.

Software and documentation are copyrighted materials. Making unauthorized copies is prohibited by law. No part of the software or documentation may be reproduced, transmitted, transcribed, stored in a retrieval system, or translated into any language or computer language, in any form or by any means, without prior written permission of Telit

Components, units, or third-party products used in the product described herein are NOT fault-tolerant and are NOT designed, manufactured, or intended for use as on-line control equipment in the following hazardous environments requiring fail-safe controls: the operation of Nuclear Facilities, Aircraft Navigation or Aircraft Communication Systems, Air Traffic Control, Life Support, or Weapons Systems (High Risk Activities"). Telit and its supplier(s) specifically disclaim any expressed or implied warranty of fitness for such High-Risk Activities.

TELIT and the Stylized T Logo are registered in Trademark Office. All other product or service names are the property of their respective owners.

The software may include Third Party Right software. In this case, you agree to comply with all terms and conditions imposed on you in respect of such separate software. In addition to Third Party Terms, the disclaimer of warranty and limitation of liability provisions in this License shall apply to the Third-Party Right software. TELIT HEREBY DISCLAIMS ANY AND ALL WARRANTIES EXPRESS OR IMPLIED FROM ANY THIRD PARTIES REGARDING ANY SEPARATE FILES, ANY THIRD PARTY MATERIALS INCLUDED IN THE SOFTWARE, ANY THIRD PARTY MATERIALS FROM WHICH THE SOFTWARE IS DERIVED (COLLECTIVELY “OTHER CODE”), AND THE USE OF ANY OR ALL THE OTHER CODE IN CONNECTION WITH THE SOFTWARE, INCLUDING (WITHOUT LIMITATION) ANY WARRANTIES OF SATISFACTORY QUALITY OR FITNESS FOR A PARTICULAR PURPOSE. NO THIRD PARTY LICENSORS OF OTHER CODE SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT LIMITATION LOST PROFITS), HOWEVER CAUSED AND WHETHER MADE UNDER CONTRACT, TORT OR OTHER LEGAL THEORY, ARISING IN ANY WAY OUT OF THE USE OR DISTRIBUTION OF THE OTHER CODE OR THE EXERCISE OF ANY RIGHTS GRANTED UNDER EITHER OR BOTH THIS LICENSE AND THE LEGAL TERMS APPLICABLE TO ANY SEPARATE FILES, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.


80560NT11734A Rev. 0 of 65 2019-01-22

APPLICABILITY TABLE

PRODUCT

GS2K based Modules

SW Version

5.7.2 onwards

Note: The features described in the present document are provided by the products equipped with the software versions equal or higher than the versions shown in the table. See also the Revision History chapter.


80560NT11734A Rev. 0 of 65 2019-01-22

Revision History

Version Date Remarks

0 Jan 2019 Initial release


80560NT11734A Rev. 0 of 65 2019-01-22

Table of Contents NOTICE……… ........................................................................................................................... 2

COPYRIGHTS. ........................................................................................................................... 2

COMPUTER SOFTWARE COPYRIGHTS ................................................................................. 2

USAGE AND DISCLOSURE RESTRICTIONS .......................................................................... 3

APPLICABILITY TABLE ........................................................................................................... 4

CHAPTER 1. INTRODUCTION .............................................................................................. 11

1.1. SCOPE ........................................................................................................................ 11 1.2. OVERVIEW................................................................................................................... 11 1.3. TERMINOLOGY ............................................................................................................. 12 1.4. STANDARDS AND CONCEPTS ........................................................................................ 13

CHAPTER 2. FEATURES AND CAPABILITIES .................................................................... 14

CHAPTER 3. THEORY OF OPERATION .............................................................................. 15

CHAPTER 4. ARCHITECTURE ............................................................................................. 16

4.1. COMPONENT ............................................................................................................... 17 4.1.1. Hardware ........................................................................................................ 18 4.1.2. Software ......................................................................................................... 21

4.2. TASKS ......................................................................................................................... 22 4.3. PROCESS FLOW .......................................................................................................... 23 4.4. BUTTON AND ITS FUNCTIONALITIES ............................................................................... 24 4.5. TASK INTERFACE ARCHITECTURE FOR APPLET–1 .......................................................... 26 4.6. TASK INTERFACE ARCHITECTURE FOR APPLET–3 .......................................................... 30 4.7. GS-OV INTERFACE ...................................................................................................... 31

4.7.1. Load OV Binary .............................................................................................. 31 4.7.2. Read OV Video Data ...................................................................................... 32 4.7.3. Read OV Audio Data ...................................................................................... 33 4.7.4. Write Audio Data ............................................................................................ 34

CHAPTER 5. EMBEDDED APPLICATION EXECUTION SEQUENCE ................................. 35

5.1. PROGRAM OV BINARY IN GS NODE .............................................................................. 37 5.1.1. Method 1: Upload OV Binary to GS Module ................................................... 37 5.1.2. Method 2: Load OV Binary into External Flash via S2W ................................ 38 5.1.3. Method 3: Load OV Binary Directly to OV Chip via UART Interface ............... 38

5.2. VIDEO APPLICATION EXECUTION ................................................................................... 40 5.2.1 GS Boot up ..................................................................................................... 40 5.2.2. Load OV from External Flash to OV Chip via SPI ........................................... 41 5.2.3. Start RTSP Server (and RTP Task) ................................................................ 42 5.2.4. RTSP Message Exchanges ............................................................................ 43 5.2.5. Start Forward Audio and Video Streaming ..................................................... 48 5.2.6. Reverse Audio Streaming ............................................................................... 52 5.2.7. Video Rate Adaptation using Remote RTCP Information and Local Data ...... 53

5.3. CONFIGURATIONS ........................................................................................................ 57 5.3.1. Remote Configuration of Video Encoder Settings .......................................... 58

APPENDIX A REFERENCE CODES ...................................................................................... 60

A.1.1 RTSP SDK REFERENCE CODE FLOW ........................................................................... 60 A.1.2 RTP VIDEO SDK REFERENCE CODE FLOW ................................................................... 60 A.1.3 RTP AUDIO SDK REFERENCE CODE FLOW ................................................................... 60


80560NT11734A Rev. 0 of 65 2019-01-22

A.1.4 REVERSE AUDIO SDK REFERENCE CODE FLOW ........................................................... 61 A.1.5 VIDEO ENCODER INTERFACE CODE FLOW ..................................................................... 61 A.1.6 SIF PROTOCOL ............................................................................................................ 61


80560NT11734A Rev. 0 of 65 2019-01-22

List of Figures Figure 1: Client Mode .................................................................................................................... 16

Figure 2: Hardware Components – Front Side .............................................................................. 18

Figure 3: Hardware Components – Rear Side ............................................................................... 19

Figure 4: IO Expander ................................................................................................................... 20

Figure 5: Task Interface Architecture ............................................................................................ 22

Figure 6: Process Flow Diagram ................................................................................................... 23

Figure 7: Task Structure – Applet-1 ............................................................................................... 26

Figure 8: Task Architecture for Applet- 3 ....................................................................................... 30

Figure 9: Loading OV Binary ......................................................................................................... 31

Figure 10: Reading OV Video Data ............................................................................................... 32

Figure 11: Reading OV Audio Data ............................................................................................... 33

Figure 12: Video ADK - Forward Audio ......................................................................................... 34

Figure 13: Video ADK - Reverse Audio ......................................................................................... 34

Figure 14: Firmware uploads Screen ............................................................................................ 37

Figure 15: Serial Port Selection ..................................................................................................... 38

Figure 16: OV Binary in Program Mode ........................................................................................ 38

Figure 17: Transfer Window .......................................................................................................... 39

Figure 18: OV Binary Location ...................................................................................................... 39

Figure 19: OV Binary in RUN Mode .............................................................................................. 40

Figure 20: GS UART Print after GS Module Restart ..................................................................... 40

Figure 21: RTSP Communication between GS Module and Smartphone ..................................... 44

Figure 22: Sample RTSP Packets ................................................................................................. 44

Figure 23: Fields in Actual Data .................................................................................................... 49

Figure 24: Fragmentation Unit ....................................................................................................... 50

Figure 25: RTP Packet Header ..................................................................................................... 51

Figure 26: Sample RTP Packets ................................................................................................... 52

Figure 27: Sample TEARDOWN Message .................................................................................... 52

Figure 28: SPI and Signaling Configurations ................................................................................. 57

Figure 29: Header Information ....................................................................................................... 62

Figure 30: Payload ........................................................................................................................ 62


80560NT11734A Rev. 0 of 65 2019-01-22

Figure 31: Get Data ....................................................................................................................... 62

Figure 32: ACK/NACK Stage ......................................................................................................... 63


80560NT11734A Rev. 0 of 65 2019-01-22

List of Tables Table 1: Glossary of Terms ........................................................................................................... 12

Table 2: Hardware Components .................................................................................................... 19

Table 3: Software Components ..................................................................................................... 21

Table 4: GS2200M Button and Related Pins ................................................................................. 24

Table 5: Button Functionalities ...................................................................................................... 25

Table 6: Description of Fields in RTP Packet ................................................................................ 51

Table 7: Profile and its Corresponding Rates ................................................................................ 53

Table 8: Profile and its Quality ....................................................................................................... 55


80560NT11734A Rev. 0 of 65 2019-01-22

Chapter 1. Introduction

1.1. SCOPE This document provides an overview about the Telit GS2000® based HD Full-Duplex Audio Video Application Development Kit (ADK). It describes the software and hardware architecture, network topology, functional features, API’s, and Application execution sequence of the Embedded and Mobile applications. The Video ADK provides a complete video reference design that allows users to stream video and audio data to their iOS or Android based smartphones and receives audio from smart phone or PC that played it on ADK.

This document assumes that the reader is generally familiar with Telit Wi-Fi products, Internet Protocol (IP) networks and the operation of 802.11 wireless devices.

1.2. OVERVIEW Telit HD Full-duplex Audio Video ADK is a development platform enabling customers to build Wi-Fi IP video streaming applications with HD video quality. By integrating the major hardware and software components for Wi-Fi IP camera solution, this reference platform facilitates fast time to market for customer projects.

Targeted applications for this platform include Wi-Fi IP cloud cameras, baby monitors, toy cameras, automotive backup/drive recorder cameras, and variety of other systems where customer would like to add IP video streaming functionality.

NOTE:

GainSpan SDK is mandatory for any development on OV1080p HD Full-duplex Audio Video ADK.


80560NT11734A Rev. 0 of 65 2019-01-22

1.3. TERMINOLOGY Following table lists the different terminologies used in this document.

Table 1: Glossary of Terms

Term Explanation

ADK Application Development Kit

AP Access Point

API Application Programmer’s Interface

DIN Data Input

DOUT Data Output

HD High Definition

IP Internet Protocol

OTAFU Over the Air Firmware Update

RTP Real time Transport Protocol

RTSP Real Time Streaming Protocol

SPI Serial Peripheral Interface

STA Client Station

TCP Transmission Control Protocol

UART Universal Asynchronous Receiver/Transmitter

UDP User Datagram Protocol

VGA Video Graphics Array

RTCP RTP Control Protocol

WPS Wi-Fi Protected Setup


80560NT11734A Rev. 0 of 65 2019-01-22

1.4. STANDARDS AND CONCEPTS Standards

Generic:

None

ADK Specific:

• RTP • RTSP • SIF • RTCP

Concepts

OV1080p HD Full-duplex Audio Video ADK uses RTP/RTSP streaming mechanism.


80560NT11734A Rev. 0 of 65 2019-01-22

Chapter 2. Features and Capabilities

This section provides the features and capabilities of the Telit OV1080p HD Full-duplex Audio Video ADK:

Features:

• Resolution

o Video resolution: 1280*1080

o Bit rate: 1 Mbps

o Frame rate: 30 fps

• Format: h.264

• Audio

o Sample rate: 8k

o Formats: G711 (Linear PCM)

• Provisioning

• OTAFU

• mDNS/DNS-SD methods

• Standby

• Wakeup on Wi-Fi

• Applet mechanism

• Hibernate

Capabilities:

• It demonstrates dynamic loading of video application based on conditions. • It demonstrates saving of network parameters before going into standby and restoring the

same once GS node comes out of standby thereby removing the need of establishing new session across standby.

• It demonstrates putting the device in Hibernate mode which is the lowest power consumption mode available.

• It demonstrates waking up from Wi-Fi beacon being in lowest power consumption mode. • In Hibernate mode it sends video snap shot data to user and waits for few seconds

(configurable) for user request to stream. • In Hibernate mode, switch to PSPOLL mode while waiting for user request.


80560NT11734A Rev. 0 of 65 2019-01-22

Chapter 3. Theory of Operation

The Telit OV1080p HD Full-duplex Audio Video ADK demonstrates wireless video streaming using OV798 video encoder chip along with OV2736 camera sensor and two-way audio streaming using Microsemi ZL38051. It uses RTSP for control path and RTP for video & audio data. For reverse audio, GS module acts as a client, establishes UDP connection with smartphone as a server and receives the data.

When the board is powered ON, it comes up as Limited AP, ready for verified provisioning, and starts the RTSP server and RTP tasks. It cannot stream once provision starts RTSP Server. The smartphone/laptop connects to GS module as STA. When VLC is launched, it acts as an RTSP client and starts streaming and h.264 data in RTP packets is received. When a network stream is opened, the URL that needs to be entered while using VLC is rtsp://<GS2K IP ADDRESS> /live.h264. The Video ADK can be operated in any one of the three modes (Use Cases) as mentioned below:

• Always ON: This mode is used when there is line power supply available. • Wake on wireless (Deep Sleep/Standby): Based on beacon interval it will be on Deep

sleep or in Standby mode. • Event based Wakeup (Alarm): The module will wake up based on event otherwise it’ll be

on hibernate mode. The mode type can be selected while building the binary from the SDK Builder.


80560NT11734A Rev. 0 of 65 2019-01-22

Chapter 4. Architecture

The Video Low Power ADK architecture consists of GS node working in limited AP/STA mode. Limited-AP mode is used only for provisioning the module. It is in STA mode that GS node showcases low power functionality.

The Telit Video LP works in two Low Power modes:

PSPOLL

HIBERNATE

PSPOLL Mode

Figure 1: Client Mode

On wake up from standby, Base code loads into RAM check for any Data request packet notification from Access Point. If there is no packet it goes to standby and wakes up every DTIM interval. This cycle repeats.

Once mobile application opens on smart phone or Open VLC player and request for video using RTSP URL it sends mdns/rtsp packet to find Video service in LAN.GS Video module gets notification through mdns/rtsp packet and starts the RTSP Server and waits for PLAY request. The video mobile application uses the discovery feature provided by the video embedded firmware application to enable automatic discovery of the available video profiles and services.

Client launches the video application to initiate RTSP communication and acts as an RTSP client in this process. It fetches h.264 video and G711 audio data from OV chip through SPI interface, fragments and formats both video and audio data, and sends it over RTP/UDP. In reverse audio, the audio data is received from the smartphone and is sent to Micro semi decoder through OV798 chip.

Hibernate mode:

In Hibernate mode, GS node wakeup on Button Press notification and sends video snap shot data to pre-configured Server. After snap shot it starts RTSP Server application and waits for PLAY request. On smart phone open GS Video app discovery video services that launches RTSP client and request for data.


80560NT11734A Rev. 0 of 65 2019-01-22

4.1. COMPONENT The HD Full-Duplex Audio Video ADK components consist of:

• External flash (where OV image is optionally stored) • GS Wi-Fi module and embedded application • OV798 video encoder and its firmware • OV2736 digital camera sensor • ZL38051 audio encoder and decoder • LEDs indicators • Native application • I/O expander (SX1503 16 Channel)

The video application board supports the Wi-Fi HD video streaming capability of the GS module which,

1. Has a GS Wi-Fi module and embedded application package to support its prescribed set of features and functionalities.

2. Connects to the OV798 chip through SPI serial interface which fetches data from Camera sensor OV2736.

3. OV798 chip has also interface with Microsemi’s ZL38051 audio processor which does both audio encoding and decoding.

4. Has an external serial flash to store the OV binaries.

5. Has various LED indicators for indicating,

o AP mode: IO expander glows Red

o Client mode: IO expander glows Red

o OV power on: IO expander glows Green

6. Has IO expander which provides additional GPIO’s for future development. It creates I2C interface between GS and OV and helps to reset OV before any other functionality is started.

Native Application

GainSpan native app on Android is used as an application for video and two-way audio streaming.


80560NT11734A Rev. 0 of 65 2019-01-22

4.1.1. Hardware The Telit HD Full-duplex Audio Video ADK uses GS2200M Wi-Fi module-based video application board that consists of the following hardware components.

Figure 2: Hardware Components – Front Side

921600 BAUD RATE SELECTOR IN PGM MODE

OV USB GS UART GS2200MIE

PGM/RUN MODE SWITCH

OV2736 CAMERA MODULE

POWER ON/OFF

OV798

DC JACK

OV UART


80560NT11734A Rev. 0 of 65 2019-01-22

Figure 3: Hardware Components – Rear Side

Table 2: Hardware Components

Component Description

Telit Wi-Fi Module GS2200M module streams video data over Wi-Fi.

Omnivision OV798 Omnivision H.264 video compression chip which supports up to 30 fps VGA and HD resolution.

Omnivision OV2736 HD Image

Sensor and Lens module

Omnivision OV2736 HD image sensor and lens module which supports 1280*1080 video resolutions.

Microsemi ZL38051 Microsemi audio processing chip which supports G711 A/U law with 16k sampling rate.

GS Serial Flash Used for storage of provisioning, custom application web pages, backup firmware, and OV firmware.

LEDs Indicates Power On, Operation mode (Limited AP or Client) and Run or Program mode.

SPEAKER

ZL38051


80560NT11734A Rev. 0 of 65 2019-01-22


GS USB port Used to power the board and upgrade firmware on the Wi-Fi module.

I/O Expander It is connected to GS I2C port and is used to enable power to OV798, ZL38051, reset OV and glow AP/Client mode LED’s.

4.1.1.1. I/O Expander The SX1501, SX1502 and SX1503 are complete ultra-low voltage General Purpose Parallel Input/output (GPIO) expanders ideal for low power handheld battery powered equipment. They allow easy serial expansion of I/O through a standard I2C interface GPIO devices and can provide additional control and monitoring when the microcontroller or chipset has insufficient I/O ports, or in systems where serial communication and control from a remote location is advantageous.

These devices can also act as a level shifter to connect a microcontroller running at one voltage level to a component running at a different voltage level. The core operates at as low as 1.2V while the I/O banks can operate between 1.2V and 5.5V independent of the core voltage and each other. Each GPIO is programmable via 8-bit configuration registers. Data registers, direction registers, pullup/pull-down registers, interrupt mask registers and interrupt registers allow the system master to program and configure 4 or 8 or 16-GPIOs using a standard 400kHz I2C interface.

Figure 4: IO Expander


80560NT11734A Rev. 0 of 65 2019-01-22

4.1.2. Software The Telit HD Full-Duplex Audio Video ADK uses the following software components.

Table 3: Software Components


GS Video Embedded Firmware Application

An embedded application that runs on the Telit Video ADK board which is the source of the video stream and runs the RTSP server and RTP.

A wrapper application for Android Discovers and launches appropriate player that is installed on the system.

Video Mobile Application for Android Smartphones

An RTP and RTSP based video player for playing live video stream.

OV798 firmware binary The OV configurations and functionalities come in a packaged binary with GS SDK/AEK.

NOTE: OV source can be shared for customization requiring NDA with OV.


80560NT11734A Rev. 0 of 65 2019-01-22

4.2. TASKS

Figure 5: Task Interface Architecture

• The main task is divided into two sections: o Base code, and o Applet.

• Base code performs all basic system initialization. • Applet performs the following tasks:

o Network initialization. o Start Video encoder thread. o Register for call backs and encoder events. o Start RTSP thread and register call back for RTSP events. o Start thread for video and audio streaming

• SPI configurations are configured with GS node as SPI master and OV798 as SPI slave. • GS SPI is configured as SPI Master, Mode 0, and DMA mode. • OV binary is loaded from GS flash to OV chip. • RTSP server is created in RTSP task and poll for events DESCRIBE, SETUP, OPTIONS,

PLAY and TEARDOWN. • As soon as PLAY command is received, RTP task is executed and starts requesting for video

and audio data from OV on SPI. • RTSP Client task establishes the UDP connection with the server waits for the data and if data

is received, it decodes and plays the audio. • RTSP task can receive TEARDOWN (ASYNC) event anytime while video streaming is going

on in RTP task. If the TEARDOWN receives all the three tasks (forward audio, video, and reverse audio) get close.


80560NT11734A Rev. 0 of 65 2019-01-22

4.3. PROCESS FLOW

Figure 6: Process Flow Diagram


80560NT11734A Rev. 0 of 65 2019-01-22

4.4. BUTTON AND ITS FUNCTIONALITIES The following table details the button functionality with respect to the pin used:

Table 4: GS2200M Button and Related Pins

Button Pin Name Description

Reset Button EXT_RTC_RESET_N This button resets the GS2200M module. It remains unused during standby or hibernate mode.

WPS/Snapshot Button IO-Expander GPIO[4] This button is is tied to the IO-Expander GPIO4. This will trigger the interrupt on GS RTC IO DC_DC_CNTRL_N/RTC_IO_4 and can be used to wake up the system from standby or hibernate mode. It is available always.

Restore Button GPIO25 This is a GPIO in the core domain which is only available when CPU is running. It is used by the BOOTROM for the Firmware Restore Functionality. When the BOOTROM is executing the pin is checked in. Otherwise, pushing this pin has no effect.

During normal resets, the pin is likely to cause undesirable effects. Therefore, it is highly recommended not to overload this GPIO. However, due to limited availability of GPIOs on our ADK Hardware, we are overloading it in RUN mode after the initial boot.


80560NT11734A Rev. 0 of 65 2019-01-22

The buttons and their functionalities are explained in the following table:

Table 5: Button Functionalities

Button Functionality Description

Remarks 5.3.0 (Video) 5.5.0 Low Power (Video)

Provisioning OTAFU Button

In AP mode, always ON and available all the times.

Push the WPS snapshot button for 5 secs to toggle between RUN(Station) mode and provisioning/OTA-FU(AP) mode.

NA

Reset Button Push the reset button. Push the WPS/Snapshot button followed by the RESET button.

In case of always ON, the push of WPS/Snapshot button is optional.

Factory Default Button

Push the WPS/Snapshot button.

Push the WPS/Snapshot button for 10 sec.

Note: The dynamic file system is erased to restore the board to factory default settings.

Firmware Restore Button

Push the restore button along with the reset button.

Push the WPS/Snapshot button followed by restore plus the reset button.

In case of always ON, the push of WPS/Snapshot button is optional.


80560NT11734A Rev. 0 of 65 2019-01-22

4.5. TASK INTERFACE ARCHITECTURE FOR APPLET–1

Figure 7: Task Structure – Applet-1


80560NT11734A Rev. 0 of 65 2019-01-22

The above Use Cases are explained as follows: Use Case 1: Always ON (Line powered supply)

This mode is used when there is line power supply available. The module is always ready to take a client request and comes up instantly. It loads the OV binary, starts RTSP server and waits for the user request.

After teardown, it remains active searching for a new user request.

Use Case 2: Wake ON Wireless

This mode operates on battery. The module goes to Deep Sleep or Standby mode depending upon the following settings:

• Standby mode - Delay of greater than 500ms • Deep Sleep mode - Delay of less than 500ms

In Wake on Wireless mode, on the first boot up, the module waits for 15 secs (configurable) for a user request and then goes to Deep Sleep/Standby mode depending on the beacon interval selected.

On teardown, it immediately goes back to Deep Sleep without any delay.

Functionality:

This mode is also known as ‘PS POLL Applet mode’ where GS node will be constantly in low power associated state waiting for Asynchronous events from the network. Low Power is achieved by going into standby between the configured beacon interval. The beacon interval is customizable and has a default wakeup on five beacons. The GS node wakes up just after five beacons. Only unicast data will be buffered on the AP side which will be received by GS node after five beacons by using the PS poll mechanism. Note that if any broadcast data comes during this time, it will be missed.

Operation:

• Initializes clock and gets boot reason. • Initializes basic hardware, soft timer, GPIO, alarm, UART, and persistent timer. • Initializes the file system, loads default configuration from flash file system, initialize OV

and start RTSP server along with PLAY request timeout timer, if it is first time boot. • On expiry of PLAY request timer or Receive TEARDOWN, GS node goes back to standby. • After Wakeup from standby, it initializes the lower system modules such as exception

handler, HI layer, and sys ctl. • Releases the reset of the WLAN and sends boot-up request informing WLAN that the APP

CPU is operating in PS POLL Applet Mode. • Registers for various events such as DAB Ready (Data After Beacon), DAB Done with

Wireless Device Driver (WDD). • Indicates WLAN to wake up on configured beacon interval.

• The WLAN wakes up and checks the boot-up message, in case the App CPU is functioning in PS poll Applet Mode, it will load a small footprint binary capable of handling only a subset of events (DAB Ready, DAB Done). It will check the beacon for any data available and accordingly send the event to the APP CPU.

• APP CPU will take the following action: If data is present:

• It receives a DabReady event from WLAN CPU.


80560NT11734A Rev. 0 of 65 2019-01-22

• It will load the Applet code dynamically which does the full initialization of the system, starts NCM, initializes the network interface, and is ready to receive data packets.

• It calls and sends an Interface Resume event to WLAN. • If the data packet is a RTSP or mDNS packet, it loads the OV and start RTSP Server. • The RTSP Server waits for PLAY request from client. • In case, if APP CPU receives TEARDOWN after play or if no request is received within 30

seconds (configurable time) it will stop NCM, save the parameters in RTC, and put the system in standby.

If data is not present:

• APP CPU receives a DabDone event from WLAN CPU. • APP CPU on receiving this event will put the system into standby.

Use Case 3: Event based wakeup (Alarm)

In this Use Case, the module goes on hibernate mode if no request is received.

Functionality:

On first boot, the module loads everything and waits for 15 secs (configurable time) for a user request. If no request is received, after loading the OV binary, it goes to Standby/ Deep Sleep mode based on the beacon configuration for the next 30 secs and still waits for a user request.

After these 30 secs are completed, it goes into hibernate mode. The Switch SW1 (snapshot/WPS button) is configured as an alarm trigger before it goes to hibernate.

When SW1 is pressed, it wakes up and goes to Deep Sleep/Standby based on beacon configuration for 30 secs. In case, if the user starts streaming, it loads the OV binary and starts streaming. Once the stream is stopped, it immediately goes back to hibernate mode without a delay.

Operation:

If macro APP_HIBERNATE_ENABLE is enabled, GS node works in Hibernate mode. ‘Basic code’ also called as base application is started which performs the following:

• Initializes clock and receives boot reason. • Initializes basic hardware, soft timer, GPIO, alarm, and UART. • Initializes file system and loads default configuration from flash file system, initialize OV

and start RTSP server along with PLAY request timeout timer, if it is first time boot. • On expire of PLAY request timer or receive teardown, GS node goes back to hibernate

mode. • If wakeup is from hibernate boot, it restores the hibernate data from LATCH RAM. • Powers up the WLAN Core. • Initializes the lower system modules such as exception handler, HI layer, sys ctl, and

configures the clock source. • Posts a message to the Applet loader thread to load Applet.

Applet is dynamically loaded by the Video LP base application which performs the following:

• Releases WLAN reset so that it gets initiated. • Initializes the supplicant and random number generator. • Initializes the remaining system (wdd msg handler, network stack, BSD, SM, and NCM).


80560NT11734A Rev. 0 of 65 2019-01-22

• Initializes the interface, BSD, state machine (SM), and Network Connection Manager (NCM).

• Starts NCM to connect to the predefined SSID. If Hibernate mode, then it will take the scan entries and goes ahead with the Association. Also, it will be a lean DHCP where it will request for the last saved IP address.

• Power saving is achieved by switching to PSPOLL mode/Deep Sleep mode in WAITING_PERIOD. In case if play request is received, it loads OV and starts RTSP Server stream Video data to client.

• On timer expire or receiving teardown goes to hibernate mode. • Saves the scan entry, IP address, PSK in RTC latch memory and goes to hibernate mode

and is awakens only by the RTC alarm pins.


80560NT11734A Rev. 0 of 65 2019-01-22

4.6. TASK INTERFACE ARCHITECTURE FOR APPLET–3 The Applet-3 is for verified provisioning. The process is as described below:

Entry Function Applet 3 – Thread Entry

{ Initiate system (File System, Wi-Fi, HI supplicant) Give Boot reverse to WLAN Initialize N/W stack, BSD layer, SM Initiate NCM and start NCM Event handling loop {

Wait ON message Process message

} Starts concurrent mode Handles provisioning events Connect to Access Points, if credentials are correct else stay in Limited AP }

Figure 8: Task Architecture for Applet- 3


80560NT11734A Rev. 0 of 65 2019-01-22

4.7. GS-OV INTERFACE

4.7.1. Load OV Binary

Figure 9: Loading OV Binary

• The OV binary is programmed in external flash*. • On boot up, after GS initialization, the OV binary is loaded into OV chip. • The OV binary has the default decoder configurations for fetching OV video. • OV Binary is loaded in four parts:

o First OV bootloader gets loaded with clock configuration 1 MHz o Followed by double data rate(DDR), MPU binaries, and firmware which is loaded

and remains the same for data transfer.


80560NT11734A Rev. 0 of 65 2019-01-22

4.7.2. Read OV Video Data

Figure 10: Reading OV Video Data

• After GS node initialization and network initialization, the main task creates RTSP and RTP video tasks.

• Once RTSP task receives PLAY event, RTSP server state is set from idle to running state.

• RTP waits until the server status goes to running and initiates the video and audio streaming.

• Video streaming: o Start video encoder. o Get video data length from the encoder. o Check for the frame size. It should not be more than 65KB. o Get video data from OV chip:

Make GPIO A* high. Wait for GPIO B* high. Enable the SPI0 CS. Read data. Disable the SPI0 CS. Make GPIO A* low and wait for GPIO B* to be low.

o Stop video encoder.


80560NT11734A Rev. 0 of 65 2019-01-22

4.7.3. Read OV Audio Data

Figure 11: Reading OV Audio Data

• Along with the main task, RTSP, and RTP video tasks, creates RTP audio task. • Once RTSP task receives PLAY event, RTSP server state is set from idle to running

state. • RTP waits until the server status goes to running and initiates the audio streaming. • Audio streaming:

o Start audio encoder. o Get audio data length from the encoder. o Check for the frame size. It should not be more than 65KB. o Get audio data from OV chip:

Make GPIO A* high. Wait for GPIO B* high. Enable the SPI0 CS. Read data. Disable the SPI0 CS. Make GPIO A* low and wait for GPIO B* to be low.

o Stop audio encoder


80560NT11734A Rev. 0 of 65 2019-01-22

4.7.4. Write Audio Data

• Along with the RTP video and audio tasks, creates RTSP client task. • It creates UDP socket and listens on audio port. • When the user selects the call option from the native app, GS module receives the

audio data and decodes the received data using g711 decoder. • Sends data to OV:

o Start audio decoder. o Send audio data length to the OV. o Send audio data to OV chip:

Make GPIO A* high. Wait for GPIO B* high. Enable the SPI0 CS. Write data. Disable the SPI0 CS. Make GPIO A* low and wait for GPIO B* to be low.

o Stop audio decoder.

Figure 12: Video ADK - Forward Audio

Figure 13: Video ADK - Reverse Audio

NOTE:

GPIO A, GPIO B and other GPIO configurations are explained in section 5.3 Configurations.

16-Bit Linear PCM, 8KHz

ZL OV798 GS

Convert to 8-Bit, 8KHz, G.711 µ-Law

16-Bit Linear PCM, 8KHz

ZL OV798 GS

Gets 8-Bit, 8KHz, G.711 µ-Law And converts to 16-Bit, 8KHz, Linear PCM


80560NT11734A Rev. 0 of 65 2019-01-22

Chapter 5. Embedded Application Execution Sequence

The Video Low power application by default comes in PSPOLL mode. The PSPOLL boot sequence is as follows:

1. Program OV binary in GS

a) Upload OV binary to GS Module

2. Video application execution

a) After GS first Boot up

b) Video_lp binary checks for Boot type as PSPOLL or Hibernate

1. If PSPOLL Boot, restore the standby data and check DAB Ready indication. 2. If DAB Ready indication notified loads applet (applet_1.bin) otherwise, goes to

standby

After applet loads and do default configuration initialization. The sequence is as follows:

c) Load OV from GS flash via SPI at 10MHz

a. Load Boot Loader

b. Load OV binary

d) Start RTSP Server (and RTP Task) and play time out timer

e) RTSP Server waits for PLAY method till timer expires. On timer expires it goes to standby

f) Discover the video service in video application or launch via VLC

g) RTSP message exchanges

h) On receive of “PLAY”, sets the server status as running and initiate video and audio streaming

i) RTSP thread establishes UDP connection, connects to UDP server on the app and waits for the data. When the call button is pressed in the native app, UDP server in the smartphone sends data and GS module receives the audio stream and plays.

j) Terminate streaming (RTSP Server keeps running, TEARDOWN is sent when native APP is closed).

k) On receiving TEARDOWN, it goes back to standby.

In case Video Low power application compiled with APP_HIBERNATE_ENABLE flag application boot up in Hibernate mode. Sequence of Hibernate is as follows:

1. Program OV binary in GS

a) Upload OV binary to GS Module

2. Video application execution

a) GS node wakeup by pressing Alarm 3 button. On wakeup Video_lp binary loads into RAM.

b) Video_lp binary checks for Boot type. As code compile with APP_HIBERNATE_ENABLE it boots up in Hibernate mode and Boot type set to HIBERNATE.

1. Video_lp binary restore the Hibernate data. 2. Loads applet (applet_1.bin).

After applet loads and do default configuration initialization. The sequence is as follows:


80560NT11734A Rev. 0 of 65 2019-01-22

a) After NCM starts, loads OV binary.

b) After send data waits for user response If user request to stream.

c) Start RTSP Server (and RTP Task).

d) Discover the Video service in Video Application or launch via VLC

e) RTSP message exchanges

f) On receive of “PLAY”, sets the server status as running and initiate video and audio streaming.

g) RTSP thread establishes UDP connection, connects to UDP server on the app and waits for the data. When the call button is pressed in the native app, UDP server in the smartphone sends data and GS module receives the audio stream and plays.

h) Terminate streaming (RTSP Server keeps running, TEARDOWN is sent when native APP is closed).

i) On receiving TEARDOWN, it goes back to Hibernate.


80560NT11734A Rev. 0 of 65 2019-01-22

5.1. PROGRAM OV BINARY IN GS NODE The OV binary is loaded into GS module via three methods:

5.1.1. Method 1: Upload OV Binary to GS Module

Perform the following steps to do an over the air update for OV binary into the external flash of GS module:

NOTE: OV binary are provided as part of the release package from Telit which can be loaded directly via S2W into external flash.

• Over the air firmware update for OV can be done only after once OV binary with required superblocks and control blocks have been programmed in external flash (It comes as a part of Quick start guide and can be done via S2w application).

• OV firmware consists of dsif_sl.bin (slave image) and dsif_bt.bin (bootloader image), which are combined to make a 1MB (1024KB) file system for programming into external flash. This can be directly programmed to external flash via s2w application.

• For performing OTAFU for OV image 4KB control block need to be appended at the start of image followed by the OV image. The size of image compatible for OTAFU comes to 1028KB. The steps and scripts to create the same has been shared at,

../SDK/embedded/tools/GS_Extflash_FS_Create_tool folder

1. Generate the OV binary as explained above and copy into a Laptop or a PC.

2. Switch on the Telit Video ADK board. It comes up in limited AP mode with SSID as GS_PROV_VIDEO_XXXXXX, and passphrase is “GS_XXXXXX” where XXXXXX are the last six digits of the MAC address of GS module.

3. Connect the laptop or PC with the Telit Video ADK board through Wi-Fi.

4. Access the GS module URL http://192.168.240.1/extotafu.html from Laptop or PC.

5. Browse the valid firmware image (ov_av_otafu.bin) and click upload.

Figure 14: Firmware uploads Screen

6. GS module restarts itself.


80560NT11734A Rev. 0 of 65 2019-01-22

5.1.2. Method 2: Load OV Binary into External Flash via S2W

The package consists of a single image which is built from OV bootloader, and an OV firmware image which can be directly loaded to serial flash via S2W. When the module boots up, it loads the OV binary from serial flash to OV chip.

5.1.3. Method 3: Load OV Binary Directly to OV Chip via UART Interface

1. Open the Tera Term terminal emulation application. Select the serial port for OV UART USB and set the Baud rate to 115200.

Figure 15: Serial Port Selection

2. Press SHIFT+X keys simultaneously and a continuous stream of CCCCCC is displayed. This indicates the OV binary is in Program mode.

Figure 16: OV Binary in Program Mode

3. From the Tera Term window, select File>Transfer >XMODEM > Send.


80560NT11734A Rev. 0 of 65 2019-01-22

Figure 17: Transfer Window

4. Open the folder where the OV binary is located. Select the ov_boot.bin file. Click Open.

Figure 18: OV Binary Location

5. ov_boot binary starts downloading. Once complete, the Tera Term window displays the version information of video ADK/AEK OV binary and automatically places the OV chip in Run mode.


80560NT11734A Rev. 0 of 65 2019-01-22

Figure 19: OV Binary in RUN Mode

5.2. VIDEO APPLICATION EXECUTION The sequence of execution after the GS module restarts is explained in the following sections.

5.2.1 GS Boot up The sequence of execution followed for loading boot loader and OV firmware is as follows:

1. On boot up, the GS node starts up with system initialization.

Figure 20: GS UART Print after GS Module Restart

At this point, the main task completes system initialization.

Code Ref:

AppMainCtx_ThreadEntry()

{


80560NT11734A Rev. 0 of 65 2019-01-22

---//All system initializations are done here-

---

}

5.2.2. Load OV from External Flash to OV Chip via SPI First boot case applet loads without any notification which starts loading OV from external flash by calling App_OVInit () that initializes OV.

Code Ref:

AppOV_Init(){

---//All system initializations are done here-

---

AppVideo_OVBootupFirmwareTransfer() // This functions initializes SPI for loading bootloader and firmware to OV chip.

}

App_OVInit() contains AppGpioConfig_OVInit() which initializes I/O expander in AppGpioConfig_Initialize(). This also creates I2C interface between GS and OV that helps us to reset OV.

After first boot, the module goes to standby and wakeup, loads the applet by getting notification from DAB Ready Indication call back. App_NwRecvFilter finds the packet type. If it is mDNS or RTSP packet it pushes OV initialize message to App main Task message queue which is then processed by AppSm_MsgProcess function.

OV Load PSPOLL Boot:

BOOL

App_NwRecvFilter (GSN_NWIF_CTX_T *pNwIf, UINT32 pktLen,

GSN_ETHERNET_FRAME_T *pEthFrame)

{

--//Check for Packet type and push message to MainTask.

if((GSN_SYS_IS_PS_BOOT_NORMAL() == TRUE ) && ((dType == APP_RTSP_PKT)

|| (dType == APP_MDNS_PKT)) && (isOVInitDone == FALSE))

{

/*Initialise OV Once data request received */

AppMainCtx_TaskNotify(APP_EVENT_OV_START, 0);

}

PUBLIC GSN_STATUS

AppSm_MsgProcess (APP_MAIN_CTX_T *pAppCtx, APP_MAINTASK_MSG_T msg )

{


80560NT11734A Rev. 0 of 65 2019-01-22

case APP_EVENT_OV_START :

{

/* Flash OV binary to OV chip*/

App_OVInit();

}

break;

}

Telit module performs the following steps to load OV firmware from external flash to OV chip through SPI interface:

1. Initially, the Wi-Fi module checks the actual length of the OV bootloader and provides the clock configuration with speed of 1 MHz

NOTE: Telit SPI interface is the master and OV chip is the SPI slave.

2. The module sends the header configuration parameters to the Video chip.

3. Once OV receives header successfully, it sends ACK(0x48454144) to the module.

4. Next, the module sends bootloader to OV through the SPI interface.

5. Once OV receives the bootloader successfully, OV send an ACK(0x46574F4B) to the module.

6. The Wi-Fi module changes the SPI clock to 10MHz and sends double data rate(DDR), MPU, and firmware binaries.

NOTE: Before sending the above binaries, the module must send 4 bytes of header which informs the Video chip about the data flow.

5.2.3. Start RTSP Server (and RTP Task)

2. At this point of time, OV binary is loaded to the OV chip, and the main task is completed by creating RTP and RTSP tasks. Code Ref: AppRtsp_Init(){

--- //All system initializations are done here-

---

#ifdef RTSP_SERVER_EN

//It acts like interface between App and VideoEnc chip

// that performs start,stop the encoder and do encode audio.video frames


80560NT11734A Rev. 0 of 65 2019-01-22

AppVideoEnc_ThreadStart();

// Register application call back function to

// Notifies when encoded frame is Ready

AppVideoEnc_EventCbReg(AppRemainInit_VideoEncEventNotify);

//Create RTSP Server socket and handles the RTSP methods

RTSPThreadStart();

AppVideoThreadSendStart();

AppAudioThreadSendStart();

//Register application call back function to get Notify when

//event occur in RTSP framework

GsnRtsp_EventNotifyCbReg(AppVideo_RtspCb);

#endif

NOTE:

• RTSP_SERVER_EN - Macro to enable the RTSP session

• RTSP_AUD_EN - Macro to enable forward Audio

• AUDIO_ONLY - Macro to enable audio only session

• RTSP_CLIENT_EN - Macro to enable reverse audio

3. GS module starts the RTSP and RTP services after the boot loader and OV firmware are loaded on to the video chip. When RTSP receives the PLAY request, sets the status to running. RTP waits until the server state goes to running, reads the video and audio data over SPI interface and send the video and audio to the smartphone over the air. At the same time RTSP client task establish UDP connection waiting for audio data from smart phone. Once the user press call back button on Native app audio data starts streaming from smart phone to GS module.

NOTE: RTSP uses the standard port number 554.

5.2.4. RTSP Message Exchanges RTSP task is handled in the function call RTSPThreadStart() and the detailed description is as follows.

For more details about the code flow, refer section A.1.1 RTSP SDK Reference Code Flow.


80560NT11734A Rev. 0 of 65 2019-01-22

The RTSP communication between GS module and smartphone is as shown in the following figure.

Figure 21: RTSP Communication between GS Module and Smartphone

The RTSP server which is part of the embedded firmware supports DESCRIBE SETUP, OPTIONS, PLAY and TEARDOWN methods. The sample RTSP packets are as shown in the following figure:

Figure 22: Sample RTSP Packets


80560NT11734A Rev. 0 of 65 2019-01-22

RTSP Messages Details

In the following description, VLC is the client and GS RTSP server is the Server.

OPTIONS

Client Server

Client sends the connection request with OPTIONS.

Server Client

Server replies stating all features it supports.

DESCRIBE

Client Server

The client requests for SDP details.

Server Client

The server replies with its SDP configurations


80560NT11734A Rev. 0 of 65 2019-01-22

SETUP

Client Server

The client requests the server to start the session.

Server Client

The server replies stating it has established the session.

PLAY

Client Server

The client requests to start streaming.

PLAY rtsp://192.168.240.1/live.h264 RTSP/1.0 CSeq: 6 User-Agent: LibVLC/2.2.0-git (LIVE555 Streaming Media v2014.01.21) Session: 12321 Range: npt=0.000-

Server Client

Server replies stating that the session has been established and streaming should be started on particular URL.

RTSP/1.0 200 OK CSeq: 6 Server: RtpRtspServer/1.0 RTP-Info: url=rtsp://192.168.240.1/live.h264;seq=1;rtptime=0 Content-Length: 0 Cache-Control: no-cache Session: 12321


80560NT11734A Rev. 0 of 65 2019-01-22

TEARDOWN

Client Server

Client sends Teardown (When APP /VLC closes) to stop streaming.

TEARDOWN rtsp://192.168.240.1/live.h264 RTSP/1.0 CSeq: 7 User-Agent: LibVLC/2.2.0-git (LIVE555 Streaming Media v2014.01.21) Session: 12321

Server responds with 200 OK.

RTSP/1.0 200 OK CSeq: 7 Server: RtpRtspServer/1.0 Session: 12321


80560NT11734A Rev. 0 of 65 2019-01-22

5.2.5. Start Forward Audio and Video Streaming Once the connection is established between RTSP Server and Client, RTSP Server receives the PLAY request from client that change the Server status to running. RTP waits until the server status goes to running and gets the data over SPI interface. GS module reads the audio and video data from OV chip in respective RTP audio & RTP video task and sends the data over air on RTP /UDP. The code flow has been briefed in Appendix section and the steps have been described in the further subsection.

5.2.5.1. Start Audio & Video Encoder As the RTSP play command is processed RTP streaming request is received by RTP audio and video tasks.

The following function starts Video encoder in RTP video task.

VENC_Status (VE_Start); // start OV encoder

The following function starts Video encoder in RTP audio task.

AENC_Status (AE_Start): // start Microsemi audio encoder interfaced with OV

5.2.5.2. Get Audio & Video Data Once encoders have been started GS module sends request to get the Audio and video data from the OV chip. The following function is used to get the video data in RTP video task.

Get_VStream (&streamPtr, &stream Len, &pkt_type);//The function in turn further requests for length of data before fetching the actual data.

Get_Vinfo(data_len);// Apart from data length Get_Vinfo function will give timestamp of frame, frame number, frame type information. Check for the frame length is valid or not. One frame size shouldn't more than 65Kbytes.

Get_VData(unsigned char **data_ptr, unsigned int *data_len):// Once valid video data length is known , request the video data with data_len size.

The following function is used to get the video data in RTP audio task.

Get_AStream (&streamPtr, &stream Len, &pkt_type);//The function in turn further requests for length of data before fetching the actual data.

Get_Ainfo(data_len);//

Get_AData(unsigned char **data_ptr, unsigned int *data_len):// Once audio data length is known , request the video data with data_len size.


80560NT11734A Rev. 0 of 65 2019-01-22

5.2.5.3. Fragment the Video Data Fragmentation is a mechanism that allows fragmenting the video data. This is performed by fragmenting a single Network Abstraction Layer (NAL) unit over multiple RTP packets. Each fragmented frame is known as a Fragmentation Unit (FU).

GS module verifies if NAL’s value is 0001 and checks the header to determine the type of actual data and continues to check for the NAL separator after every 16 bytes in the actual data.

• If there is a NAL separator after every 16 bytes and the actual data exceeds 1400 bytes, fragmentation mechanism is applied and the FU which consists of 12 bytes of data that includes 2 bytes of header and actual data are fetched and streamed.

• If there is no NAL separator after the 16th byte, fragmentation mechanism is not applied and 12 bytes data which includes 2 bytes of header and 10 bytes of actual data is fetched and streamed.

Figure 23: Fields in Actual Data

NOTE: • Fragmentation mechanism is applied when the size of the actual data exceeds

1400 bytes. • Fragmentation mechanism is not applied when:

o Size of the actual data is less than 1400 bytes or o The type of data from the video chip is Sequence Parameter Set (SPS)

or Picture Parameter Set (PPS) Frame size of audio data is less than maximum packet size so no need of fragment the audio data.


80560NT11734A Rev. 0 of 65 2019-01-22

5.2.5.4. Add FU Header for Video data A Fragmentation Unit (FU) consists of a FU Indicator and a FU Header as shown in the following figure.

Figure 24: Fragmentation Unit

Byte 0 is the FU Header where,

• F is the forbidden zero bit which is set to 0

• NRI is the nal_ref_idc which is 0x01. It is taken from second and third bits of NAL Header (first byte after NAL separator)

• Type indicates the NAL unit type which is FU-A/28

Byte 1 is the FU Indicator where,

• S indicates Start of the fragmented unit

• E indicates end of a fragmented unit

• R is the reserved bit which is set to 0 and ignored by the receiver

• Type is the NAL unit payload type. It is taken from the next byte after NAL separator

5.2.5.5. Add RTP Header RTP defines a standardized packet format for delivering audio and video over IP networks. It is a UDP based streaming protocol which is designed for end-to-end real-time transfer of stream data. GS module uses RTP and RTSP to fetch data from the video chip and stream it on the Smartphone.

Gain Span module fetches audio & video data from the OV chip through SPI interface in LPCM and h.264 format respectively and encapsulates it in RTP packet format. On GS request call OV fetches audio data from Microsemi audio encoder and sends it to GS module.

The application on the smartphone gets these RTP packets, decodes it, and gives it to the decoder application to parse and display the video and audio, the Payload type field carries information on data type and format.


80560NT11734A Rev. 0 of 65 2019-01-22

The RTP packet header is of 12 bytes. The fields in the header are as follows:

Figure 25: RTP Packet Header

The following table lists and describes the fields in an RTP packet header.

Table 6: Description of Fields in RTP Packet

Field Description

Version It is of 2 bits and indicates the version of the protocol.

P (Padding) It is of 1 bit and indicates if there are extra padding bytes at the end of the RTP packet.

X (Extension) It is of 1 bit and indicates presence of an Extension header between standard header and payload data.

CC (CSRC Count) It is of 4 bits and contains the number of CSRC identifiers (defined below) that follow the fixed header.

M (Marker) It is of 1 bit and it is used at the application level and defined by a profile. If it is set, it means that the current data has special relevance for the application.

PT (Payload Type) It is of 7 bits and indicates the format of the payload and determines its interpretation by the application. This is specified by an RTP profile.

Sequence Number It is of 16 bits and it is incremented by one for each RTP data packet sent. It is used by the receiver to detect packet loss and to restore packet sequence.

Timestamp It is of 32 bits and it is used to enable the receiver to play back the received samples at appropriate intervals.

When several media streams are present, the timestamps are independent in each stream, and may not be relied upon for media synchronization.

SSRC It is a 32 bits synchronization source identifier which uniquely identifies the source of a stream.

CSRC It is a 32 bits contributing source IDs which enumerates contributing sources to a stream which has been generated from multiple sources.


80560NT11734A Rev. 0 of 65 2019-01-22

5.2.5.6. Send Data over UDP After adding the RTP header to audio and video data, packets are sent over the air on RTP via UDP. The sample RTP packets are captured below.

Figure 26: Sample RTP Packets

RTP Packet Description:

5.2.5.7. Terminating Audio and Video Stream Once smartphone closes the application, the application sends a TEARDOWN RTSP message to the RTSP server (GS module) to stop the video stream. The request to stop streaming is acknowledged by the GS module. The sample TEARDOWN message communication is as follows:

Figure 27: Sample TEARDOWN Message

5.2.6. Reverse Audio Streaming Reverse audio is feature which enables talk back i.e., along with forward audio RTSP Client captures audio packets from Smartphone App, sends to GS module and GS module play it back. As GS modules sends and receives audio data it’s a full-duplex audio streaming application. Forward audio video streaming must be started to initiate reverse audio streaming.

The reverse audio streaming is handled in RTSP client task. Once RTSP Server status changes to STREAM_RUNNING i.e., PLAY request success. Reverse Audio Process opens a UDP socket


80560NT11734A Rev. 0 of 65 2019-01-22

on port 2049. Once socket connection establishes it waits for audio data from the native app (mobile app). GS Module receives Audio packets from app will be send to OV chip.

5.2.6.1. Start Audio Decoder The following function starts the audio decoder in RTSP client task.

ADEC_Status(AD_Start);// starts audio decoder.

5.2.6.2. Convert G711 to LPCM audio format Smartphone application sends audio data in G711 format. GS modules receives audio packets and converts the audio data into Linear PCM with the following function call.

Gsn_g711decode ((INT8 *)&client.inbuff[0], (INT16 *)(&(g711DecodePtr[dataRcvd])), G711_ALAW, a_len);

5.2.6.3. Send audio data to OV for playback Once the data is received by GS module it needs to be sent to OV to play it on the decoder. OV needs to know the length of data before sending the actual audio data. It can be achieved by the following functions.

Send_Ainfo (data_len); // Sends the data length before sending data.

Send_AData (data_ptr, data_len); // Sends the audio data.

5.2.6.4. Disable Reverse Audio: If user disables the talk-back (i.e., without stopping the Forward streaming), Connection will not be closed, and process will wait for the data. This procedure will interrupt by either ENABLING talk-back or TEARDOWN received by server.

5.2.7. Video Rate Adaptation using Remote RTCP Information and Local Data

Video quality is obtained based on bitrate, framerate and resolution. Different profiles are used to maintain the video quality based on the information of the packets lost from the remote (RTCP) and the local transmit throughput data. Profiles includes fixed bitrate, framerate and resolution which ensures video quality. Profile 1 has the very high video quality with 1Mbps bitrate, 30Fps framerate and 1080p resolution.

Profiles and its corresponding rates are as follows:

Table 7: Profile and its Corresponding Rates

Profile IDs Bitrate Framerate Resolution

1 1Mbps 30Fps 1080p

2 1Mbps 15Fps 1080p

3 512Kbps 30Fps 1080p

4 512Kbps 15Fps 1080p


80560NT11734A Rev. 0 of 65 2019-01-22

Video frame rate adaptation is calculated based on the fraction of packets lost during the last reporting interval since the pervious RR packet was sent, the Transmit data throughput and the percentage of packet sent in 5msec.

Based on the fraction lost and the transmit throughput, video quality is maintained by setting the appropriate profile. During the initialization process, Profile 1 – Very high video quality is being set and streaming will be started.

Based on the average of packets lost and bandwidth thresholds (based on the SIF bitrate) and data throughput video quality is compromised or maintained by selecting the profile and update the video configuration.

Exceptional is derived when the local data throughput is 100% - 95% and the packet lost is zero, where it is understood as the video can be streamed with better quality than the existing. If only the derived quality is exceptional, the current profile is updated to better configuration settings, in all other cases quality is compromised based on the table shown below.

Throughput

(in %)

Packet Sent RR Packet

(% Lost)

Derived Quality

100-98% 100% - 95% 0% Exceptional

100% - 95% > 95% 0% - 1% Very high

1% - 2% High

2% - 3% Medium

3% - 4% Low

> 4% Very Low

95% - 80% 0% - 1% High

1% - 2% Medium

2% - 3% Low

3% - 4% Very Low

< 80% 0% - 2% Medium

2% - 4% Low

> 4% Very Low

Following are the steps to select the profile for better video quality:

1) Determine the video quality.


80560NT11734A Rev. 0 of 65 2019-01-22

a. To determine the video quality, the source of inputs are the fraction lost from the RR packet, the transmit data throughput and the packet sent in 5msec.

2) Obtain the range of packet loss from 1) the fraction lost in the RR packet, 2) the transmit throughput and 3) Packet sent percentage in 5msec.

3) Based on the range of packet lost quality is derived. With respect to the derived quality action is taken. The below table shows the corresponding action for the derived quality.

Table 8: Profile and its Quality

Profile (Current) Quality (Derived) Action

1 Exceptional No Action

2 – 5 Exceptional New Profile ID =

Current Profile ID - 1

Any Very High No Action

High No Action

Medium New Profile ID =

Current Profile ID + 1

Low New Profile ID =


Very Low New Profile ID =


Current Profile ID - 1, Increases the current profile to next good video quality profile

Current Profile ID + 1, Decreases the current profile to next lower video quality profile

4) Update the configuration with the new profile ID. 5) If No RR packet has been received in 12 sec time, the current profile is set to next lower

video quality profile. Derivation of quality:

1) Video is streaming with profile 1 settings and If the fraction lost is 0%, Packet sent in 5msec is 100% and data transmit throughput is 100-95%

• No action will be taken as the video is already running in very high quality. • Instead of profile 1, if the video is streaming with profile 3 configuration, the video

configuration is set to profile 2. 2) Video is streaming with profile 2 settings and if the fraction lost is between 0.5% to 1% and

throughput is greater than 95%, there will be no change in the settings. • If the fraction lost is between 1% to 1.5% then the derived quality is medium, and

the video settings is set to profile 3 and so on.


80560NT11734A Rev. 0 of 65 2019-01-22

• If the throughput is less than 95% only fraction lost is taken to determine the quality of the video.

• If the throughput ranges 95% - 80% or less than 80% only the fraction lost is taken to determine the quality of the video.


80560NT11734A Rev. 0 of 65 2019-01-22

5.3. CONFIGURATIONS Control and data paths between GS2200M and OV798 are configured using SPI and signaling configurations. The graphical representation of the pin configurations is provided in the following figure.

Figure 28: SPI and Signaling Configurations

List of GPIO pins used in SDK with their macros:

For GS2011 modules

#define GSN_OV_RESET GPIO 26

#define GSN_OV798_GS_TO_OV_INT GPIO 19 //GPIO A

#define GSN_OV798_OV_TO_GS_INT GPIO 20 // GPIO B

#define GSN_OV798_GPIO_SPI0_CS GPIO 33

#define GSN_OV798_GPIO_SPI0_DIN GPIO 34

#define GSN_OV798_GPIO_SPI0_CLK GPIO 35

#define GSN_OV798_GPIO_SPI0_DOUT GPIO 36

For GS2200 modules

#define GSN_OV798_GS_TO_OV_INT GPIO 3 //GPIO A

#define GSN_OV798_OV_TO_GS_INT GPIO 26 // GPIO B

#define GSN_OV798_GPIO_SPI0_CS GPIO 33

#define GSN_OV798_GPIO_SPI0_DIN GPIO 34

#define GSN_OV798_GPIO_SPI0_CLK GPIO 35

#define GSN_OV798_GPIO_SPI0_DOUT GPIO 36


80560NT11734A Rev. 0 of 65 2019-01-22

5.3.1. Remote Configuration of Video Encoder Settings 5.3.1.1. In band – During the start of video session establishment through

RTSP URL Using rtsp://[ ip_address/server_URL[?param1=val1[&param2=val2]...[&paramN=valN]]

Example:

Play "rtsp://<ip>/live.h264?btr=512&fmr=15&flip=false

Parameters and values allowed are,

res=1080

btr=1024/ 512

fmr=30/ 15

flip=true/ false

mirror=true/ false

Settings Params

bitrate btr

framerate fmr

resolution res

flip flip

mirror mirror


80560NT11734A Rev. 0 of 65 2019-01-22

5.3.1.2. Out of band – During run time (over HTTP) Send “http://<ip>/gainspan/profile/video/control” URL from Advance rest client with the body "<bit_rate>value</bit_rate><frame_rate>value</frame_rate><resolution>value</resolution><flip>value</flip><mirro>value</mirror>"the method as POST to set the video configuration.

Value Configuration

BitRate: 1 1024k

2 512k

FrameRate: 0 30fps

1 15fps

Resolution: 0 1080p

Flip: 0 Normal

1 flip

Mirror: 0 Normal

1 mirror


80560NT11734A Rev. 0 of 65 2019-01-22

Appendix A Reference Codes

A.1.1 RTSP SDK REFERENCE CODE FLOW RTSPThreadStart(); //Creates a thread for RTSP Server task from main task

rtsp_server_task(UINT32 arg) //RTSP Server task

rtsp_main(); // RTSP Server starts

create_sercmd_socket(rtsp[0]->host_name,port,SOCK_STREAM); //Creates socket for server

tcp_accept(); //Function does client connection request accept

vd_rtsp_proc ((void *)free_conn); //Reads data from the client and Parse the data

rtsp_cmd_match(S32 method, S32 cur_conn_num) //Handles the received RTSP method

rtsp_options(cur_conn_num) //Set options command response buffer and Sends success response

rtsp_describe(cur_conn_num) //Forms SDP message and will be sent to the client

rtsp_setup(cur_conn_num) //Parses the client ports for RTP/RTCP and Sends success response

rtsp_play(cur_conn_num) //Sets the server status as running by setting is_running in rtsp structure and will initialize the rtp by creating the socket with the extracted ports from the setup rtsp_teardown(cur_conn_num) //Used to terminate the session, Process the teardown request from the client and Sends success response

A.1.2 RTP VIDEO SDK REFERENCE CODE FLOW AppVideoThreadSendStart(); //Creates video encoder thread and mailbox to receive messages AppVideoSend_Task (UINT32 arg); //Receive messages, Handle events AppVideoSend_MsgProcess(msg); //Processes the video encoder task events GsnRtp_VideoDataSend (unsigned char *buffer, UINT16 bufLen, UINT8 pkt_type) //Sends video buffer to RTSP framework that create RTP packet and send to network send_frame_on_rtp_nw(buffer,bufLen, 0, marker,video_timestamp); //Sends data to the client

A.1.3 RTP AUDIO SDK REFERENCE CODE FLOW AppAudioThreadSendStart(); //Creates Audio encoder thread and mailbox to receive messages AppAudioSend_Task (UINT32 arg); //Receive messages, Handle events AppAudioSend_MsgProcess(msg); //Processes the audio encoder task events GsnRtp_AudioDataSend (unsigned char *buffer, UINT16 bufLen) //Sends audi buffer to RTSP framework that create RTP packet and send to network send_frame_on_rtp_nw(buffer, bufLen, 1, a_marker, audio_timestamp); //Sends daa to the client


80560NT11734A Rev. 0 of 65 2019-01-22

A.1.4 REVERSE AUDIO SDK REFERENCE CODE FLOW Rtspclient_init(); //Initialize reverse audio as rtsp client Handl_Get_Data(); //Handle the data received from server serv_socket(char *host, char *port); //Create socket and listen on audio port, Register call back to get notification on audio socket rtsp_revaud_cb (INT sockId, NX_BSD_CB_EVENT_TYPE_T event, void* cbCtx); //Handle the event received on rtsp client socket for reverse audio AppVideoEnc_ReverseAudioStart(); // Starts the audio decoder AppVideoEnc_AudioDataRecv(); // Get reverse audio from socketId AppVideoEnc_AudioDataWrite (unsigned char *ptr,INT32 a_len) //Gets encoded audio data and converts to G711 format then sends to OV to play

A.1.5 VIDEO ENCODER INTERFACE CODE FLOW //It acts like interface between App and VideoEnc chip //It performs start, stop the encoder and do encode video and audio frames AppVideoEnc_ThreadStart(); //Create video encoder interface thread and mailbox to receive messages AppVideoEnc_Task (UINT32 arg) //Runs in infinite loop in which messages on the queue are received and the requisite action taken AppRemainInit_VideoEncEventNotify(UINT8 event, VOID *frame,APP_BUFF_META_DATA_T * frameMetaData); //Notifies whenever audio/video buffer has data. Application will send the buffer to RTSP interface

A.1.6 SIF PROTOCOL All communications between OV and GS node are based on SIF (Standard Interchange Format) Protocol. The section below details the SIF command interface mechanism.

Steps involved in SIF Protocol:

1. Command stage initializations

SIF command is sent using SIF_Cmd_snd_rcv ((unsigned char *) &Tx_pkt, sizeof(Tx_pkt), SPI_WR); Brief description of sending and receiving SIF command are explained in separate.

2. Lock the resources with semaphore while using SIF commands.

3. Acknowledgement stage:

With the Read operation, SIF command is sent. If read is success, get the ACK status using the function Get_Ack_Status (&Rx_pkt); which tells ACK/NACK received and error details.

4. After sending the data to the client by framing as RTP packets, encoding should be stopped.

5. Command stage initializations are done here.


80560NT11734A Rev. 0 of 65 2019-01-22

Figure 29: Header Information

BUFFER max size is set to 10KB. SIF command is sent with write operation (SPI_WR), it sends the data in the buffer (Tx_pkt) to SPI slave.

Acknowledgement stage

SIF command is sent with operation read (SPI_RD). ACK/NACK and information is stored in Rx_pkt. Structure member payload gives the information of total available stream, data_len, timestamp, frame number, frame type. If the frame length is 0, it indicates that the encoder is not ready.

Figure 30: Payload

6. Command stage initializations are done here. SIF command is with write operation to SPI.

len = data_len + DATA_OFFSET

Figure 31: Get Data


80560NT11734A Rev. 0 of 65 2019-01-22

7. After aligning the given length in terms of OV alignment word length which is to be used in the length field in the SIF command. Acknowledgement stage read operation is sent. If ACK status is success, assigns the starting address of the video/audio frame to the requested data pointer.

Figure 32: ACK/NACK Stage

8. Get the frame length and SPI buffer size. If the frame length is less than the SPI buffer size, entire frame is marked as single packet (SINGLE_PKT). If the frame length is more than the SPI buffer size, first part of frame is marked as start packet (SINGLE_PKT), all middle part of frame is marked as inter packet (INTER_PKT), Last part of the frame is marked as end packet (END_PKT). It tells the packet type.

Sending and Receiving SIF Command

int SIF_Cmd_snd_rcv (unsigned char *buf_ptr, unsigned int buf_len, UINT8 operation) //buf_ptr: Depending on the operation, buffer will be used, buf_len: contains the length of the buffer, Operation: SPI read or write operation to be performed.

1. This function checks the operation. If the operation is write (SPI_WR) or read (SPI_RD), and also checks whether timing mode is TIMING_DOMINANT which is initialized in AVLB_Init().

2. Makes GPIO A high and waits for GPIO B to become high. Uses call back to get the details from OV to GS.

3. Enables the SPI0 CS.


80560NT11734A Rev. 0 of 65 2019-01-22

In case of SPI write,

1. Checks the timing mode,

2. Checks the max length size with DMA max size(1k). If size < DMA max size, then take full length. Otherwise reads/writes multiples of DMA size

3. Sends the data.

In case of SPI read, with ACK/NACK info,

1. Video data will be stored in the buffer. Members Ack_Nack and Ack_Cmd_Result give the status and reason of failure.

2. Disables the SPI0 CS.

3. Makes GPIO A low. Waits for GPIO B to low and Return.

gs2k ov720p hd full duplex audio video adk application note · fault-tolerant and are not designed,...

Documents