FEATURE

3

With the progress in digital broadcasting, products that make it possible to use internet services on television receiv-ers, often referred to as “connected TVs” or “smart TVs”, have appeared. Integrated Broadcast-Broadband (IBB) sys-tems are also being standardized, Hybridcast, which is one of the IBB systems, began in Japan in 2013, and many TVs can now be connected to the internet. Smartphones are now also being widely used as mobile devices for connecting to the internet to obtain information and watch videos as well as for personal communication. As a result, there have been many attempts at creating various types of services that in-volve the interworking of TV receivers, which feature the simultaneous delivery of information to large numbers of people, and smartphones, which are used as personal de-vices in everyday activities. Here, we describe “device link-ing technology” that achieves the interworking between TV receivers and smartphones.

1. IntroductionTelevision has long served as a focal point for family

activity, such as watching TV together in the living room. However, this situation is changing with increasing use of smartphones, tablets, and other mobile devices in our daily lives, and now there are many opportunities to spend time with friends who are not all in the same place using social network services (SNS) and other forms of interaction. We now see this tendency even while people are watching TV at home.

This lifestyle trend has been drawing attention since about 2010, and there is an expectation of new services and business opportunities that are based on the interworking of television, which is watched by a large number of people at the same time, and smartphone applications, which are used by individuals. Second-screen and multiscreen services have appeared. In such services, the TV serves as the first screen, and information relevant to what is being viewed on the TV receiver is presented on the screen of a smartphone as the second screen. For example, a commercial being pre-sented on the TV can be accompanied by a relevant discount

coupon presented on a smartphone.Smartphones have become even more important in our

lives and wireless broadband communication technology such as wireless LAN (local-area network), and 3G and LTE (long-term evolution) communication networks have developed, increasing the opportunities for viewing OTT (over the top) video with smartphones. Amidst this develop-ment, a screencasting function, in which the content of the smartphone screen can be viewed on a TV receiver has ap-peared. Together with smartphone services that allow video to be viewed anywhere and at any time, the screencasting function makes it possible to also view video on a nearby large-screen TV. In this case, the smartphone can be called the first-screen device.

Technology for the interworking of TV receivers and mo-bile devices such as smartphones (device linking technolo-gy) is being devised and developed for handling the types of service content described above. Device linking technology was adopted early in the IBB Systems such as Hybridcast services in Japan, in HbbTV (Hybrid Broadcast Broadband Television) services in Europe. Such systems make it pos-sible to operate HTML apps that are related to the broadcast in the TV receiver, so communication with HTML apps on the smartphone by inter-app communication is possible, and the expansion of services that use these features can be expected. NHK is researching “content matching technolo-gies” that can connect viewing behavior to daily life and is investigating new Hybridcast device linking technology as fundamental technology for the combined use of broadcast information obtained from the TV receiver and viewer be-havior information obtained from the smartphone. In this paper, these device linking technologies are described. Sec-tion 2 introduces second-screen and multiscreen functions and section 3 explains the trend in mobile-first devise link-ing technology.

2. Second-screen and multiscreen services2.1 Automatic content recognition (ACR) technologies

There have been attempts at interworking between

Device Linking Technology for TV viewing

Hiroshi Fujisawa

FEATURE

4

broadcast services and smartphone apps, focusing on be-havior such as using social network service apps to com-municate with friends and family by sending tweets, etc., while watching television. Consider a smartphone that is in-terworking with a TV receiver while a commercial is being broadcast, for example. Services such as making a video of a singer that appears in the commercial available for down-load to a smartphone or displaying a discount coupon for a retail store on the smartphone to match the timing of the commercial would be possible.

Such services for interworking with mobile devices have been available since the early 2010’s, implemented with au-tomatic content recognition (ACR) technology. The main techniques used for ACR are audio fingerprinting and audio watermarking. Audio fingerprinting uses acoustic features that distinguish different sets of audio data and are stored in a database. The features of the broadcast audio are extracted on the second-screen. The interworking operations are per-formed if a particular feature is matched with the feature in the database. Audio watermarking, on the other hand, uses watermark information such as channel information and time codes. The watermark information is embedded in the broadcast audio data and extracted and decoded on the sec-ond screen to control the interworking operation.

Fingerprinting will work if a detectable feature has been stored in the database, so it is not greatly affected by any re-vision to a program that may be necessary after the program has been completed in the production process. With water-marking, on the other hand, revising the embedded water-

mark information in a program is a troublesome task, so the difficulty of dealing with it in the later stages of production is a disadvantage. An advantage of audio watermarking is that it can be detected with higher accuracy than is possible with audio fingerprinting, which has the disadvantage of be-ing affected by noise in the viewing environment.

2.2 Second-screen services in an Integrated Broadcast -Broadband system

The use of QR codes has been tried in services that con-form to the HbbTV 1.5 specifications for the IBB system used in Europe. HbbTV makes it possible to create an app with HTML on the TV receiver at the time the program is being broadcast. A QR code is generated and displayed on the TV screen with the HTML app, and the camera of a mobile device is then used to read the QR code. The infor-mation from the QR code is used to link the TV receiver and the mobile device via an internet server so that the HTML app on TV and the app on mobile device can work together. A feature of this approach is that a service can be provided without adding any special function to the TV receiver.

With Hybridcast, the IBB system used in Japan, direct communication between TV receivers and mobile devices that are on the same local network is a standard feature that has been available since 2013. For this service, various TV receiver manufacturers have supplied Hybridcast-compat-ible receivers that provide a dedicated remote control app for mobile devices. The Hybridcast system configuration is illustrated in Fig. 1. When the TV is connected to a mobile

HTML app (content)Hybridcast app

Second-screen service

Inter-app communication(API for device interworking)

Device connection(discovery, etc.)

Hybridcast browser

TV of Manufacturer A

Companion app

Remote control app of Manufacturer A

Mobile device(Smartphone or tablet)

Figure 1: Hybridcast system configuration

FEATURE

5

device with a remote control app, communication between a Hybridcast browser on TV and a web browser that runs on the remote control app (referred to as a companion app be-low) is possible. The Hybridcast app runs on each browser, so apps can communicate via a standard API.*1 It is thus possible to provide services that run synchronously with broadcast programs and commercials as well as interactive broadcast services that are operated with mobile devices.1)

3. Mobile-first Device Linking3.1 Viewing internet video with a screencast function

OTT media services*2 that provide video distribution services over the internet have come into wide use in re-cent years, and the use of mobile devices for viewing video has increased. It is not unusual for video-viewing apps to include a screencasting function that allows the screen of a mobile device to be viewed on a TV. For example, if a video-viewing app is running on a smartphone and a TV or set-top box is nearby, a selected internet video or the inter-net video that is being watched can be viewed on the TV with a simple operation of the app. This kind of function is being provided by platform providers via Google Chro-mecast and Apple AirPlay devices, for example. Services that are compatible with app-independent screencasting us-ing the DIAL (discovery and launch) protocol,2)*3 a de facto

standard used by Netflix, are also appearing. The increase in video-viewing apps that use such screencasting functions is considered a result of the increasingly important role that smartphones play in the daily lives of viewers.

3.2 Extension of the device linking function in Hybridcast

The device linking function in the IBB system is be-ing extended based on the DIAL protocol. In HbbTV 2.0, standardization of DVB-CSS (companion screens and streams)3)*4 has enabled companion device and device syn-chronization services by direct communication on the same network. The Hybridcast service in Japan, which had al-ready begun companion device services, has also extended the device linking function to accommodate the increasing use of smartphones by viewers. This section explains the new device linking functions of Hybridcast.

Greater use of smartphones in the daily lives of view-ers will lead to diversification of the devices used, such as smart watches and smart speakers, as well as diversifica-tion of smartphone apps. Providing various kinds of infor-mation from broadcasting services, including news, sports, drama programs, and lifestyle information, to appropriate devices at appropriate times in daily-life scenarios will in-crease the utility of broadcasting services. For example, video on demand (VOD) programs or catch-up programs can be displayed on TV with the simple operation of smart-phone app (Fig. 2). We can also expect to receive notifica-tions of the starting times of important programs sent from

*1 Application programming interface, a command interface for constructing apps.

*2 Video and audio services that are provided on the internet independently of telecom

*3 A communication protocol for detecting TV receivers and launching apps from mobile devices.

Simple selection of a VOD video

Start-up from smartphone that interworkswith the Hybridcast app

Select the video you want to watch

Catch-up app from broadcast station Stop/end play

Immediate transition to the broadcast

Figure 2: Simple smartphone operation lets you view a VOD or a program you missed on the TV

*4 Standards for companion device technology in Europe.

FEATURE

social network services, news apps, and other such sources on the smartphone, the use of the smartphone to control TV receivers, and the seamless viewing of on-air news pro-grams, etc. (Fig. 3).

To deal with diverse application programs, the IPTV Fo-rum Japan standardized a protocol for the interworking of TV receivers and mobile devices (referred to here as the companion device communication protocol).4) The stan-dard protocol makes it possible to connect a single app to TV receivers made by different manufacturers rather than requiring a different companion app for each manufacturer (Fig. 4).

Further more, the standard of companion device com-munication protocol has been extended to enable the con-nection from mobile devices to Hybridcast.5) This protocol extension enables interworking with the diverse services that run on various types of IoT (Internet of Things) devic-es such as robots, smart speakers/watches as well as with smartphones and other mobile devices (Fig. 5).

Here, we briefly describe the extended companion de-vice communication protocol (Fig. 6). First, the terminal authentication function for the TV receiver and the com-panion device determines whether or not the app of the companion device is a legitimate app for connection to the

6

Standard companion device communication protocol

A indivisual companion app for the TV of each manufacturer is required

A Inc.

B Inc.

C Inc.

A Inc.

B Inc.

C Inc.

Figure 4: A single app can connect to TV receivers from different manufacturers

Simple selection of a broadcast program

Receive push notification that a broadcast program is about to begin

Start up app provided by broadcast station

Select “View on TV”

Start-up from terminal that interworks with the Hybridcast app

Use a second-screen service

Figure 3: Smartphone control of TV after receiving a notification

FEATURE

TV receiver. If the app is authenticated, it attempts to in-voke the target Hybridcast app. The TV receiver uses the channel selection information from the companion device and the Hybridcast app information to query an external server that determines whether or not the specified Hybrid-cast app can be launched. If the decision is that startup is possible, the TV receiver selects the specified channel and launched the Hybridcast app, and communication between the TV receiver app and the companion device app begins.

The decision of the external server can prevent the launch of a Hybridcast app that is unrelated to the current broad-cast service being watched. For example, it would not be possible for the Hybridcast app of broadcast station A to launch mistakenly on the channel of broadcast station B.

The significance of implementing the extended com-panion device communication protocol for services is de-scribed below.A) The viewer need not be aware of the device or app that

7

Home network

Extended protocol

TV receiver

Server for deciding whether or not the Hybridcast app can be started up

Companion device

Device discovery

Channel selection + Hybridcast app launch request

Channel selection + Hybridcast launch

Terminal authentication

Query for whether or not the app can be started up

Inter-app communication

Existing protocol

Existing protocol

*When startup is possible*When startup is possible

Figure 6: Extension of communication protocol for Hybridcast companion devices

Now

Future

TV compatible with Hybridcast(broadcast services, video services)

Companion device

Second-screen service

Access from diverse services

Second-screen service

Various IoT devices

Figure 5: Interworking between various IoT device services and broadcast services is possible

FEATURE

invokes a service.B) Interworking of TV programs and other broadcast

content with mobile device and IoT device services is possible.Concerning item A), viewers are not restricted to using

a particular companion app, so it is possible to provide ser-vices in which, for example, viewers that often use smart-phone apps can use a smartphone app as the entry point to the service and viewers that often watch broadcast pro-gramming can use a broadcast program as the entry point. In other words, it is possible to develop services that are suitable for diverse app and device environments.

Concerning item B), linked terminals such as TV receiv-ers and smartphones communicate bidirectionally to pro-vide the services that connect viewing behavior to daily life activities with using broadcast information. That is to say, information from broadcast programs that daily flows back and forth between the TV and mobile devices or IoT devices that can be carried about gives viewers access to various types of broadcast information in various situations in daily life. It might be possible in the near future to auto-matically obtain information on viewer interest expressed by behavior from a smartphone or smart watch, such as the viewer looking at a sign that reads “Introduced in the TV program” while they are shopping.

4. ConclusionIn Japan, digital broadcast satellite broadcasting began

in 2000, terrestrial digital broadcasting began in 2003, and Hybridcast for linking broadcasting and the Web began in 2013. This evolution advanced interworking between broad-casting and internet services and resulted in the rapid prog-ress and widespread use of device linking technology. The result is that television has moved beyond being just about the enjoyment of watching broadcast programs in the liv-ing room and now includes interworking with smartphones and IoT devices. We are thus at the stage of exploring the creation of new services that cross the boundary between indoors and outdoors.

In May 2018, NHK STRL released the Hybridcast Con-nect library, which facilitates device linking. At the NHK STRL Open House 2018, this library was used by NHK and commercial broadcasters to propose various service sce-narios and demonstrate the possibility of new broadcasting services.5)

Currently, it is being said that we are moving towards a smart society in which all kinds of things are connected by IoT. In the future, we can expect the creation of technology that enables content to be presented with any kind of device through expansion of the Hybridcast Connect library.

References1) IPTVFJ STD-0011, “HTML5 Browser Specification 2.1”

(2015)2) DIAL 2nd Screen Protocol Specification (2014)3) ETSI TS 103 2896-2 (V1.2.1), “Digital Video Broadcasting

(DVB); Companion Screens and Streams: Part 2: Content Identification and Media Synchronization”

4) IPTVFJ STD-0013, “Hybridcast Operational Guideline Ver-sion 2.4” (2016)

5) NHK Open House 2018, https://www.nhk.or.jp/strl/open2018/tenji/8.html

8