convergence of broadcasting and …...business models. we studied the management of program-re-lated...

NHK STRL ANNUAL REPORT 2014　｜　21

3.1 Hybridcast

We are continuing our research on Hybridcast, which is a service platform for fl exibly providing extensible new services that take advantage of the characteristics of broadcasting and telecommunications networks. In FY 2014, we made progress on R&D aimed at making the platform more efficient and on implementing multimedia broadcasting for Super Hi-Vision, in-cluding 8K Hybridcast (SHV multimedia broadcasting). We also continued our standardization activities. To further promote the Hybridcast service we launched in September 2013, we showed the appeal of Hybridcast by demonstrating it to visitors at the NHK STRL Open House and other exhibitions at home and abroad.

■Advanced HybridcastThe IPTV Forum published the fi rst edition of the Hybridcast

Technical Specifications in 2013. In FY 2014, we studied ele-mental technologies for advanced services, conducted experi-mental verifications, and promoted the establishment of the second edition of the specifi cations.

Version 2.0 of the specifi cations includes additional features

such as support for video on demand (VOD) and recorder func-tionality, development of non-broadcast-oriented managed ap-plications, and high-precision synchronization of broadcasting

We continued researching and developing technologies and services that will lead to the convergence of broadcasting and telecommunications, targeting new broadcasting services for the era in which broad-casting and telecommunications will be tightly integrated.

In our research on Hybridcast, we studied recording and playback functionality, support for non-broad-cast-oriented managed applications that will enable service providers other than broadcasters to providecontent services, and technologies for highly precise synchronization between broadcasting program andbroadband content. We contributed to the establishment of Hybridcast technical specifi cations ver. 2.0 atthe IPTV Forum. We also investigated new multiplexing schemes and coding technologies for closed cap-tions and superimposition based on the Hybridcast technical specifi cations that will enable provision of sophisticated multimedia services for Super Hi-Vision (SHV) broadcasting. We contributed to the estab-lishment of the ARIB STD-B62 standard specifying multimedia broadcasting for SHV.

For device linkage services, we prototyped the TV note service as a non-broadcast-oriented managedapplication for Hybridcast. The service enables viewers to record subjects of interest by simply tapping ona tablet device screen and to interact with external Internet services. We also developed the AugmentedTV service, which offers a new viewing experience: capturing images on a TV screen by using the camerain a mobile device and then displaying them with the related virtual objects on the device. We developed atechnology for estimating the position and rotation of the mobile device relative to the TV screen andconducted implementation evaluations.

In our research on information networks for broadcasting programs, we developed technologies for uti-lizing program-related information for Internet services and for analyzing data collected by social net-working services (SNSs). We prototyped a program information data hub that provides information relatedto broadcaster programs in the Linked Open Data (LOD) format and studied ways to analyze and utilizevarious types of external big data such as data collected by SNSs.

In our research on program viewing analysis, we developed technologies for determining the viewingstatus and estimating the viewer’s interests. We prototyped a system for estimating keywords represent-ing viewer interests on the basis of their TV viewing behavior and for displaying the keywords on a tabletdevice. We also developed ways to generate user profi les based on the estimated keywords and recom-mend some related programs to viewers.

Regarding security technologies for more secure and reliable services, we prototyped a searchable en-cryption system that can search encrypted data on the cloud without decrypting them. We also developeda watermarking algorithm for embedding a receiver’s identity information into compressed data streamsas a way of preventing illegal distribution of content.

For IP content delivery, we began developing a platform that can deliver video efficiently to diverseviewing terminals including Hybridcast receiver-equipped TVs and mobile devices. We also made progressin our research on a distributed server-organized broadcasting system for easy viewing of past programs.We prototyped and evaluated a time-shifting system that enables zapping by tags such as cast member name and on-air time.

3 Convergence of broadcasting and telecommunications

Figure 1. Example program guide that enables direct transition to VOD

22　｜　NHK STRL ANNUAL REPORT 2014


and broadband content. At the STRL Open House, we exhibited our prototype receiver in cooperation with receiver manufac-turers.

The support for VOD and recorder functionality includes VOD playback from an application, recording and playback of broadcast programs, and launch of cooperative applications during viewing of recorded programs. We fabricated a receiver to verify these functions(1) and prototyped and exhibited a pro-gram guide application that enables direct transition to VOD (Figure 1). We are also studying MPEG-DASH (Dynamic Adap-tive Streaming over HTTP) as a VOD content delivery scheme. We conducted technical verifi cations by creating test streams and reproducing them on the TVs of various manufacturers, aiming at inclusion of MPEG-DASH in the Hybridcast operation rules and application for actual services.

Non-broadcast-oriented managed applications are provided by service providers other than broadcasters and are available regardless of broadcast channel. They enable new services and business models. We studied the management of program-re-lated data distributed to service providers(2) (Figure 2) and thepush transmission system. We also prototyped a receiver mod-el(3) and applications and examined an overall system model for using the services(4). An example of our prototype non-broad-cast-oriented managed applications is shown in Figure 3. This application, which receives and displays in real time the data distributed by push technology, provides viewers useful and timely information such as train operation status and weather reports.

High-precision synchronization is the display of content de-livered by broadcasting and broadband networks in a synchro-nized manner in video frame units using time information. This technology is indispensible for multi-angle services, which en-able viewing of images from different angles. In FY 2014, we developed a basic technology for reproducing video on mobile devices such as tablets and smartphones in synchronization

with the main broadcasting program displayed on the TV screen(5). We will continue to test the feasibility of this technol-ogy, which is applicable to various purposes including synchro-nization with data information.

■Promotion of HybridcastWe are providing technical support for NHK’s program pro-

duction departments to enable wider use of the program-linked Hybridcast service, which was launched in December 2013. We provided prototype applications for the Hybridcast service for additional programs that started in September 2014, thereby contributing to its smooth launch.

We also exhibited examples of Hybridcast services offered by commercial broadcasters at the STRL Open House and demon-strated the growing popularity of Hybridcast.

■Effort for SHV multimedia broadcastingUsing the Hybridcast technologies, we prototyped sample 8K

Hybridcast services that take advantage of large, ultra-high-res-olution screens, aiming at the realization of SHV multimedia broadcasting (Figure 4). Displayed on the screen along with the golf program are multiple images provided by broadcasting and broadband networks, multilingual closed-captions, posts on social networking services (SNSs), and various other types of information. We exhibited this example at the STRL Open House.

■Activities related to standardizationDomestic standardization activities included promoting the

establishment of Technical Specifi cations ver. 2.0(6) at the IPTV Forum. We also contributed to the establishment of HTML5 Browser Specifications ver. 2.1 and the Hybridcast Operation Rules ver. 2.0 for a new VOD system supporting HTML5.

We worked on standardization of technologies related to SHV multimedia broadcasting in cooperation with relevant or-ganizations. We examined such technologies as MPEG Media Transport (MMT)-based transmission schemes, browser func-tionality and data coding, and a Timed Text Markup Language (TTML)-based coding method for closed captions and character superimposition(7). Our efforts led to the establishment of the ARIB STD-B62 standard(8) in July 2014, which specifies SHV multimedia broadcasting covering all of these technologies. Regarding closed-captioning and character superimposition technologies, we participated in discussions on exchange for-mats and operation rules, which are indispensable for facility preparation, and created test data required for receiver devel-opment.

International standardization activities included incorporat-ing Hybridcast into a draft recommendation for hybrid systems at the International Telecommunication Union, Radiocommuni-cation Sector (ITU-R) and Telecommunication Standardization

Figure 3. Example non-broadcast-oriented managed application (useful

and timely information)

Figure 2. Prototype management system for distributing program-related data

Figure 4. Example 8K Hybridcast service



Sector (ITU-T). Moreover, Technical Specifi cations ver. 2.0 and ARIB STD-B62 were refl ected in a report on hybrid systems.

At the Technical Plenary/Advisory Committee (TPAC) Meet-ing of the World Wide Web Consortium (W3C) working on HTML5 standardization, we presented an overview of the func-tions specified in Technical Specifications ver. 2.0 to promote understanding and diffusion of Hybridcast worldwide.

[References](1) H. Furumiya, M. Takechi, H. Ohmata, K. Matsumura, S. Sunasaki and

K. Kai: “Support for VOD and recorder function on Hybridcast,” ITE Annual Conference, 24-5 (2014) (in Japanese)

(2) Y. Hironaka, K. Majima, K. Kai and S. Sunasaki: “An Application Dis-tribution Management System for Hybridcast,” IEEE ICCE 2015 (2015)

(3) H. Endo, Y. Hasegawa, H. Ohmata, A. Baba, K. Matsumura and K. Kai: “Implementation Study of a Hybridcast Receiver for Non-broad-cast-oriented Managed Application,” ITE Annual Conference, 24-2

(2014) (in Japanese)(4) H. Ohmata, H. Endo, Y. Hasegawa, A. Baba, K. Matsumura and K.

Kai: “System Architecture for Cross-Channel Application Services on Hybridcast,” IEEE ICCE 2015 (2015)

(5) M. Onishi, H. Ohmata and K. Matsumura: “A Method for Accurate Synchronization between Broadcast and Broadband on Wi-Fi Con-nected Multi-screen Devices,” ITE Annual Conference, 24-7 (2014) (in Japanese)

(6) IPTV Forum Japan: “Integrated Broadcast-Broadband System Specifi -cation Version 2.0,” IPTVFJ STD-0010 (2014) (in Japanese)

(7) A. Baba, S. Takeuchi, M. Takechi and K. Kai: “A Study of Schemes for Multimedia Services on Super Hi-Vision Satellite Broadcasting Sys-tem,” ITE Technical Report, Vol. 38, No. 35, pp. 21–24 (2014) (in Japa-nese)

(8) ARIB: “Multimedia Coding Specifi cation for Digital Broadcasting (2nd Generation),” ARIB STD-B62, Ver. 1.0 (2014)

3.2 Device linkage services

For more sophisticated services that take advantage of broadcasting and telecommunications, we are researching ways to make use of user data as well as device-linked system technology.

■Technology for utilization of user dataTo promote non-broadcast-oriented managed applications

for Hybridcast, we are developing the “TV note” system(1),which supports viewers in their search, purchase, and social networking service (SNS) activities on the Internet by effi ciently collecting information on their interests from broadcast pro-grams and commercials.

This system records the viewer’s interests, which the viewer indicates by simply tapping on the screen of a tablet device linked with a TV on which various subjects of interest are dis-played. It then stores the recorded information as authenticated viewer interest data. It can then guide the viewer to Internet services matching his/her interests after the program.

We developed a method that makes it easier to collect infor-mation from the Internet by linking metadata provided by ex-ternal providers with the subjects of viewer interest. We devel-oped a prototype system as a non-broadcast-oriented managed application that can be shared among multiple broadcasting channels and exhibited it at the STRL Open House (Figure 1).

■Device-linked system technologyWe are proceeding with R&D on a device linkage system that

will provide a new viewing experience (“Augmented TV”). It will capture images displayed on a TV screen with the camera in a mobile device such as a smartphone or tablet and then display them on the device. This system will enable broadcast content to be shown on devices other than receiver equipped TVs, with 3DCGs and other telecommunications content over-laid on the captured images of the TV screen on the mobile de-vice.

We developed a method for quickly estimating the position and rotation of the mobile device relative to the TV screen(2).

The display location of the telecommunications content can beeasily adjusted by using the gyro sensor of the mobile deviceand a simple frame that shows the area of the TV screen.

Implementation evaluations using commercial tablet devicesshowed that capturing the TV screen just once to estimate theposition and rotation is sufficient to maintain the mobile de-vice’s position relative to the TV screen and that the device cankeep displaying telecommunications content naturally andcontinuously even when capturing areas off the TV screen.These results confi rmed the effectiveness of this method.

[References](1) C. Yamamura, H. Ohmata and T. Nakagawa: “TV note – program

viewing annotation and its utilization with Hybridcast -,” ITE AnnualConference, 24-6 (2014) (in Japanese)

(2) H. Kawakita, T. Nakagawa and M. Sato: “Estimation of TV ScreenPosition and Rotation using Camera and Gyro Sensor,” ITE AnnualConference, 13-1 (2014) (in Japanese)

Figure 1. TV note system

Pointer to indicatesubject of interest

Tap on tabletdevice screen

Hybridcast TV

3.3 Content information utilization and program viewing analysis

To provide more attractive services that exploit the advan-tages of broadcasting and telecommunications, we researched

and developed technologies for making use of program-relatedinformation for Internet services, analyzing data collected by

24 ｜　NHK STRL ANNUAL REPORT 2014


social networking services (SNSs), and monitoring viewer viewing behaviors and estimating their interests.

■ Information network for broadcasting programsTo explore the possibility of broadcaster Internet services we

are studying ways to enable information related to broadcast programs to be used for various services provided by broad-casters and service providers. We prototyped a program infor-mation data hub that structures broadcaster program informa-tion into computer-readable Linked Open Data (LOD) format and provides it through application programming interfaces (APIs).

We designed a database structure for describing and sharing program guide information and related information on the ba-sis of program schedule information and implemented auto-matic linkage with the vocabulary of the Japanese version of DBpedia, which is the Internet encyclopedia Wikipedia struc-tured in LOD format. This has enabled the data hub to be con-nected with external open data(1). We also examined individualdata structures and APIs for each program genre, assuming specifi c ways program-related information can be used for var-ious Internet services. For example, we linked health-related programs with a concept map and content map, which present data for program-related knowledge, that we have been re-searching. This has enabled advanced search using knowledge data. For education-related programs, we studied a method for systematizing science video content by linking it with struc-tures extracted from Wikipedia(2).

This program information data hub can facilitate service co-operation with broadcasters and third-party providers who wish to use program-related information. For example, a pro-vider offering cooking and living information can enrich its ser-vices by utilizing information from broadcast health programs. Diverse new Internet services making use of program informa-tion can be expected in various areas.

We also studied ways of analyzing and utilizing a wide varie-ty of big data besides those of broadcasters such as SNS big data and external open data. From the analysis of tweets on Twitter at the time of the Great East Japan Earthquake, we found that messages during a large-scale disaster contain a signifi cantly higher frequency of use of proper nouns such as names of places than in ordinary times. This makes it impossi-

ble to use feature extraction by using such indicators as word signifi cance, which is a common technique for text summariza-tion. We therefore developed a message categorization method that is effective for particular situations like an earthquake(3). This method should be helpful for appropriate and effective message extraction for regional disaster prevention and disas-ter reporting.

■Determination and analysis of viewing behaviorsWe are researching technologies for automatically determin-

ing and analyzing viewing status and for estimating viewer in-terests, with a view to developing broadcasting and life servic-es matching viewer preferences and viewing styles.

Using the analysis results of TV viewing experiments we conducted in FY 2013, we developed a system for estimating keywords related to viewer interests from their TV viewing sta-tus and for displaying the keywords on a tablet device(4). The system, which was exhibited at the STRL Open House in 2014 (Figure 2), collects keywords from the program guide and sub-title information in advance and calculates the signifi cance of each keyword on the basis of appearance frequency and other indicators. During program viewing, the system calculates the degree of viewer interest from his/her viewing status, that is, gaze (head pose)(5) and facial expression. The system then esti-mates candidate keywords related to the viewer’s interests from the time average of the product of the degree of interest and the signifi cance of keywords appearing in the subtitles and displays the keywords in rank order on the tablet in real time. The results of verification using the viewing experiment data showed that the higher ranked keywords were consistent with the keywords the test participants had listed in the question-naire, demonstrating the effectiveness of this method. This technology was used for application development by partici-pants at the NHK HACKATHON event held in May(6). For deter-mination of viewing behaviors, research on gaze behavior esti-mation and on facial expression recognition was conducted in cooperation with Tokyo City University and the University of Tokyo, respectively.

We also developed a method for generating a user profile based on program viewing behavior. The proposed method categorizes the keywords of interest stored during TV viewing and generates a user profi le automatically. More specifi cally, it

Figure 1. Conceptual diagram of program information data hub

Related-content information

Link information to program sites and to

moving and still images

Program ontology

External knowledge

Concept mapContent map

Existing program information DB

Program schedule information, etc.

Program description schema DBpedia

gram informationProg tion

data hub

External LOD data

inked Open DataL(LOD)

format dataBroadcaster

serviceapplications

External

service applications

Cro

ss-s

ectio

na

l u

se

of

da

ta



refl ects the weight of the category of the interest keyword on a tree structure graph built from the categories registered in Wikipedia. We collected viewing behavior data from ten test participants in general households for two weeks and generat-ed their user profi les with the proposed method. Comparison of the profi les with the information in the questionnaire they an-

swered showed that the category distribution generated auto-matically was consistent with their genres of interest. We alsofound that using interest keywords for each participant in-creased the accuracy of estimating the programs that are likely to arouse the participant’s interest.

[References](1) M. Miyazaki, M. Urakawa, I. Yamada, K. Miura, H. Sumiyoshi, H. Fuji-

sawa and T. Nakagawa: “Study on Program Information Database asLinked Open Data,” The Japanese Society of Artificial IntelligenceSIG-SWO-A1402-01, pp. 01.01–01.04 (2014) (in Japanese)

(2) M. Urakawa, M. Miyazaki, I. Yamada, T. Nakagawa and H. Fujisawa:“A study on systematical view of video content based on biologicalontology from Wikipedia,” The 77th National Convention of IPSJ(2015) (in Japanese)

(3) K. Ariyasu and H. Fujisawa: “SNS message analysis for regional pre-vention at a large-scale disaster,” ITE Winter Annual Conference, 3-2(2014) (in Japanese)

(4) Y. Yamanouchi, M. Okuda, M. Takahashi, S. Clippingdale and M.Naemura: “A Prototype System for Estimating TV Viewers’ InterestsBased on Attentional Behavior,” ITE Annual Conference, 7-6 (2014)(in Japanese)

(5) M. Takahashi, S. Clippingdale, M. Naemura and M. Shibata: “Estima-tion of Viewers’ ratings of TV programs based on behaviors in homeenvironments,” Multimedia Tools and Applications, DOI 10.1007/s11042-014-2352-0 (2014)

(6) http://www.nhk.or.jp/kokusaihoudou/hackathon/

3.4 Security technologies

To support the security and reliability of the convergence of broadcasting and telecommunications, we are researching cryptography and watermarking technologies.

■ Cryptography and authentication algorithm forhybrid servicesWe researched cryptography technologies that can be used

to provide secure and reliable hybrid services that take advan-tage of broadcasting and telecommunications.

Service providers need information about viewers in order to personalize services while viewers expect amazing services from providers that they have not previously accessed. An effi -

cient way to meet these requirements is to store viewer infor-mation in the cloud and allow providers to freely access the in-formation (Figure 1). From the perspective of preservingviewers’ privacy, however, it is best to keep the information ac-cessible to the providers to the minimum necessary. We there-fore studied a system that will allow providers to view only theminimum necessary data. The system stores encrypted infor-mation(1), searches encrypted data in the cloud without decrypt-ing them, and provides the provider with only the data match-ing the search condition. We verifi ed the basic operation andeffectiveness of the system using a prototype.

■ Watermarking algorithm for HEVC-compressed streamsFor copyright protection of content, we researched a water-

marking technology that can embed information to identify areceiver into streams compressed with a new coding scheme,High Effi ciency Video Coding (HEVC).

An effective way to protect content from illegal distributionis to identify who is distributing content illegally. At the sametime, various measures must be taken for HEVC-compressedcontent. We therefore developed a watermarking method inwhich micro auxiliary information sent from content providersand the receiver’s identity are embedded into HEVC-com-pressed streams within the receiver(2).

[References](1) G. Ohtake and K. Ogawa: “Privacy Preserving System for Integrated

Broadcast-broadband Services,” IEEE ICCE 2015, pp. 304-305 (2015)(2) K. Ogawa and G. Ohtake: “Watermarking for HEVC/H.265 Stream,”

IEEE ICCE 2015, pp. 110-111 (2015)

Figure 2. Example of determining viewing status and displaying interest

keywords

Interest keywords displayed on tablet device

Determined viewingstatus

Figure 1. Provision of personalized services that match viewers’ preferences

Recommended tionRecommended

ationRecommended information

TV program

Broadcast

stationViewerPart of

information

Part of

information

Part of

information

Cloud server

Encrypted viewerinformation

Provider A

Provider B

26　｜　NHK STRL ANNUAL REPORT 2014


3.5 IP content delivery technology

With the aim of distributing programs over a network, we are researching a video delivery technology that uses the Inter-net and a distributed server-organized broadcasting system to provide broadcast programs over a network.

■Video delivery technology using the InternetTo support large-scale delivery of live video, we began our

development of a delivery platform that can support diverse viewing terminals including Hybridcast receiver-equipped TVs and mobile devices. We released a prototype player that runs on an HTML5 browser and verifi ed its operation on TV receiv-ers, mobile devices, and PCs. These results contributed to the establishment of relevant technical specifi cations(1) at the IPTV Forum Japan. We also conducted delivery experiments with test participants in various regions throughout Japan by using a player capable of measuring the video delivery quality. We col-lected basic data for use in further stabilizing large-scale deliv-ery. We also studied a way of using a distributed processing technology for generating streams for Internet video delivery. We examined the processing performance of a method for gen-erating video streams from an original video fi le by using dis-tributed processing technology that can process a large amount of data at high speed using multiple servers. We obtained data on the relationship between the number of servers used for stream generation and the processing speed as well as on the extensibility of input/output video formats(2).

■ IP transmission equipmentWe developed stored video transmission equipment that can

extract an image captured at a certain time specified by the broadcaster from accumulated images recorded at reporting sites and transmit it over an IP network. This equipment di-vides time-continuous video data into multiple files with a length of a few seconds before storing them. It can also trans-mit multiple files simultaneously so that stable transmission can be maintained even if the bandwidth of the IP network fl uctuates.

We also developed underwater wireless IP transmission equipment using blue LEDs for stably transmitting live video captured by an underwater camera without using a cable.

■Distributed server-based broadcasting systemWe are developing a time-shift viewing system that will ena-

ble program zapping by tags such as cast member name and on-air time so that users can view any of the large number of past programs. To develop this system, we researched a dis-tributed server-organized broadcasting system for accumulat-ing, reprocessing, and distributing broadcast content using multiple servers on the network.

We improved tag search functionality and the interfaces of the time-shift viewing system we prototyped in FY 2013 and conducted system evaluation experiments. The speed of tag search was increased by sorting tag data by on-air time at the time of registration into the database and by creating a list of tags for the same time in advance(3). We also created a keyworddictionary focusing on personal names from the past four years

worth of program guide and subtitle information. Using the dictionary, we built a database containing over ten million re-cords of program-related tags. For the interface, we upgraded the system to enable easy zapping with a remote-control de-vice (Figure 1). We conducted evaluation experiments of the upgraded system and verifi ed its operability and zapping func-tionality(4). Through the experiments, we found that tag zapping gives viewers a chance to find a program they had never viewed and helps them view it. We also developed a system for visualizing viewing logs for use in analyzing viewing behavior. The system can calculate the amount of viewing for each spec-ifi ed tag in video frame units from the collected viewing logs.

We developed a viewing system based on cloud computing that runs the program viewing player for the time-shift viewing system on a server instead of the viewing terminal, so it does not depend on the processing performance of the viewing ter-minal. Only the reproduction of video and sound is handled on the viewing terminal; the other processes requiring more CPU power are run on a cloud server. This enables the usage of var-ious types of program viewing software even on a low-end ter-minal.

[References](1) IPTV Forum Japan: “HTML5 Browser Specifi cation Version 2.1,” IPT-

VFJ STD-0011 (2014) (in Japanese)(2) S. Oda, M. Kurozumi and Y. Endo: “Verifi cation of ultra-high defi ni-

tion video coding on distributed processing framework,” 77th Na-tional Convention of IPSJ, 2E-02 (2015) (in Japanese)

(3) Y. Kaneko, S. Takeuchi, M. Fan and M. Naemura: “A Consideration of Tag Retrieval Including On-air Time Range Field,” FIT 2014, No. 2, pp. 75–76 (2014) (in Japanese)

(4) S. Takeuchi, K. Hiramatsu, Y. Kaneko and M. Naemura: “An Experi-ment of Time-shift Viewing System with Zapping along the On-air Time Axis and Tag,” Proc. of the IEICE General Conference, D-9-29 (2015) (in Japanese)

Figure 1. Time-shift viewing system for tag-based program zapping

convergence of broadcasting and …...business models. we studied the management of program-re-lated...

Documents