rr~kaye xcurrry

DigitalWatermarking: ~From Concepts to \ ’ .Real-Time VideoAmdicationsII- --- - -

re digital watermarking field was bornaround the same time that the World Wide

Web started its phenomenal growth. In recent years,digital watermarking has become increasingly appliedto hide information in digital multimedia data, thusenlisting the water-marking technology in the difficultfight against intellectual property rights infringement.

Watermarking fundamentalsDigital watermarking evolved from the ancient tech-

nique of steganogmphy, which is arguably older than theciphers used in cryptographic systems. Steganographycan be defined as hiding messages in plain sight-thatis, rather than encrypting the message, it is embeddedin a larger volume of data that doesn’t require secrettransmission; this volume of data is the message carri-er. The ultimate goal is to hide messages in such a waythat only the recipient, knowing what to look, for, canextract them. Before the availability of computer sys-tems, this was done manually and evolved almost intoan art all its own. Popular examples include addingminor, coded variations to drawings and paintings;slightly modifying certain words or letters in handwrit-ten material so that the recipient could pick them out todecipher the hidden message, or, as a boundary case,the classic medieval palimpsest written in invisible inkon a normal letter. Obviously, manual steganography isa rather tedious and time-consuming task at best andcan be used only for small amounts of information.

Historically, transmission media was limited to canvasand paper, as our examples show. However, steganog-raphy’s applicability is not limited to visual media. Withthe advent of digital multimedia data and transmis-sion-which includes still-image, video, audio, andgeometry data among others-the fundamental con-cept of steganography can be transferred from the fieldof analog data to the digital world using modern cryp-tography concepts and digital signal processing (DSP)technology.

Digital watermarks began humbly around 1993 with

Christoph Busch, Wolfgang Funk,and Stephen WolthusenFraunhofer Institute for Computer Graphics

the exploration of simple least-sig-nificant bit (LSB) hiding schemes;’ We have developed a secure,most of the seminal works appearedtwo years later, including articles by robust watermarkingIngmar Cox, Ioannis Pitas, and Eck-hard Koch and Jian Zhao.2-4 Since algorithm and applied it inthat time, both the research and lit-erature on the subject have grown digital streaming MPEC-2beyond the boundaries of this intro-duction. We therefore refer you to format video-the format ofsurvey articles such as the one byIngmar Cox and Matt Miller,’ and choice in the broadcast andthe digital-watermarking bibliogra-phy maintained online and in print video stock industry.by Fabien Petitcolas (http://www.cl.cam.ac.ul+fapp2/steganography).

There are many possible ways to define desirable dig-ital-water-marking traits, depending on the application.Some requirements conflict, such as visibility versus .robustness. Here we offer one definition; we later offera brief rationale for its use as a yardstick.

Definition 1. Digital watermarking embeds infor-mation into digital data (the carrier) in asecret and incon-spicuous way. The embedded information is denoted as adigital watermark. The digital watermarkshall be robustagainst distortions that do not significantly degrade thereceiver’s perception of the carrier signal.

Suitable applicationsDifferent kinds of digital data are available on a mul-

titude of storage media types (or storage representa-tions). Obviously, watermarking must be independentfrom the representation in which the digital media datais stored, as otherwise, removing the watermark wouldbe as simple as converting the data from one represen-tation to another. (We assume here that data are storedin digital form; converting digital watermarked dataand storing them in analog can be considered an attackon the watermark.)

Form SF298 Citation Data

Report Date("DD MON YYYY") 01011999

Report TypeN/A

Dates Covered (from... to)("DD MON YYYY")

Title and Subtitle Digital Watermarking: From Concepts to Real-Time VideoApplications

Contract or Grant Number

Program Element Number

Authors Project Number

Task Number

Work Unit Number

Performing Organization Name(s) and Address(es) Fraunhofer Institute for Computer Graphics

Performing Organization Number(s)

Sponsoring/Monitoring Agency Name(s) and Address(es) Monitoring Agency Acronym

Monitoring Agency Report Number(s)

Distribution/Availability Statement Approved for public release, distribution unlimited

Supplementary Notes

Abstract

Subject Terms

Document Classification unclassified

Classification of SF298 unclassified

Classification of Abstract unclassified

Limitation of Abstract unlimited

Number of Pages 12

REPORT DOCUMENTATION PAGEForm Approved

OMB No. 074-0188Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering andmaintaining the data needed, and completing and reviewing this collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,including suggestions for reducing this burden to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188), Washington, DC 20503

1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE

1/1/993. REPORT TYPE AND DATES COVERED

Article4. TITLE AND SUBTITLE

Digital Watermarking: From Concepts to Real-Time VideoApplications

5. FUNDING NUMBERS

6. AUTHOR(S)

Christoph Busch, Wolfgang Funk, Stephen Wolthusen

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER

IATACInformation Assurance Technology AnalysisCenter3190 Fairview Park DriveFalls Church VA 220429. SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORING / MONITORING

AGENCY REPORT NUMBER

Defense Technical Information CenterDTIC-IA8725 John J. Kingman Rd, Suite 944Ft. Belvoir, VA 2206011. SUPPLEMENTARY NOTES

12a. DISTRIBUTION / AVAILABILITY STATEMENT

A

12b. DISTRIBUTION CODE

13. ABSTRACT (Maximum 200 Words)

The digital watermarking field was born around the same time that the World WideWeb started its phenomenal growth. In recent years, digital watermarking has becomeincreasingly appliedto hide information in digital multimedia data, thus enlisting the water-markingtechnology in the difficultfight against intellectual property rights infringement.

14. SUBJECT TERMS

Digital watermarking15. NUMBER OF PAGES

16. PRICE CODE

17. SECURITY CLASSIFICATION OF REPORT

Unclassified

18. SECURITY CLASSIFICATION OF THIS PAGE

UNCLASSIFIED

19. SECURITY CLASSIFICATION OF ABSTRACT

UNCLASSIFIED

20. LIMITATION OF ABSTRACT

None

The watermarking algorithms apply to individual sig-nal domains (audio, video, and so on) and must bedesigned specifically for each domain. It’s insufficientto simply adapt algorithms from one domain to acceptsignal data from another because the robustness crite-rion would not be met for distortions in the new domain.A robust watermarking algorithm, however, must adaptitself to expected attacks within the specified domainand must be tunable to allow a balancing of robustnessand visibility (audibility, and so on). Safe conversionsbetween multimedia data types are therefore possibleonly within one such media domain.

Generally, because covert messages have some noiserobustness, they can be integrated into all multimediadata types and later retrieved from the manipulateddata. But is this really watermarking? Certainly hidingand retrieving messages is not enough. Following defi-nition 1 above, the watermark also must be robust. Resis-tance to manipulation is the discerning feature of adigital watermark.

The key to a robust watermark is to introduce it witha high degree of redundancy. This gives us a criterion todecide if a medium is well suited for water-marking. Datainherently high in redundancy can carry a more robustwatermark because a highly redundant watermark canbe embedded without being noticeable. Besides redun-dancy, the remaining prerequisite for a suitable carriersignal is that the signal must tolerate at least small, well-defined modifications without changing its semantics.

Real world still-image, video, and audio data aretherefore well suited for watermarking. Problems arisewith some artificially generated data (such as raytracedscenes) because, unlike samplings from real-world data,they do not contain noise unless they have been sub-jected to lossy compression algorithms, which mask thewatermarking effects and make it possible for thehuman eye to detect irregularities at lower insertion lev-els than with “natural” sources. We discuss this type ofprocessing in more detail below.

The most difficult media for watermarking are plaintext and executable files. Although binary executablesmay contain some redundancy, introducing randommodifications into such data is discouraged. Even whenthe modifications are safe, attacks are trivial since anyexecutable file can be converted into a canonical for-mat-and the watermark thus removed-without los-ing any information needed for execution.

The question with textual data itself is, “How do youintroduce additional information without rendering thedata useless?” There are schemes for solving this prob-lem, such as encoding the secret message as the first let-ter of each word in the text, but such schemes are easilybroken. A newer technique for text watermarking is tointroduce slight format changes to the text,6 such as lineand word shifts, to encode a message. Obviously, sucha watermark will not survive format conversion.Because text does not contain any natural noise, you canextract its contents and easily remove all formattingredundancy. For example, a commercial optical-character-recognition package operating on a scanned

Iprintout or other bitmapped representation would eas-ily eliminate all text watermarks.

Because of these difficulties, we deal here only with Ithe more promising watermarking candidates: in par-titular, image and video data. With the advent of Digi- Ital Versatile Disk (DVD) as a serious distributionmedium, the intellectual property of the video commu-nity is now exposed to the same dangers that haveplagued digital imaging and digital audio interestgroups for at least a decade.

LimitationsThere are two important issues here that must be

understood. First, digital watermarking for copyrightprotection comes into play only after offenses have beencommitted (disregarding for the moment the deterrenteffect some proponents claim). Second, detecting secretwatermarks is a computationally complex operation,limiting the practicality of spidering the Web for water-marked data.

Bearing this in mind, digital watermarks should beused as the method of last resort for intellectual propertyrights problems, after more conventional means havefailed. Conventional mechanisms include those used forcontrol ap auditing as well as copy protection schemes.These mechanisms are valid in their own right and can-not be fully replaced by digital watermarks, even though ’the intrusiveness and limitation of system usability suchmechanisms bring often causes potential users to rejectthem. In any case, these methods and their implemen-tation issues exceed the scope of this article.

The watermarking processThe steps for implementing a watermarking system

are straightforward and exploit the properties of datathat include a natural noise margin. We now reviewissues to consider.

Basic considerationsWatermarking naturally splits into two rather differ-

ent areas. The first area is the security of a watermark-ing system: We must decide where to embed thewatermark in the data and how to restrict access to theinformation. The location may refer to either the spa-tial or frequency domain and should be determinedusing some randomness source. The second area is sig-nal processing: We must decide how to modify the orig-inal data to embed the watermark.

SecurityThe design criteria for a secure watermarking algo-

rithm are the same as for a cryptographic algorithm. Infact, the same criteria the National Bureau of Standards ,(now the National Institute of Standards and Technol-ogy) specified in 1973 for a proposed standard crypto-graphic algorithm’ also apply to a watermarkingalgorithm.

Definition 2. To bemarking algorithm must

considered secure, the water-

n provide a high level of security,n be completely specified and easy to understand,n rely on a key for security rather than on the algo-

’ rithm’s secrecy (this was already a requirement ofKerckhoffs in 1883)8

n be available to all users,W adapt to diverse applications,n be economically implementable in electronic devices,w be efficient, andn permit validation.

A watermark can be implemented using a secret keyto restrjct access to the locations where it is embedded.A secret key regulates access to the watermark.

The security of any watermarking algorithm can beincreased easily by encrypting the watermark before itis embedded. Any standard cryptographic algorithm canbe used for this purpose. However, the encryptingapproach has an important drawback: All message bitsmust be retrieved correctly or the original message can-not be decrypted. While it is possible to include error-correcting code, this increases the size of the message,which should be as small as possible.

Some commercially available algorithms for imagewatermarking forego the secure-watermarking princi-ples in their search for higher robustness and ease of useand distribution. Such algorithms use a single, commonsecret key to embed information (usually the data cre-ator’s identity compres$ed to a short string), which canthen be read by anyone with the detection program; nokey is required beyond a common one used for allembeddings and extractions. As we explain below, allother parameters being equal, the amount of data thatcan be embedded is inversely proportional to the water-mark’s robustness. This requires a mapping of moredetailed information in a central database onto a com-pact identification string.

The danger in this case is that the entire algorithm,including the secret key, must be embedded in the codefor both embedding and detecting the watermark. Bydisassembling the code, you can modify the algorithm’s

’ behavior. For example, removing the block against over-watermarking allows attackers to simply rendenthe orig-inal watermark unreadable and even claim &mershipthemselves. Another potential attack is to embed a thirdperson’s (publicly available) author-identification stringinto material and thus create ownership that could beused as fraudulent evidence for criminal activities.

When watermarking schemes use a single secret keyfor ail embeddings and extractions, there is no way todetermine who watermarked data first unless youexpand the scheme significantly by introducing a cen-tral trusted registration database. This can provide atime-stamp facility for thwarting attacks like thosedescribed above. However, such registrations must beaccomplished over an authenticated and encryptedchannel to ascertain the claimant’s identity. This helpsprevent injection attacks, in which a third party insertsmaterial on behalf of a legitimate claimant through apreestablished, authenticated connection.

In essence, this scheme leaves no way to resolve anargument involving multiple claimants, reducing theevidentiary value to zero. Given the fact that schemesfor breaking the program code (though not the algo-rithm) of such commercial packages have been pub-

lished on the Internet, it should be very difficult to evenuse such systems for prima facie evidence.

The first requirement for a secure digital marking sys-tem therefore must be that, while the algorithm forembedding is itself publicly known, the key used todetermine the actual positions or frequencies of theembedded mark (which is different from the optionalencryption key) will never be disclosed to parties otherthan the data owner and possibly law enforcementauthorities. Such a marking has evidentiary value, butby itself does nothing to inform the user of copyrighteddata. It is therefore often advisable to embed a water-mark with both a publicly known key and a secret water-mark. This public mark provides embedded, auxiliaryinformation-such as the contact person for licensingthe material-which does not alter the data’s appear-ance and will survive most processing steps.

The second important requirement for the secretwatermark is that it must be impossible to distinguishfrom the original data’s white noise. If some patternsremain characteristic of an embedded watermarkregardless of its key (even if this occurs only after elab-orate transformations), the watermark can be removedor destroyed without knowledge of the secret key.

Media modificationThe second building block of a watermarking algo-

rithm is the signal-processing step, in which the origi-nal data is modified to enable embedding of thewatermark. This step can be considered as adding a sec-ondary signal into a primary one; assuming this mes-sage has been encrypted, it will appear as white noise.Although robustness increases with the strength of theadded’signal, the initial signal’s quality degrades. Inmost cases, you must subjectively determine the point atwhich the noise degrades the original signal too much.

During normal processing, digital data are subject tovarious manipulations that weaken the embedded sig-nal’s strength or skew it in a way that makes recon-struction and eventual discovery more difficult (suchas rotation, cropping, shearing, and color adjustments).Obviously, such processing can also be used intention-ally to attack a watermark. The threshold for the mark-ing signal’s maximum strength depends on severalfactors. One is the subjective level of degradation themarker can accept. Another factor is a perceptual issue:People are more forgiving of noise levels when the datais noisy to begin with, such as in still photographs,which are noisy due to imperfections in the signal-acquisition mechanism (such as a charge-coupleddevice sensor).

If, on the other hand, data is used that does not contain “natural” noise-such as computer-generatedimaging-people notice a much smaller addition ojnoise. In this case, you can sometimes completelyreconstruct the original data by performing a spectra:analysis over it and eliminating all frequencies witl-intensities below a certain threshold. As an alternativeyou can use the noise signal distribution by averagingthe data points presumed to be of equal value in theoriginal data set and using them to interpolate theremaining data points.

1 An unmarkedcomputer gen-erated image (a)and a water-marked com-puter generatedimage (b).

2 8 January/February 1999

For example, Figure la shows a computer-generatedimage; Figure lb shows the same image after noiseinsertion. Some artifacts appear in both Figure la andlb. These result from a fairly strong JPEG compression,which is typically used for low-bandwidth transmissionmedia. Nonetheless, the noise introduced by the water-marking process is still plainly visible. Some water-marking systems produce less noticeable artifacts. If,for example, they operate on the same block size ele-ments as JPEG or MPEG, the distinction between com-pression-induced errors and those caused by thewatermarking algorithm is less clear. On the otherhand, the inverse holds true for other compression algo-rithms such as iterated function systems (fractal com-pression systems) or wavelet-based algorithms.

Lossy compression is often a necessary processingstep for multimedia data. These mechanisms takeadvantage of the human perceptual system to eliminatecertain data that is least significant for data perception.(This complicated issue is covered in more detail else-where.‘)) Spectral energy distribution is a good indica-tor for the significance of individual frequencies. Othereffects that can be considered include auditory mask-ing, which lets a strong signal make an entire range ofweaker signals in its spectral neighborhood inaudible;and the perceptual system’s limited sensitivity to dif-ferent frequencies (this applies to both sound waves and

colors). Watermarks are problematic here, since theythemselves strive to be perceptually invisible (inaudi- /ble, and so on) and thus are likely victims of lossy corn- Ipression algorithms; they must strive to remain at theborderline of visibility.2 Most modern watermarkingalgorithms prove resilient against common lossy com-pression algorithms.

In the case of other manipulations, the main problemis recovering the watermark signal without the original(unmodified and unmarked) data. This is often arequirement, as the modified data would otherwiseneed to be matched manually against their likely origi-nals. (The alternative would be a reliable metric for sim-ilar multimedia data sets, but the current state of the artdoes not supply this.) This manual matching is unac-ceptable in commercial settings, which require auto-mated procedures.

In the case of audio and video streams, another pos- isible modification or attack is to cut or resample the datastreams. Since the watermark detection algorithmrequires some form of synchronization, it’s possible tocircumvent it without degrading the signal quality(unless more elaborate mechanisms or manual corre-lations are used). This is particularly significant whenit’s undesirable (or even impossible) to store potential- ’ly pirated data for analysis (such as in broadcast video).Further discussion of attack schemes on digital andvideo watermarks is available in the literature.‘O~”

Copyright protection of videoDigital video streams in MPEG-2 format’ form tile

basis of emerging television standards like the EuropeanDVB project, l2 the DVD video standard, and video stockarchives for instant random access to digital videostreams. Providers of digital contents and services needcopyright protection mechanisms including digitalwatermarking to track the dissemination of their digitalproducts and detect large-scale’commercial piracy andillegal copying of their data.

Here we present an algorithm for digital watermark-ing that has been integrated in a system for real-timewatermarking of video streams. We designed and tunedthe algorithm to meet the requirements of the produc-tion chain in TV studios and to enable the watermark toovercome the lossy MPEG-2 compression scheme. Thecomplete system has been set up within the project Tal-isman (Tracing Authors’ rights by Labelling Image Ser-vices and Monitoring Access Network),i3 that wasfunded by the European “Advanced CommunicationsTechnologies & Services” program. Talisman com-menced in September 1995 and was completed byAugust 1998. The system embeds watermarks into videostreams, monitors for labels in MPEG-2 bit streams, andintegrity checks in video, all in real time. We restrict ourdiscussion here to the core watermarking algorithm.

At the moment, special hardware is needed toachieve real-time watermark embedding for videostreams of TV studio quality. Our Talisman partners,Thomson CSF (France) and CSELT (Italy), managedthe extremely important hardware-related aspect ofvideo watermarking, including hardware design andDSP-unit programming.

Application scenarioFigure 2 sketches the basic sce-

nario for which the watermarkingalgorithm was designed. A videostream acquired using a digitaldevice is subsequently processed inthe TV production chain and finallybroadcast as an MPEG-2 bit stream.

The digital data is marked direct-ly after recording and before enter-ing the production chain. The datais not necessarily recorded in aclosed studio environment; it can berecorded in the field and subse-quently transmitted to the broad-caster’s headquarters. The rawmaterial is cut and assembled dur-ing the production process. The finalversion of the video stream is encod-ed according to the MPEG-2 stan-dard, and the compressed videostream is broadcast. Other partiescan digitally record the data andbroadcast it again instantly, or withsome delay, as a digital video stream without any lossof quality. The watermarkmonitoring process can checkfor the watermark and automatically detect unautho-rized use of copyrighted material.

RequirementsThe application scenario outlined above imposes

some restrictions on the water-marking algorithm:

n Watermark embedding and retrieval must be per-formed in real time to avoid slowing down TV pro-duction (recording, viewing, archiving, and so on).

n Watermarks must be of “transparent quality”; that is,they must be invisible in the high-quality original dig-

’ ital material used in TV production.n The MPEG-2 compression decreases the digit$ video

stream’s data rate by reducing the noise level withinthe data. Watermarks must survive this degradation.

m Slight translations of the video frames may occur dur-ing production and transmission. Horizontal shiftsmainly appear during analog-to-digital conversion,and a vertical shift of one line may occur in video mix-ers and digital video effects. Both problems arerestricted to older installations and to new unitsimproperly installed.

Implementation issuesWe took a frame-based approach to video water-

marking. During watermark embedding and retrieval,the algorithm separately processes each frame of theuncompressed video stream. Jana Dittmann and hercolleagues” proposed a similar approach. Other tech-niques for video watermarking work on longersequences and can process compressed bit streams.15~16

The original video data stream comes from a digitalcamera or a digital video player. The standard digitalequipment in professional TV studios implements theInternational Tplecommllnirntinnc Thinn fTTTTJ cnl

sampling standard. Digital equipment connects via theITU-R 656 serial digital interface. The data is stored indigital betacam format, which provides lossless com-pression of the video Streams, resulting in a compres-sion ratio of 1.77: 1.

The interface between the video system and thewatermarking algorithm receives ITU-R 601 digitalcomponent signal as input and redirects the luminancecomponent of each original frame to the watermarkingalgorithm. The watermark is embedded and the modi-fied data returned to the interface, which inserts themodified luminance component in the ITU-R 601 outputstream. When running in monitoring mode, the algo-rithm receives the decoded MPEG-2 stream from theinterface and performs the watermark retrieval processfor the luminance component of each frame of thedecoded stream (see Figure 2). The interface providesthe algorithm with consecutive frames of the digitalvideo stream and directs the watermarked data to theoutput device. Interfacing and watermark processingmust not take longer than 40 ms per frame to achievethe PAL frame rate of 25 frames per second.

The mbdified Koch-Zhao algorithmAt first glance, watermarking single video-stream

frames seems similar to watermarking still images. Thisis because our approach to video watermarking modifiesan algorithm designed for still pictures and applies it toeach video-sequence frame. Our starting point was thealgorithm proposed by Koch and Zhao.4

The most important modifications we introduced tothe Koch-Zhao algorithm are new discrete cosine trans-form (DCT) and inverse DCT routines, which we opti-mized for real-time implementation with hardwaresupport and additional checks of edges and texturesprior to watermark embedding and retrieval to avoidartifacts in the watermarked video sequences. Here wer?,r.+..:+ -.... -1 L ‘1 1 -1’ 1 . ., . . . , .

2 Broadcastingenvironment forthe watermark-ing algorithm.

3 Luminancequantkationtable proposedby the JPECstandard.

4 Sub-bandsused for water-

marking,

1

I I16

I 72 92 95 98 112 100 103 99

fx

rv

/I//II

Level 1 n Level 2 q Level 3 n PublicI

The algorithm is block based and shares some fea-tures with the JPEG standard for still-image compres-sion.’ The image’s luminance component is divided into8 x 8 pixel blocks. The algorithm selects a sequence ofblocks and applies the DCT to each of the selectedblocks. The transformed blocks are quantized with theluminance quantization table proposed in the JPEGstandard and shown in Figure 3. The quantization stepdivides a DCT coefficient’s value by an integer numberthat depends ‘on the coefficient’s position within theblock matrix. Generally speaking, high-frequency coef-ficients are divided by higher quantization values thanlow-frequency coefficients. The integer values formingthe quantization table can be multiplied or divided by aconstant value to allow scaling of the quanitzation’simpact on the coefficients.

‘Iwo components of the algorithm must be considered:

1. The position for the watermark embedding must begenerated. To do this, we use a key to initialize a

pseudo-random-number generator that determinesthe order of block processing and the coefficient tobe modified within the block. The key may be pub-lic or secret, leading to a public or secret watermark.

2. We must embed the watermark. To do this, we mustchoose a method to modify the coefficients selectedduring the position-generation step. Even thoughthe number of watermark bits is not restricted bythe algorithm, but rather by the number of suitableblocks in the frame, we will assume a fixed lengthof 64 information bits. This lets us redundantlyembed the watermark, which is necessary for it tosurvive the MPEG-2 compression scheme.

The algorithm can embed up to four different, non-interfering watermarks in each frame. This is accom-plished by dividing the frequency range forwatermarking into four sub-bands, as Figure 4 shows.The sub-bands denoted as levels 1 through 3 are usedfor secret watermarking; the fourth band is used for pub-lic watermarking. We choose one sub-band prior toembedding or reading the watermark and use itthrough&t the video stream.

For each selected block, we encode one bit as follows(we use ti to denote the absolute value of quantized DCT ’ Icoefficients’ absolute value; the subscript i identifies thecoefficient position within the sub-band) :

1. The pixel values within the block area are trans-formed using the DCT.

2. A mechanism for detecting edges is applied thattakes advantage of the block’s DCT representation.

3. A pair of DCT coefficients (L, Q ) is selected from theappropriate sub-band of the transformed block.Each sub-band consists of three coefficients, lead-ing to six possible coefficient pairs.

4. The selected coefficient set is quantized.5. The set’s tivalues are used to determine if the block is

well textured and suited for watermark embedding.6. Depending on the bit value to be embedded, tk and

&I must hold a predefined relationship. The condi-tion for encoding a “1” bit is tk 1 tl + d; the relation-ship for encoding “0” is tk + d S 4. If the requiredrelationship does not already occur naturally, thecoefficients are changed accordingly. This is equiva-lent to overlaying a 2D cosine pattern on the origi-nal block data. The impact of the pattern on theblock’s visual quality in the pixel domain can bescaled by adjusting the noise level d (the difference).

7. The changed coefficients are multiplied by the quan-tization value at the corresponding position of the ,quantization matrix and embedded into the DCTtransformed block. The block data is inverse DCTtransformed to the pixel domain, and the alteredblock is put back to its original position inside theimage matrix.

To increase the watermark’s robustness against thelossy MPEG-2 compression, the watermark is embed-ded with maximum redundancy. All blocks available inthe video frame are subjected to the watermarkingprocedure.

The retrieval process is symmetrical to the embed-ding procedure. The secret key is needed to find the cor-rect sequence of blocks and the coefficients within them.These coefficients are evaluated, and if a pattern fromthe predefined set is found, the corresponding bit valueis recorded. If no valid pattern is found (tk = S) the bit ismarked as not readable. Sixty-four “slots” are set up (onefor each code bit), and each retrieved bit is put into theappropriate slot. After processing all blocks, the corre-sponding bits are used to reconstruct the original bits ofthe code word. The decision in each slot is based on its

Time f

\ majority entries.

The visibility problemThe transition from watermarking for still images to

video sequences revealed properties of the original Koch-Zhao algorithm that escaped noticebefore. The most prominent featureof video sequences is the increasedsensitivity to changes introduced bythe watermarking process. Even forparameter settings that minimize thewatermark3 strength, watermarkingartifacts are visible in high-qualitydigital !,etacam video sequences.

Figure 5 illustrates the visibilityproblem. The block position is nolonger restricted to a s&gle image (x (a)

and y axes) but extends to the timeaxis, t. The modification of blocksthat arc close to each other in x andy as well as in t can result in flickeringeffects. To avoid such degradation,video streams must be processed (b)more carefully than still images.

6 Visibility of edge blocks. Theoriginal image (a), the unmarkedclipped image (b), and the imagemarked without the check algo-rithms (c). Artifacts are visible inboth edge and homogenousregions.-Once the check algorithm

is applied, the artifacts are nolonger visible (d).

Homogenous areas within frames

w (a

fx

are particularly sensitive to this type of degradation asare regions containing sharp edges. mo criteria for

’ checking blocks before actually embedding the water-mark information have been introduced: edge $eetectionandplain Qneu detection mechanisms. Figure 6 shows DCI’artifacts in edge and homogenous areas. The effects areexaggerated to make the problem visible using a stillimage. Figure 6b shows the original clipped image; 6c

\ shows the clip marked without the check algorithms.Here, artifacts inboth homogenous and edge regions areclearly visible. Figure 6d shows the same part marked andchecked for edge and homogenous blocks.

Edge detection. Numerous edge-detection algo-rithms are available, ranging from simple ones like theSobel operator to more sophisticated ones, such aswavelet-based schemes.” Because our algorithm aims atreal-time capabilities, tools we use for edge detectionmust work fast. Even simple schemes like the Sobel oper-ator have a computational complexity too high to beapplied in real time.

To minimize the overhead for edge detection: we usefeatures computed within the basic algorithm cycle. Thelowest frequency DCT coefficients of a transformedblock can be used to decide whether the block containsan edge. We can illustrate this bv lookinn at the DCT

transform of a step function, where the lowest frequen-cy terms have very high amplitudes compared to thetransform of smoother functions. To separate edgedblocks from textured blocks, we apply the following cri-terion: If the absolute value of one of three lowest ACcoefficients exceeds a predefined static threshold, the

” block is classified an “edge block.” Figure 7 shows the

5 Visibilityproblem inconsecutivevideo frames.

Modified block 1

7 Coefficients .used for edgedetection.

8 Retrievalresults for“football” at6 Mbps.

9 Retrievalresults for“Julia” at6 Mbps.

1 - football 6 Mbps

0’ I I I I I I I

0 1 0 0 200 300 400 ,500 600 700

Frame number

80YB

0 I I I I I I I

600 7000 100 200 300 400

Frame number

500

coefficients used for edge detection. We determined athreshold value of 40 through numerous heuristicexperiments. Our goal was to detect all edge blockswithout catching too many textured blocks well suitedfor watermark embedding. However, the thresholdvalue depends on the specific implementation of theDCT transform used.

Plain-area detection. The detection of smoothblocks is equivalent to a block’s texture analysis. To avoidcomputational overhead, we use a criterion based onthe block’s quantized DCT coefficients. Instead of count-ing the number of non-zero coefficients in a transformedblock’s predefined region, as proposed by Benham eta1.,18 we look at the set of quantized coefficients select-ed for modifications. If one of the set’s quantized coef-ficients equals zero, the block is classified as a “plainblock.”

Treatment of invalid blocks. You can follow twobasic strategies after detecting a block that may beunsuitable for watermarking. First, you could skip allinvalid blocks, losing one code bit per block. The sec-ond strategy is an adaptive technique that reduces the

amount of changes to the DCT coef-ficients to minimize the impact onthe block’s visual quality. We pro-pose a third strategy that attemptsa compromise between theseapproaches. For blocks classified asplain, we adjust the modificationsto lower values-that is, we reducethe changes to the coefficient ampli-tudes. Blocks classified as edgeblocks are always skipped.

The watermark-reading processhas to apply the same mechanismsas the embedding process. If anedge block is detected, it is ignoredduring watermark retrieval. Forplain area blocks, the algorithmuses a higher reading sensitivity toevaluate the current bit valueencoded in the block. The criteriafor block checking prior to water-mark embedding and retrieval mustbe stable. Changes introduced bywatermark embedding or opera-tions on the watermarked framemust not affect the criteria; it theydo, block misclassifications signifi-cantly decrease the bit-retrievalrate.

Real- time considerations?tvo components prove crucial to

achieving real-time watermarkembedding and retrieval:

n the interface between the algo-rithm and the luminance compo-nent of the digital video signal,and

n the signal processor that executesthe algorithm’s core (that is, theDCT and IDCT).

We rewrote the DCT and IDCT to work on 16-bit inte-ger values. The DCT only computes the quarter of DCTcoefficient actually needed for watermarking (as shownin Figure 4). The IDCT has been restricted to trans-forming back only the changes introduced to the select-ed coefficient pair.

Shift resistance \Like other block-based watermarking methods, our

algorithm is not resistant to geometrical transforma-tions unless these transformations are inverted beforewatermark detection. Shifts can occur in some videoinstallations, so we have to apply a mechanism for syn-chronization before monitoring the watermark. The Tal-isman system implemented a special watermarkingalgorithm for detecting a watermarked frame’s origin.This algorithm neither interferes with the core water-marking algorithm nor significantly degrades imagequality.

Robustness against MPEC-2encoding

Despite the quality demands inwatermarked video streams, thewatermark must be robust againstdigital TV’s MPEG-2 encoding. Thewatermark information must sur-vive MPEG-2 encoding, which isapplied immediately before trans-mission.

Two features of the MPEG-2, standard are very important (and

challenging) for watermarkingalgorithms:

n The MPEG-2 encoder is veryeffective at removing spatial andtemporal redundancy from thevideo stream.

n The MPEG-2 encoder keeps thevideo stream’s data rate constant.The data are compressed as muchas needed to reach this goal. Inrare cases, the encoder will skipcomplete frames.

.‘1.We tested detection performance

and invisibility of the watermarkedvideo streams on 12 digitized se-quences of digital betacam quality(I’I’U-R 601 format). Each sequencewas 30 seconds long, correspondingto 750 video frames. The sequenceswere very different in character,ranging from feature films to syn-thetic clips.

We used the following test setupfor each sequence:

8

100

0

- leonardo 6 Mbps

0 100 200 300 400 500 600 700Frame number

100

80 t

II

II- synthetic 6 Mbps 1I

01 I I I I I I I

0 100 200 300 400 500 600 700Frame number

n First, we fixed the algorithm’s parameters and water-marked the single frames, then stored thetin disk.

n Next, we animated the frames using a high-quality M-JPEG codec (compression ratio 1:2 to 1:3) and esti-mated the perceptual quality of the watermarkedsequence. If artifacts were visible, we adjusted thealgorithm’s parameters and generated a newsequence of watermarked frames. We repeated thisprocedure until we found no more artifacts.

H We input the watermarked frames into the softwareMPEG-2 encoder/decoder of the MPEG SoftwareSimulation Group. The data were encoded at MainProfile and Main Level. Two data rates were tested, 4Mbps and 6 Mbps.

n We decoded the data, stored it on disk as singleframes, and read the watermark from each frame.

n Finally, we repeated all the steps until we found themaximum watermark retrieval rate without intro-ducing visible artifacts into the original sequences.The resulting watermarked sequences were submit-ted to TV professionals, who checked for watermarkvisibility under studio conditions (digital betacamrecorders and digital monitors). If they objected to

the quality of the watermarked sequences, we modi-fied the algorithm and repeated the test cycle.

We selected test results from four representativesequences to show the algorithm’s performance and lim-its with respect to MPEG-2 compression. Figures 8through 11 show the results with the frame number asthe independent variable and the dependent variable asthe percentage of bits retrieved correctly from a singleframe.~For each sequence, we tested 750 frames (30 sec-onds of video). The results are from watermarkedsequences that professionals viewed as of acceptable qual-ity. The data rate of the encoder was 6 Mbps.

Figure 8 shows results from “football,” a TV broadcastof a football match that had fast and medium movementsFigure 9 shows results from “Julia,” an advertisement fora Hollywood movie that had fast and slow movementFigure 10 shows results from “Leonardo,” a high-qualityTV production with slow movement. Figure 11 showaresults from “synthetic,” a rendered sequence of a truckwith very fast movement.

The retrieval rate depends heavily on the sequence?characteristics. Two aspects of video watermarking muslbe considered here. First, sequences containing a nat

10 Retrievalresults for"Leonardo" at6Mbps.

11 Retrievalresults for“synthetic“ at6 Mbps.

12 Time inte-gration of25 frames for“)ulia” at6 Mbps.

. 13 Time inte-gration of59 frames for“football” at6 Mbps.

I

I- julia 6 Mbps 25 1

5 10 15 20 25Window number

5t 60s t

- football 6 Mbps 50

0 2 4 6 8 10 12 14

Window number

ural noise level obviously suit watermarking wellbecause more blocks pass the check for edges and tex-ture. The same holds true for watermarking of stillimages.

Second, as we noted above, MPEG-2 guarantees a con-stant data rate by exploiting the redundancy of videos inspace and time. The sequence is decorrelated in time bya technique called motion compensation, which storesonly the motion vectors that are needed to derive con-secutive frames from each other. This mechanism workswell as long as those frames don’t differ too much. If thesequence is moving fast, MPEG-2 can’t use motion com-pensation, and the frame-based lossy compression isenforced.

The test results reflect these issues. “Julia” is a real-world sequence with a noise margin sufficient for water-marking. Though the sequence contains some fastmoving parts, the embedded watermark’s redundancyenables it to survive the compression. The “football”sequence behaves very similarly, although the back-ground in many frames is uniform and more blocks areclassified as ‘.‘plain.” “Leonardo” is a very calm sequence;it has high visual quality and contains little noise.Nonetheless, the retrieval rate is very high because highcompression can be achieved by motion compensation.“Synthetic” shows the worst case of video watermarking.The sequence contains no noise at all and changes veryquickly. The retrieval rate is far from significant.(Retrieval rates below 50 percent are possible becauseblocks with tl = t2 are classified as “not readable.“)

Integrating the retrieval results for single frames canhelp overcome the encoder’s influence without increas-

ing the strength of the watermark insertion process.Essentially, we can exploit the retrieval results of 25 or50 consecutive frames in the same way as single bits areread within a frame. This significantly increases thedetection rate. Figures 12 and 13 show the time inte-gration of two sequences; we used the integrated win-dow as the independent variable and the percentage ofcorrect bits as the dependent variable. Figure 12 showsthat integrating 25 frames of the “Julia” result from Fig-ure 9-an integration window of 25 frames, 1 secondof PALvideo-is sufficient to detect the watermarkwithvery high accuracy. The result of integrating 50 framesof the “football” result appears in Figure 13.

ConclusionsOur algorithm for watermarking and monitoring

video streams in a TV-broadcasting environment sur-vives MPEG-2 compression of high-quality, real-world ivideo sequences without degrading their quality. Apply-ing the algorithm to fast-moving synthetic videosequences requires much longer time-integration inter-vals than the SO-frames-wide window we used in thetests presented here. The algorithm’s current version iswell suited for water-marking digitalvideo streams suchas movies or sporting events. ,

The format of digital video is restricted to some well-defined geometry. This makes geometric distortionsdetectable and removable. To introduce geometric dis-tortions, an attacker would have to decode the videostream, process it, and encode it again.

Much video data is only valuable as a live event, suchas a soccer match. This significantly reduces the value ofthe data when stored for later broadcast. Thus, becausepirates must attack the watermark in real time, the costof the attack increases.

Future work will focus on the reconstruction of theoriginal geometry of distorted watermarked videoframes and more elaborate time integration methods.Other topics to be considered are copyright protectionschemes for MPEG-4 coded video data and synchro-nized watermarking of the video and audio channel ofbit streams. n

References1. A. Tirkel et al., “Electronic Watermark,“Di,&zZlmage Com-

puting, Technology andApplications-DICTA 93, MaquarieUniversity, Sidney, 1993, pp. 666-673.

2. I. Cox et al., “Secure Spread Spectrum Watermarking forMultimedia,” IEEE nuns. on Image Processing, Vol. 6, No.12, Dec. 1997, pp. 1673-1687 (originally published in 1995’as NEC TR 95- 10).

3. I. Pitas, and T.H. Kaskahs, “Applying Signatures on DigitalImages,” IEEE Workshop on Nonlinear Image and Signal Pro-cessing, IEEE Computer Society Press, Los Alamitos, Calif.,1995, pp. 460-463.

4. E. Koch and J. Zhao, “Towards Robust and Hidden ImageCopyright Labeling,” Proc. 1995 IEEE Workshop on Nonlin-ear Signal and Zmuge Processing, IEEE Computer SocietyPress, Los AIamitos, Cahf., 1995, pp. 452455.

5. I. Cox and M. Miller, ‘A Review of Watermarking and the

Importance of Perceptual Modeling,” Human Vision andElectrtonic Zmuging II, No. 3016, Society of Photo-OpticalInstrumentation Engineers (SPIE), Bellingham, Wash.,1997, pp. 92-99.

6. Low et AI., “Document Marking and Identification UsingBoth Line and Word Shifting,” Znfocom 95, IEEE Comput-er Society Press, Los Alamitos, Cahf., 1995.

7. B. Schneier, Applied Cryptography, 2nd edition, John Wiley& Sons, New York, 1996.

8. A. Kerckhoffs, “La Cryptographic MiIitaire,“J. des SciencesMilitaire, Ninth Series, Feb. 1883, pp. 161-191.

9. K. Rao and J.J. Hwang, Techniques & Standardsfor Image,Video andAudio Coding, Ch. 11, Prentice Hall, Upper Sad-dle River, N.J., 1996, pp. 273-322.

10. EA. Petitcolas, R.J. Anderson, and M.G. Kuhn, “Attacks onCopyright Marking Systems,” Information Hiding: SecondIntel Workshop, Vol. 1525 of LNCS, Springer-Verlag, Berlin,1998, pp. 219-239.

11. S. Wolthusen, “On the Limitations of Digital Watermarks:A Cautionary Note,” Proc. ofWorZdMu2ticonference on Sys-tem& Cybernetics, andlnformatics (SCI98)/Int’l Con& onInformation Systems Analysis and Synthesis (ISAS 98), Vol.4, Int’l Institute of Informatics and Systemics (IIIS), July

1998.12. U. Reimers, “Concept of a European System for the Trans-

mission of Digitized Television Signals via Satellite,”SMF’TE J., Vol. 103, No. 11,1994, pp. 741-747.

13. Talisman Project, 1998, http://www.tele.ucI.ac.be/TAL-ISMAN.

14. J. Dittmann, M. Stabenau, and R. Steinmetz, “RobustMPEG Video WatermarkingTechnologies,“ACMMultime-dia 98, ACM Press, New York, 1998, pp. 71-80.

15. F. Hartung and B. Girod, “Digital Watermarking of Rawand Compressed Video,” Proc. PIE 2952: Digital Com-pression Technologies and Systems jot Video Communica-tion, Society of Photo-Optical Instrumentation Engineers(SPIE), Bellingham, Wash., 1996, pp. 205-213.

16. L. Qiao and K. Nahrstedt, “Watermarking Methods forMPEG Encoded Video: Toward Resolving Rightful Qwn-ership,” IEEE Multimedia Computing and Sysrqms Con,,IEEE Computer Society Press, Los Ahunitos, C&f., 1998.

17. S. MaIIat and W. Hwang, “Singularity Detection and Pro-cessing with Wavelets,” IEEE 7Pans. on Znformation Theo-ry, Vol. 38, No. 2, IEEE Computer Society Press, LosAIamitos, Cahf., 1992.

18. D. Benham et al., “Fast Watermarking of DCT-based Com-pressed Images,” Proc. Inr’l Con)? on Imaging Science, Sys-tems, and Applications, CSREA Press, Athens, Ga., 1997.

Christoph Busch is head of theDepartmentofSecurity Technologyfor Graphics and CommunicationSystems at the Fraunhofer Institutefor Computer Graphics, where he is

1 responsible for the acquisition, man-1 agement, and control of various

applied research and developmentprojects. He is also a lec-turer on applied wavelet transforms in the educationalprogram ofthe Compute; Graphics Center and is current-ly a partner in several European projects, including ACTS’Okapi, Talisman, and Ocatlis projects and Esprit’s Aime-diaproject, all of which deal with copyrightprotection andconditional access for interactive multimedia services.

Busch studied geodetic sciences at the Technical Uni-versity of Darmstadt, where he received a Ph.D in comput-er graphics in 1997.

Wol&an.g Funk is a member of theresearch staBat the Fraunhofer Insti-tute, where he works on digitalwatermarking, biometric authenti-cation methods, and image andvideo processing. Funk received hisdiploma in physics in 1994fiom the

University of Wiirzburg, Germany.

Stephen Wolthusen is an assis-tantscientist at the Fraunhoferlnsti-tute. His research interests includedigital watermarking, host and net-work security, cryptography andcryptanalysis, and number theory.He is currently working on a distrib-

uted digital content security system that incorporates cryp-tography and watermarking. Wolthusen is also workingtoward a diploma in computer science at the Technical Uni-versity of Darmstadt, Germany.

Readers may contact the authors at the FraunhoferInstitutefor Computer Graphics IGD, DepartmentSecuri-ty Technologyfor Graphics and Communication Systems,Rundeturmstr 6, D-64283 Darmstadt, Germany; {bus&,wfink, woZt)@igd.fhg.de