Huffman Codes

Juan A. RodriguezCS 326

5/13/2003

Presentation Content Introduction Encoding Huffman’s algorithm Huffman’s code Dynamic Huffman encoding Quiz

Introduction Suppose we have to encode a text that compromises of n

characters.

Huffman code is a coding scheme that yields a shorter bit string by applying the idea of assigning shorter codes words to more frequent characters and longer code words to less frequent characters.

Same idea that was used in the mid-19th century by Samuel Morse, where frequent letters such as e(.) and a(._) are assigned short sequence of dots and dashes while infrequent letters such as q (_ _ . _) and z (_ _ . .) have longer ones.

Encoding Fixed Length

encoding assigns to each character a bit string of the same length.

That is what the standard seven-bit ASCII codes does

Letter CodewordA 000B 001C 010D 011E 100F 101

Encoding Variable length

encoding assigns code words a different lengths to different characters

Huffman codes is an example of variable length encoding

Letter CodewordA 0B 101C 100D 111E 1101F 1100

Encoding Prefix codes

No code word is a prefix of another code word

Simply scan a bit string until the first group of bits that is a code word for some character and repeat this operation until the bit string’s end is reached.

Simplifies encoding and decoding.

Huffman’s Algorithm Suppose you have a 1000 character

data file with the following properties:

Character D I _ U E J L R O SFrequency 0.10 0.20 0.01 0.13 0.18 0.10 0.06 0.03 0.15 0.04

Huffman’s Algorithm Step 1: Initialize n one-node trees

and label them with the character of the alphabet

I _ U E J L R O SD

Huffman’s Algorithm Step 2: Record the frequency of

each character in its tree’s root to indicate the tree’s weight.

D

.1

I

.2

_

.01

U

.13

E

.18

J

.1

L

.06

R

.03

O

.15

S

.04

Huffman’s Algorithm Step 3: Find 2 trees with the

smallest weight and make them left and right sub-tree of a new tree and record the sum of their weights in the root of the new tree as it’s weight

D

.1

I

.2

_

.01

U

.13

E

.18

J

.1

L

.06

R

.03

O

.15

S

.04

Huffman’s Algorithm Step 3 continued

D

.1

I

.2

_

.01

U

.13

E

.18

J

.1

L

.06

R

.03

O

.15

S

.04.04

Huffman’s Algorithm Step 3 continued

D

.1

I

.2

_

.01

U

.13

E

.18

J

.1

L

.06

R

.03

O

.15

S

.04.04

.08

Huffman’s Algorithm Step 3 continued

D

.1

I

.2

_

.01

U

.13

E

.18

J

.1

L

.06

R

.03

O

.15

S

.04.04

.08

.14

Huffman’s Algorithm Step 3 continued

D

.1I

.2

_

.01

U

.13E

.18

J

.1

L

.06

R

.03

O

.15

S

.04.04

.08

.14 .2

Huffman’s Algorithm Step 3 continued

D

.1I

.2

_

.01

U

.13E

.18

J

.1

L

.06

R

.03

O

.15

S

.04.04

.08

.14

.2 .27

Huffman’s Algorithm Step 3 continued

D

.1I

.2

_

.01

U

.13E

.18

J

.1

L

.06

R

.03

O

.15

S

.04.04

.08

.14

.2 .27 .33

Huffman’s Algorithm Step 3 continued

D

.1I

.2

_

.01

U

.13E

.18

J

.1

L

.06

R

.03

O

.15

S

.04.04

.08

.14 .2

.27 .33 .40

Huffman’s Algorithm Step 3 continued

D

.1I

.2

_

.01

U

.13

E

.18

J

.1

L

.06

R

.03

O

.15

S

.04.04

.08

.14.2

.27 .33.40

.60

Huffman’s Algorithm Step 3 cont

D

.1I

.2

_

.01

U

.13

E

.18

J

.1

L

.06

R

.03

O

.15

S

.04.04

.08

.14.2

.27 .33.40

.601

Huffman’s Algorithm Step 4: We take the convention

that going left down the binary tree means adding a 0, and going right down the binary tree means adding a 1.

Huffman’s Algorithm Step 4 cont

D

.1I

.2

_

.01

U

.13

E

.18

J

.1

L

.06

R

.03

O

.15

S

.04.04

.08

.14.2

.27 .33.40

.6010

0 1

0

0

0

0

0

0

0

1

1

1

1

1

11

1

Huffman’s Algorithm The algorithm is greedy, which

means that it makes choices that are locally optimal and hopes it yields a globally optimal solution.

Notice that this is a full binary tree: every non-leaf node has two children. This is true of all optimal codes.

Huffman’s Algorithm The operation that we need to

perform repeatedly is the extraction of the two sub-trees with the smallest frequencies. This can be implemented using a priority queue.

Example of Huffman’s algorithm code implementation ML.

The operation that we need to perform repeatedly is the extraction of the two sub-trees with the smallest frequencies. This can be implemented using a priority queue.

Building the initial queue takes time O(n log n) since each enqueue operations takes O(log n).

Then we perform n-1 merges, each of which takes O(log n). Thus this implementation of Huffman’s algorithm takes O(n long n).

datatype HTree = Leaf of char * int | Branch of HTree * int * Htree

fun huffmanTree(alpha : (char * int) list) : HTree = let val alphasize = length(alpha) fun freq(node:HTree):int = case node of Leaf(_,i) => i | Branch(_,i,_) => i

val q = new_heap (fn (x,y) => Int.compare(freq x, freq y)) alphasize fun merge(i:int):HTree = if i = 0 then extract_min(q) else let val x = extract_min(q)

val y = extract_min(q) in insert q (Branch(x, freq(x)

+freq(y), y)); merge(i-1) end

in app (fn (c:char,i:int):unit => insert q (Leaf(c,i))) alpha; merge(alphasize-1)

end

Huffman’s Code The output of Huffman’s

algorithms is Huffman’s code:Charcter Frequency Codeword

D 0.10 010I 0.20 00_ 0.01 101100U 0.13 100E 0.18 111J 0.10 011L 0.06 1010R 0.03 101101O 0.15 110S 0.04 10111

Huffman’s Code Encoding of LORI:

1010 110 101101 00 Given the probabilities

and codeword length the expected bits per character in the code is 2.66.

Had we used a fixed-length encoding for the same alphabet, we would use at least 4 bits per character

Charcter Frequency CodewordD 0.10 010I 0.20 00_ 0.01 101100U 0.13 100E 0.18 111J 0.10 011L 0.06 1010R 0.03 101101O 0.15 110S 0.04 10111

Huffman’s Code This code achieves the compression

ratio, a standard measure of the compression algorithm’s effectiveness, of 35.5%.

Huffman’s encoding of this text will use 35.5% less memory than it’s fixed length encoding.

Extensive experiments with Huffman’s codes have shown the compression ratio for this scheme typically falls between 20% and 80%.

Dynamic Huffman Encoding Huffman’s encoding yields an optimal

(minimal length) encoding providing the probabilities of characters occurrences are know in advance.

Draw back: a preliminary scanning of a given text to count the frequencies of the character occurrences in it. We use the algorithm to compute the an optimal prefix tree, and we scan the text a SECOND time, writing the out the code words of each character of the text.

Dynamic Huffman Encoding Dynamic (Adaptive) Huffman

coding builds a tree incrementally in such a way that the coding always is optimal for the sequence of characters already seen.

Huffman’s Code Quiz Given the

following bit string, what does it decode into?

011111101111001011110110000101111011001010110101101010

Charcter Frequency CodewordD 0.10 010I 0.20 00_ 0.01 101100U 0.13 100E 0.18 111J 0.10 011L 0.06 1010R 0.03 101101O 0.15 110S 0.04 10111