1
00:00:42,150 --> 00:00:50,360
Necessary and sufficient condition for a code
to be instantaneous is that none of the code
2
00:00:50,360 --> 00:00:59,080
word in the cord should be a prefix of another
code word in the code. Let us try to prove
3
00:00:59,090 --> 00:01:10,980
this, now the sufficient portion of this test
is very clear from the definition of the instantaneous
4
00:01:10,980 --> 00:01:20,780
code. If you have a codeword, which is not
a prefix of another code word, then we can
5
00:01:20,790 --> 00:01:30,550
decode the sequence of code symbols in a direct
manner. Let us say when we received the code
6
00:01:30,550 --> 00:01:39,680
symbol sequence, then we start scanning this
sequence until we reach the sub sequence
7
00:01:39,680 --> 00:01:46,680
corresponding to a code word.
Now, this sub sequence must correspond to
8
00:01:46,800 --> 00:01:55,800
some code word, because we have made the assumption
that none of code word is a prefix of another
9
00:01:55,810 --> 00:02:02,810
code word in the same code. So, as soon as
we receive the subsequence corresponding to
10
00:02:02,920 --> 00:02:10,979
a particular code word we can decode it as
that subsequence belonging to a particular
11
00:02:10,979 --> 00:02:17,920
symbol in the source alphabet, so there is
not time lag in the decoding process. To prove
12
00:02:17,920 --> 00:02:29,820
the necessary portion of this test, let us
assume the contrary, and in order to prove
13
00:02:29,820 --> 00:02:37,800
the necessary portion of this test, let us
assume a contrary and derive the contradiction.
14
00:02:38,239 --> 00:02:54,119
So, let us assume that I have a code word
x i which is a prefix of some other code word
15
00:02:54,120 --> 00:03:06,840
x j in the code. Now, when we received the
code symbol sequence, then when we come upon
16
00:03:07,250 --> 00:03:19,000
the sub sequence corresponding to the code
word x i. Then at that instant we are not
17
00:03:19,000 --> 00:03:28,359
sure whether that substituent’s belongs
to the x i of x j, because we have made the
18
00:03:28,359 --> 00:03:36,459
assumption that x i is the prefix of x j.
In order to make the decision we have to observe
19
00:03:36,459 --> 00:03:45,659
few more code symbols in the main sequence
before we take the decision. So, what it means
20
00:03:45,660 --> 00:03:51,900
that there is time lag in our decoding process
and therefore, the code is not instantaneous.
21
00:03:51,900 --> 00:04:01,299
So, we started with the assumption that if
x i is a code word prefix of another code
22
00:04:01,299 --> 00:04:06,239
word x j, then the code cannot be instantaneous.
23
00:04:06,239 --> 00:04:16,919
So, this proves both the necessary and sufficient
condition of the tests that for a code to
24
00:04:16,930 --> 00:04:24,510
be instantaneous no complete code word of
the code should be a prefix of some other
25
00:04:24,660 --> 00:04:26,100
code word.
26
00:04:26,540 --> 00:04:37,960
Now, let us summarize what we have talked
about codes, in summary we can say that we
27
00:04:37,970 --> 00:04:56,960
have a class of all codes and out of all these
codes we are interested in a sub class of
28
00:04:56,960 --> 00:05:18,180
all these codes and that is non singular codes.
We have also seen that just the constrain
29
00:05:18,190 --> 00:05:26,720
of non singularity is not sufficient for us
to decode the sequence of code symbols. So,
30
00:05:26,720 --> 00:05:33,629
what we want that code should be also uniquely
decodable and that forms another subclass
31
00:05:33,629 --> 00:05:40,629
of non singular codes and this is a subclass
and that is uniquely decodable codes. Finally,
32
00:05:50,199 --> 00:05:57,199
we said that what we would be interested is,
in another subclass of this uniquely decodable
33
00:05:59,409 --> 00:06:06,409
code and that is
instantaneous codes. So, let us summarize
all this with the help of a simple example.
34
00:06:27,470 --> 00:06:34,470
So, let us assume that I have a source consisting
of four source symbols and for this source
35
00:06:45,979 --> 00:06:52,979
I have different codes. One code is where
the code symbols for this source symbols are
36
00:06:59,860 --> 00:07:06,860
0 0 0, and let us assume once that our code
symbols come from the binary code alphabet.
37
00:07:11,590 --> 00:07:18,590
So, if this code is a singular code, because
for each of this source symbol the code words
38
00:07:22,479 --> 00:07:29,479
are not unique. Another one would be 0, 0
1 0, 0 1, 1 0, so this I would say it is a
39
00:07:34,669 --> 00:07:41,669
example of singular code, where all the code
words are same for all the source symbols.
40
00:07:42,629 --> 00:07:49,629
This is an example of a non singular code,
but when I received 0 1 0, than I am not able
41
00:07:57,360 --> 00:08:04,360
to decode it in unambiguous manner, because
0 1 0 could either corresponds to S 2or it
42
00:08:09,060 --> 00:08:16,060
could correspond to S 1 followed by S 4 or
it could correspond to S 3 followed by S 1.
43
00:08:21,009 --> 00:08:28,009
So, this is another example of a not uniquely
decodable code, but this is non singular,
44
00:08:32,390 --> 00:08:39,390
not uniquely decodable. Another example would
be 1 0, 0 0, 1 1, 1 1 0, this code is uniquely
45
00:08:52,300 --> 00:08:59,300
decodable, but this code again is not instantaneous.
When I received code symbols sequence, than
46
00:09:21,690 --> 00:09:28,690
when I have 1 0 or 0 0, I can uniquely, immediately
identify this subsequence 1 0, 0 0, as either
47
00:09:32,260 --> 00:09:38,240
S 1 or S 2.
But when I received a subsequence in this
48
00:09:38,240 --> 00:09:45,240
sequence of code symbols, where the subsequences
1 1, then immediately I cannot decide, I have
49
00:09:46,900 --> 00:09:53,900
to see basically what follows this 1. If 1
follows after this, then I can be sure that
50
00:09:55,850 --> 00:10:02,850
this sub sequence corresponds to S 3, but
if a 0 follows, then I have to check for the
51
00:10:05,680 --> 00:10:11,920
number of zeros which follow after this 1.
If there are even numbers of zeros following
52
00:10:11,920 --> 00:10:18,920
after this 1, it means that this subsequence
1 1 must be S 3, but if there are odd numbers
53
00:10:22,770 --> 00:10:29,770
of zeros following this 1. Then what it means
is that 1 1and the next 0 would be S 4 and
54
00:10:34,830 --> 00:10:41,830
other zeros would be decoded as S 2.
So, I have to observe the
future symbols when I receive 1 1, so I cannot
55
00:10:52,560 --> 00:10:59,560
decode immediately, so this is example of
a non instantaneous code. Finally, I have
56
00:11:01,140 --> 00:11:08,140
0 1 0, 1 1 0 and 1 1, this is an example of
instantaneous code. In the light of what we
57
00:11:19,120 --> 00:11:26,030
discuss earlier we see that none of the code
words in this code is a prefix of another
58
00:11:26,030 --> 00:11:33,030
code word in the same code. Now, in order
to appreciate the nature of constraints on
59
00:11:41,730 --> 00:11:48,670
the code, when we want to have this code as
instantaneous, let us make some primitive
60
00:11:48,670 --> 00:11:53,860
attempts to synthesize an instantaneous code.
61
00:11:53,860 --> 00:12:00,860
So, let us say that I have a source consisting
of five symbols and I am interested in designing
62
00:12:14,140 --> 00:12:21,140
or synthesizing a binary instantaneous code,
so my code alphabet is assumed to be binary
63
00:12:24,590 --> 00:12:31,590
and it is 0, 1. So, let us start synthesizing
this binary instantaneous code. One way in
64
00:12:38,680 --> 00:12:45,680
which we could start is that I can assign
0 symbols to the source symbol S 1. So, my
65
00:12:52,860 --> 00:12:59,860
first code word consists of length 1 and it
is code symbol 0 that is assigned to S 1.
66
00:13:06,360 --> 00:13:13,360
Now, for code word which is to be assigned
to S 2, I cannot start with 0, because that
67
00:13:18,550 --> 00:13:24,730
will violate the condition for the instantaneous
code, so the only alternative left for me
68
00:13:24,730 --> 00:13:31,730
is to start with 1.
Now, again I cannot assign a single code symbol
69
00:13:33,850 --> 00:13:40,850
1 to source symbol S 2, because if I did that,
then I will be exhausted in the sense that,
70
00:13:43,820 --> 00:13:50,820
I will not have any more symbols to start
my code words for other source symbol S 3,
71
00:13:52,510 --> 00:13:59,510
S 4, S 5. If I want my code words to be instantaneous
and for the code word to be instantaneous,
72
00:13:59,880 --> 00:14:06,730
none of the code words in the code can be
a prefix of another code word. So, I cannot
73
00:14:06,730 --> 00:14:13,730
assign 1 to S 2, so the only alternative left
for me is to use 1 0. Now, once I have assigned
74
00:14:15,970 --> 00:14:22,970
1 0 to S 2, what is left out is I can use
1 1, now if I have 1 1 again I cannot solely
75
00:14:27,900 --> 00:14:34,900
assigned to S 3, because if I did that, than
I will not be able to form the code word for
76
00:14:35,980 --> 00:14:42,110
S 4 and S 5 in a way where S 1, S 2, S 3,
will not be a prefix.
77
00:14:42,110 --> 00:14:49,110
So, one way to assign S 3 would be 1 1 0,
now what is left out would be 1 1 1. Now,
78
00:14:55,250 --> 00:15:02,250
I have two more symbols, so again I cannot
assign this 1 1 1 to S 4, I can assign 1 1
79
00:15:02,660 --> 00:15:09,660
1 and 0 to S 4 and finally, to S 5 I assign
1 1 1. So, this is one way of synthesizing
80
00:15:15,260 --> 00:15:22,260
and binary instantaneous code, where I started
with a code word length of 1 for S 1 and reached
81
00:15:28,890 --> 00:15:35,890
to the code word length of 4 for both S 4
and S 5. Now, from this example it is very
82
00:15:40,070 --> 00:15:47,070
obvious that when I start with fewer symbols
initially, then for the code words or for
83
00:15:51,380 --> 00:15:57,040
the later code words, the length of the code
word increases.
84
00:15:57,040 --> 00:16:04,040
So, what it means that if I were to start
with more number of code symbols for my starting
85
00:16:06,200 --> 00:16:13,200
source symbol, than I would had more freedom
to choose my code words for my later source
86
00:16:15,770 --> 00:16:22,770
symbols and probably what would happen that
my later code words would not have been so
87
00:16:24,730 --> 00:16:31,730
large. So, let us try to test this conjecture
with synthesizing another binary instantaneous
88
00:16:33,700 --> 00:16:40,700
code where I assign two symbols, code symbols
to my first source symbols as 0 0. Now, once
89
00:16:46,270 --> 00:16:53,270
I have assigned two code symbols 0 0 to S
1, I have three more possibilities which I
90
00:16:53,770 --> 00:17:00,280
can use as an assignment of code words for
my remaining source symbols.
91
00:17:00,280 --> 00:17:07,280
Now, since I have four more left and I have
three, I cannot assign all the remaining code
92
00:17:11,400 --> 00:17:18,400
symbols of length 2 to S 2, S 3, S 4. So,
what I will do basically is assign to S 2
93
00:17:18,789 --> 00:17:25,789
as 0 1 to S 3 I will assign 1 0 to S 4 I cannot
assign 1 1, because if I did that, if I assign
94
00:17:37,559 --> 00:17:44,559
1 1, than I would not have anything remaining
to be assigned to S 5 in a way where none
95
00:17:47,460 --> 00:17:54,460
of this code words form a prefix. So, the
only way for me to assign S 4 is 1 1 0 and
96
00:17:55,400 --> 00:18:02,400
now for S 5 I can assign as 1 1 1. So, this
is another code which I generate, another
97
00:18:05,340 --> 00:18:12,340
binary instantaneous code which I have generated.
Now, if I call this code number 1 and if I
98
00:18:12,500 --> 00:18:19,500
say this is code number 2 and if I asked the
question, that which of the two code is better,
99
00:18:20,540 --> 00:18:27,540
than I cannot answer this question based on
the information given to me so far.
100
00:18:29,330 --> 00:18:36,330
So, it is not, but one thing is clear that
when I use smaller code word lengths for the
101
00:18:42,670 --> 00:18:49,450
assignment of the initial source symbols,
than the code word lengths which are left
102
00:18:49,450 --> 00:18:55,690
for the assignment to the later source symbols
will be larger. So, this is the example out
103
00:18:55,690 --> 00:19:02,690
here, but when I start with a larger code
word length for the initial symbols, than
104
00:19:04,400 --> 00:19:11,400
I get more or less balance code word lengths
for all the source symbols. So, in the construction
105
00:19:13,810 --> 00:19:20,650
of an instantaneous code, the shorter we make
the first few code words of the code, the
106
00:19:20,650 --> 00:19:27,650
longer we will have to make the later code
words, by using 0 as a code word in the first
107
00:19:27,820 --> 00:19:34,110
code word, we restricted all other code words
to starting with 1.
108
00:19:34,110 --> 00:19:41,100
Whereas, in the second code when we use 0
0as the first code word, in this case we were
109
00:19:41,100 --> 00:19:48,100
able to form later code words by starting
with 1 and in addition we could also use code
110
00:19:49,210 --> 00:19:56,210
words starting with 0 1. So, this example
helps us to understand the constraints which
111
00:20:02,170 --> 00:20:09,170
are imposed when we try to design instantaneous
codes. Now, what we have seen so far is a
112
00:20:12,720 --> 00:20:19,010
qualitatively we have looked at the constraints
on the code word lengths of an instantaneous
113
00:20:19,010 --> 00:20:26,010
code. Now, the same constraints can be put
in a quantitative fashion and that is being
114
00:20:26,980 --> 00:20:33,980
done by what is known as Kraft’s inequality.
So, Kraft’s inequality gives us a quantitative
115
00:20:49,390 --> 00:20:56,340
constraint on the word lengths of the code
words which we can use for an instantaneous
116
00:20:56,340 --> 00:21:02,200
code. So, let us look at the Kraft’s inequality.
117
00:21:02,200 --> 00:21:09,200
So, consider
an instantaneous
code with source alphabet S equal to S 1,
S 2 up to S q and code alphabet this is r
118
00:21:41,750 --> 00:21:48,750
symbol code alphabet, so I will have x 1,
x 2 up to x r. So, with the help of this code
119
00:21:52,910 --> 00:21:59,910
alphabet I form code words for this source
symbol. Let me identify this code words as
120
00:22:03,059 --> 00:22:10,059
X 1, X 2 and X q, now each of this code words
is a sequence of code symbols chosen from
121
00:22:21,780 --> 00:22:28,780
code alphabet x. So, each of this will have
a length, so l 1, l 2 and l q, l 1 means there
122
00:22:39,510 --> 00:22:46,510
are l 1 number of code symbols from this code
alphabet which forms a code word x 1 for the
123
00:22:47,580 --> 00:22:51,630
source symbol S 1.
Similarly, l q means, there are l q number
124
00:22:51,630 --> 00:22:58,630
of code symbols from this code alphabet x,
which compromise, which make up the code word
125
00:23:01,830 --> 00:23:08,830
x q, for the code for the source symbol S
q. Now, it is often desirable that the lengths
126
00:23:11,360 --> 00:23:18,360
of the code words of a code be as small as
possible. So, now necessary and sufficient
127
00:23:20,330 --> 00:23:27,330
condition for the existence of an instantaneous
code with code word lengths given by l 1,
128
00:23:29,400 --> 00:23:36,400
l 2, l q are provided by the Kraft inequality.
129
00:23:36,590 --> 00:23:43,590
So, what is the Kraft inequality says, that
necessary
and sufficient condition
130
00:24:02,140 --> 00:24:09,140
for the existence
of an instantaneous code with code word lengths
l 1, l 2, l q is that, where r is the number
131
00:25:06,110 --> 00:25:13,110
of different symbols in
the code alphabet, so this is the Kraft inequality.
Now, before we try to prove this Kraft inequality,
132
00:25:36,570 --> 00:25:43,570
it will be instructive to look at some examples
and see how we might utilize this inequality
133
00:25:50,220 --> 00:25:57,220
to decide whether a given sequence of code
word lengths can really form the code which
134
00:26:01,400 --> 00:26:08,110
is instantaneous. So, with that let us take
a simple example.
135
00:26:08,110 --> 00:26:15,110
So, if I have source S consisting of S 1,
S 2, S 3, S 4, symbols for this source, let
136
00:26:28,270 --> 00:26:35,270
me assume that I have a code which is 0 0,
0 1, 1 0, 1 1, let me say that this code is
137
00:26:39,470 --> 00:26:46,470
A. There is another code I have 0, 1 0 0,
1 1 0, 1 1 1, this is my code B. I have another
138
00:26:51,690 --> 00:26:58,690
code C which is 0, 1 0, 1 1 0, 1 1 1. I have
another code D which is 0, 1 0 0, 1 1 0, 1
139
00:27:06,690 --> 00:27:13,690
1 and finally, I have another code e which
is 0, 1 0,1 1 0 and 1 1. Now, if someone gives
140
00:27:21,110 --> 00:27:28,110
me this codes, five codes for this source
and if the question was to find out whether
141
00:27:32,170 --> 00:27:39,170
this codes are instantaneous or not, then
Kraft equality by itself cannot help me to
142
00:27:40,570 --> 00:27:47,570
decide whether code is instantaneous and not.
Yes, what the craft inequality helps us to
143
00:27:47,880 --> 00:27:54,880
find out, whether the code word lengths which
I have chosen are reasonable choice for instantaneous
144
00:27:57,400 --> 00:28:04,400
code. Let me take a simple example to explain
this, now will try to evaluate the Kraft inequality
145
00:28:04,570 --> 00:28:11,570
for each of this five codes, so we are interested
in evaluating since our alphabet in this case
146
00:28:13,220 --> 00:28:20,220
is binary. So, the value of r we have is equal
to 2, so the Kraft inequality will become
147
00:28:20,809 --> 00:28:27,809
2 minus l i, i going from 1 to 4. So, let
us try to evaluate this expression for each
148
00:28:29,410 --> 00:28:36,410
of this code, so for the code A minus l i,
i is equal to 1 2 4 will give me as 2 minus
149
00:28:47,549 --> 00:28:54,549
2 plus 2 minus 2, because each code word is
length 2, so what I get is equal to 1.
150
00:28:58,030 --> 00:29:05,030
So, this satisfies my Kraft inequality which
says that this quantity should be less than
151
00:29:06,059 --> 00:29:11,990
equal to 1. Now, because it satisfies this
Kraft’s equality, I can be assured that
152
00:29:11,990 --> 00:29:18,990
I can have code word lengths of 2 2 2 2, so
all the code word lengths can be of equal
153
00:29:19,450 --> 00:29:26,450
length 2 for this case. But just because this
equality, inequalities satisfy it is not mean
154
00:29:30,770 --> 00:29:37,260
that code words are chosen appropriately for
the instantaneous codes, for instantaneous
155
00:29:37,260 --> 00:29:44,100
code. In this example if you examine the code
words for this code A they are also instantaneous.
156
00:29:44,100 --> 00:29:51,100
So, it is important to note that the Kraft
inequality is the restriction on the word
157
00:29:51,510 --> 00:29:58,320
lengths of code and not on the code words
itself.
158
00:29:58,320 --> 00:30:05,320
So, this is we get and this code is both instantaneous
and it satisfies the Kraft inequality. Let
159
00:30:05,460 --> 00:30:12,460
us try to evaluate the same inequality for
the code B, for the code B what I would get
160
00:30:12,630 --> 00:30:19,630
is if I try to evaluate this 2 minus l i is
coming out 2 minus 1 plus 2 minus 3 plus 2
161
00:30:26,670 --> 00:30:33,670
minus 3 plus 2 minus 3 this turns out to be
7 by 8 and again this is less than or equal
162
00:30:35,320 --> 00:30:42,320
to 1. So, what it says that code word lenghts
of 1 3 3 3 are acceptable as the code word
163
00:30:44,610 --> 00:30:50,299
lenghts for instantaneous code. Again if we
examine the code words of this code, you will
164
00:30:50,299 --> 00:30:57,299
find that they are instantaneous. Example
of C
is equal to 2 minus 1 plus 2 minus 2 plus
165
00:31:08,790 --> 00:31:17,020
2 minus 3 plus 2 minus 3 and this turns out
to be 1 again this is less than equal to 1.
166
00:31:17,020 --> 00:31:25,100
So, again this length, code word length of
1 2 3 3 is acceptable and again if you examine
167
00:31:25,110 --> 00:31:32,070
this code words for C, it turns out to be
instantaneous, because none of this code words
168
00:31:32,070 --> 00:31:43,669
are prefix of other code. Code C was actually
obtained from code D by dropping code symbols
169
00:31:43,669 --> 00:31:54,419
0 from the second code word. Finally, let
us look at the code D, now this code D is
170
00:31:54,419 --> 00:32:04,079
obtained from code B by just dropping a symbol,
code symbol 0 from the last code word corresponding
171
00:32:04,090 --> 00:32:07,210
to S 4 in the code B.
172
00:32:07,210 --> 00:32:20,010
Now, for D to evaluate, for D if you evaluate
this expression, it turns out to be 2 minus
173
00:32:20,010 --> 00:32:30,080
1 plus 2 minus 3 plus 2 minus 3 plus 2 minus
2 and this turns out to be 1 is less than
174
00:32:30,080 --> 00:32:42,320
equal to 1. Now, again is very important to
note that what it says that the code word
175
00:32:42,330 --> 00:32:56,940
lengths of 1 3 3 2 is acceptable, same out
here it was 12 3 3, but if you examine this
176
00:32:56,940 --> 00:33:07,880
code D, it is not instantaneous, because code
word corresponding to S 4 that is x 4 here
177
00:33:07,880 --> 00:33:17,000
is a prefix of a code word x 3, so in no way
this can be instantaneous code. But the code
178
00:33:17,010 --> 00:33:28,620
word lengths are acceptable as instantaneous
code, and finally if you look for the code
179
00:33:28,620 --> 00:33:37,900
E and if you evaluate this expression, we
get as 2 raise to minus 1 plus 2 raise to
180
00:33:37,900 --> 00:33:47,520
minus 2 plus 2 raise to minus 3 plus 2 raise
to minus 2 and this turns out to be, now this
181
00:33:47,530 --> 00:33:57,919
is greater than 1.
Now, as soon as I have this inequality not
182
00:33:57,919 --> 00:34:04,919
satisfied, than I can immediately say that
I cannot have this as a instantaneous code,
183
00:34:06,110 --> 00:34:12,520
because the code word length itself violates
the condition which is required for instantaneous
184
00:34:12,520 --> 00:34:19,520
code. Before we look into the proof of the
Kraft’s inequality, let us consider one
185
00:34:20,210 --> 00:34:27,210
more example to appreciate the constraints
which go in the synthesizing of a instantaneous
186
00:34:30,899 --> 00:34:31,869
code.
187
00:34:31,869 --> 00:34:38,869
So, let us assume that I have a source consisting
of ten symbols and those ten symbols could
188
00:34:40,819 --> 00:34:47,819
be decimal digits from 0 to 9 and I am supposed
to code this source with a binary alphabet.
189
00:34:53,690 --> 00:35:00,690
Let us make one more assumption, that for
some reason the occurrence of 0 and 1 is very
190
00:35:06,390 --> 00:35:13,390
frequent. So, I want that when I form the
code words for the source symbols 0 and 1
191
00:35:14,549 --> 00:35:20,839
the code word lengths corresponding to this
code word should be as small as possible.
192
00:35:20,839 --> 00:35:27,839
So, let us assume that for my source symbol
0 I assign the code word which is 0 and for
193
00:35:30,729 --> 00:35:37,729
my code source symbol 1, I assign the code
word as 1 0, obviously I cannot just assign
194
00:35:41,299 --> 00:35:47,140
1 because then I will not able to form the
instantaneous code.
195
00:35:47,140 --> 00:35:54,140
Now, for the remaining eight symbols what
is desired that I formed the code words which
196
00:35:59,170 --> 00:36:06,170
are of equal length. Now, I am interested
in finding out what is that minimum length,
197
00:36:06,839 --> 00:36:13,839
minimum equal length which I can use for forming
the code words for the remaining eight source
198
00:36:17,259 --> 00:36:24,259
symbols and I want to form an instantaneous
code. So, to answer the question how short
199
00:36:25,819 --> 00:36:32,819
it is possible for me to make the remaining
eight code words, assuming that all the eight
200
00:36:34,279 --> 00:36:40,759
code words are of the same length.
Now, to answer this question we can take the
201
00:36:40,759 --> 00:36:47,759
help of Kraft inequality, Kraft inequality
says that this condition should be always
202
00:36:49,519 --> 00:36:56,519
valid i is equal to 1 to 9 less than equal
to 1, since we have made the assumption that
203
00:37:03,359 --> 00:37:10,359
the code word corresponding to symbol 0 is
of length 1, so l 0 is 1 and l 1 is 2 and
204
00:37:10,759 --> 00:37:17,759
l 2 l 3 up to l 9, the code word lengths for
this code words are equal, so let us say they
205
00:37:25,180 --> 00:37:32,180
are l.
Therefore, I get half plus 1 by 4 plus 8 times
206
00:37:34,210 --> 00:37:41,210
l less than equal to 1 and this simplifies
to l is greater than or equal to 5. So, what
207
00:37:44,640 --> 00:37:51,640
it says that if I want the remaining eight
code words to be of equal length and if the
208
00:37:52,729 --> 00:37:59,729
code has to be instantaneous, than I cannot
form a code which is of length less than 5.
209
00:38:01,539 --> 00:38:08,539
So, the minimum length I should have is 5
and it is easy to see that I can form such
210
00:38:10,150 --> 00:38:13,910
a code for this source.
211
00:38:13,910 --> 00:38:20,910
So, a binary code for decimal digits would
be. So, this would be the instantaneous code
212
00:39:14,059 --> 00:39:21,059
where all the code words corresponding to
a source symbol from S 2 to S 9 would be all
213
00:39:26,460 --> 00:39:33,460
equal length 5. Now, with this background
let us go ahead and try to prove the Kraft’s
214
00:39:42,450 --> 00:39:49,450
inequality, so the Kraft inequality their
two portions, one is a sufficient condition,
215
00:39:50,670 --> 00:39:57,670
another is the necessary. So, first let us
try to prove so for the sufficient condition,
216
00:39:59,249 --> 00:40:06,249
so first is sufficient condition.
217
00:40:08,269 --> 00:40:15,269
So, what sufficient condition says that, if
this is satisfied, if this is satisfied then
218
00:40:29,509 --> 00:40:36,509
I should be able to construct instantaneous
codes. So, let us assume that the required
219
00:40:48,440 --> 00:40:55,440
code word lengths are l 1, l 2 up to l q,
this code word lengths need not be the distinct.
220
00:41:03,729 --> 00:41:10,729
Now, it will be useful for us to consider
all the code words of the same length at one
221
00:41:13,499 --> 00:41:20,499
time. So, let me assume that I have n 1 code
words of length 1, I have n two code word
222
00:41:27,890 --> 00:41:34,890
of length 2. Now, let me assume that l is
the maximum length of a code word, so l is
223
00:41:52,400 --> 00:41:59,400
maximum of l i. So, if I assume this then
let me assume that n l corresponds to a number
224
00:42:08,729 --> 00:42:15,729
of code words of maximum length l.
225
00:42:16,460 --> 00:42:23,460
So, it means that summation of n i, i is equal
to1to l is obviously equal to number of code
226
00:42:31,349 --> 00:42:38,349
words which I can have, the number of code
words will be equal to q and I have this condition.
227
00:42:45,289 --> 00:42:52,289
Now, using this equation we can re-write this
equation which corresponds to the Kraft’s
228
00:43:01,729 --> 00:43:08,729
inequality, I assume that this is true. So,
what it means that there will be in this inequality
229
00:43:11,619 --> 00:43:18,619
based on this relationship here, I will have
l 1 terms corresponding to r minus 1 and I
230
00:43:18,930 --> 00:43:25,930
will have n 2 terms corresponding to r minus
2 and I will have n l terms corresponding
231
00:43:26,710 --> 00:43:33,710
to r minus n. So, based on this I can write
this equation out here as, so given this and
232
00:43:53,289 --> 00:44:00,289
assuming that the notations we get this relationship.
Now, if you multiply this both left hand side
233
00:44:00,430 --> 00:44:07,430
and right hand side by r l what I get is,
now this forms the basis for a further discussion.
234
00:44:27,549 --> 00:44:34,549
Now, let us write out this equation and rearrange
terms, so if you write out this expression
235
00:44:36,989 --> 00:44:43,989
and rearrange terms we will get n l less than
equal to r l minus n 1 r l minus 1 minus n
236
00:44:55,039 --> 00:45:02,039
2 r l minus 2 minus n l minus 1 r, so let
me say that this is equation number 1 a. Now,
237
00:45:14,009 --> 00:45:21,009
dropping the term on the left hand side and
dividing by r I will get as n l minus 1 is
238
00:45:29,489 --> 00:45:36,489
less than equal to r l minus 1 minus n 1r
l minus 2 minus n 2 r l minus 3 minus n l
239
00:45:47,190 --> 00:45:54,190
minus 2 r. So, to drop the term on the left
hand side bring this term out here and then
240
00:45:58,489 --> 00:46:05,489
divide by r, this is what we will get let
us call this equation 1 b. So, this way if
241
00:46:05,489 --> 00:46:12,489
you proceed I will get n 3 less than equal
to r cube minus l 1 r square minus n 2 r this
242
00:46:17,630 --> 00:46:24,630
is 1 c and n 2 less than equal to r square
minus n 1 r, and finally I will get n 1 less
243
00:46:39,259 --> 00:46:46,259
than equal to r.
Now, this set of inequality is the key to
244
00:46:48,219 --> 00:46:55,219
the construction of the code we seek; now
what this inequality says that we are required
245
00:46:56,950 --> 00:47:03,950
to form n 1 code words of length 1. Now, if
we assume that our code alphabet is of size
246
00:47:07,589 --> 00:47:14,589
r, then there are r possible such code words
that we may form using an r symbol code alphabet
247
00:47:20,009 --> 00:47:27,009
and since the requirement is that n 1 should
be less than r, we can select this n 1 code
248
00:47:28,979 --> 00:47:35,979
symbols of length 1 arbitrary. So, if you
do that then we are then left with r minus
249
00:47:39,239 --> 00:47:46,239
n 1 permissible prefixes of length 1 namely
just those prefixes which were not used as
250
00:47:54,249 --> 00:48:01,249
code words. So, now these are the prefixes
which we can use, now by adding one symbol
251
00:48:09,219 --> 00:48:16,219
to this prefixes we will get code words of
length 2.
252
00:48:21,680 --> 00:48:28,680
So, the number of code words which we can
get of length 2 would be equal to this, r
253
00:48:33,229 --> 00:48:40,229
minus n 1 r and that is equal to r square
minus n 1 r. So, these are the code words
254
00:48:41,599 --> 00:48:48,589
of length two which we can form. Now, if you
look at this equation out here, what it says
255
00:48:48,589 --> 00:48:55,589
that the number of code words of length 2
required is always less than equal to r square
256
00:48:56,700 --> 00:49:00,209
minus n 1 r and we have this available with
us.
257
00:49:00,209 --> 00:49:07,209
So, as before we choose our n 2 code words
arbitrary from among these choices, so if
258
00:49:08,519 --> 00:49:15,519
you choose that the number of prefixes which
will be of length 2, which will be left out
259
00:49:15,719 --> 00:49:22,719
would be r square minus n 1 r minus n 2. So,
this will be the un used prefixes of length
260
00:49:28,779 --> 00:49:35,779
2, now using this unused prefixes of length
2, if we add one more code symbol we can form
261
00:49:42,900 --> 00:49:49,900
code words of length 3 and that would be given
by r q minus n 1 r square minus n 2 r.
262
00:49:59,239 --> 00:50:06,239
So, this are the permissible prefixes of length
3, again if you look at this equation we are
263
00:50:08,119 --> 00:50:15,119
assured that we shall not need no more than
this number and we select our code words of
264
00:50:17,869 --> 00:50:24,869
length 3 arbitrary from among this. Now, we
may proceed in this fashion until we have
265
00:50:33,839 --> 00:50:40,839
formed all the code words for a code and all
this equations assure us that at each stage
266
00:50:46,359 --> 00:50:53,359
we have enough prefixes left. So, based
on this we can always form an instantaneous
267
00:50:56,219 --> 00:51:03,219
code, so we have proved that if this condition
is satisfied, then it is possible for us to
268
00:51:04,650 --> 00:51:11,650
form an instantaneous code, so this is the
sufficient part of the Kraft inequality. Now,
269
00:51:12,180 --> 00:51:19,089
next time will try to prove the necessary
part of the Kraft equality, it is not very
270
00:51:19,089 --> 00:51:27,039
difficult we have to just reverse our arguments
and will have a look at the little more in
271
00:51:27,039 --> 00:51:28,909
depth in the next class.