1
00:00:01,170 --> 00:00:05,410
The last lecture in this unit, what we are
going to do is, actually look at some examples
2
00:00:05,410 --> 00:00:10,710
of algorithms and see how to compute their
upper bounds.
3
00:00:10,710 --> 00:00:17,390
So, we will look at two basic classes of algorithms
in this unit, in this lecture. So, we will
4
00:00:17,390 --> 00:00:21,990
look at some iterative examples and some recursive
examples. So, an iterative example will basically
5
00:00:21,990 --> 00:00:26,380
be something where there is a loop and a recursive
program of course, will be something where
6
00:00:26,380 --> 00:00:29,810
you have to solve a smaller problem before
you can solve the larger problem. So, you
7
00:00:29,810 --> 00:00:33,950
have to recursively apply the same algorithm
with smaller input.
8
00:00:33,950 --> 00:00:40,129
So, our first example is a very standard problem
which you must have done in your basic programming
9
00:00:40,129 --> 00:00:46,409
course. Suppose, we want to find the maximum
element in an array a. So, what we do is we
10
00:00:46,409 --> 00:00:52,670
initially assume that the maximum value is
the first value and then we scan the rest
11
00:00:52,670 --> 00:00:57,889
of the array and wherever we see a value which
is bigger than the current maximum value,
12
00:00:57,889 --> 00:01:01,809
maxval we replace it.
And at the end of this scan we return that
13
00:01:01,809 --> 00:01:06,689
value of maxval that we have found which should
be the largest value that we saw in the entire
14
00:01:06,689 --> 00:01:10,590
value. Now, remember that we said that if
we have two phases in this case, we have one
15
00:01:10,590 --> 00:01:15,710
phase where we do an initialization, we have
three phases actually we do a loop and then
16
00:01:15,710 --> 00:01:21,670
we do a return, it is enough to look at the
bottle neck phase, we said that if we have
17
00:01:21,670 --> 00:01:28,700
two parts f 1 and f 2 then the order of magnitude
of f 1 plus f 2 is the maximum of the order
18
00:01:28,700 --> 00:01:32,990
of magnitudes of f 1 and f 2.
So, in this case it is clear that this loop
19
00:01:32,990 --> 00:01:36,990
is what is going to take the most amount of
time. So, it is enough to analyze complexity
20
00:01:36,990 --> 00:01:43,159
of this loop. So, now, this loop takes exactly
n minus 1 steps. So, the worst case, any input
21
00:01:43,159 --> 00:01:47,320
is a worst case, because we must go from beginning
to end in order to find the maximum value,
22
00:01:47,320 --> 00:01:50,030
we cannot assume anything about where the
maximum value lies.
23
00:01:50,030 --> 00:01:56,590
Now, when we are scanning the loop in every
iteration we do at least one step. So, this
24
00:01:56,590 --> 00:02:03,869
is the comparison, one basic operation and
this may or may not happen. So, the assignment
25
00:02:03,869 --> 00:02:09,830
happens, if we find a new value A i which
is bigger than maxval. But, since we are ignoring
26
00:02:09,830 --> 00:02:17,629
constants we can treat this as some c operations,
some constant number of operations per iterations.
27
00:02:17,629 --> 00:02:26,629
So, we have some c times n minus 1 basic operations
and if we ignore the c and we ignore this
28
00:02:26,629 --> 00:02:33,920
minus 1, overall this algorithm is linear,
it takes order n time.
29
00:02:33,920 --> 00:02:42,290
So, let us now move on to an example in which
we have two nested loops. So, supposing we
30
00:02:42,290 --> 00:02:47,819
are trying to find whether or not an array
has all distinct values that is no two values
31
00:02:47,819 --> 00:02:54,120
in the array A are the same. So, what we will
do is, we will take this array A and then
32
00:02:54,120 --> 00:02:59,409
we will compare every A i and every A j and
if I ever find an A i equal to A j, then I
33
00:02:59,409 --> 00:03:04,790
will return false. If I find no such A i and
A j, then I will not return false I would
34
00:03:04,790 --> 00:03:09,129
return true.
Now, the point is in order to optimize this
35
00:03:09,129 --> 00:03:16,310
if I am at position i, then I will only look
at elements to its right. So, I will start
36
00:03:16,310 --> 00:03:22,010
with i plus 1 and go to n minus 1 and this
will be my range for j. So, in order to not
37
00:03:22,010 --> 00:03:26,000
compare A i, A j and then A j, A i just to
avoid this duplicate thing what we have written
38
00:03:26,000 --> 00:03:31,299
is for i equal to 0 to n minus 1. So, as I
look at each element j equal to i plus 1 to
39
00:03:31,299 --> 00:03:34,900
n minus 1 to its right, check if A i is equal
A j.
40
00:03:34,900 --> 00:03:40,360
So, now if I look at the number of times this
actually executes, then when i is equal to
41
00:03:40,360 --> 00:03:48,540
0, j varies from 1 to n minus 1. So, there
are n minus 1 steps, when i is equal to 1
42
00:03:48,540 --> 00:03:54,909
there are n minus 2 steps and so, on. So,
as I go down when i is equal to n minus 1,
43
00:03:54,909 --> 00:04:02,209
there will be 1 step, when i is equal to n
minus 2 there will be one step. When an I
44
00:04:02,209 --> 00:04:06,430
is equal to n minus 1 the outer loop will
terminate, but the inner would not run at
45
00:04:06,430 --> 00:04:09,909
all, because we will go from n to n minus
1.
46
00:04:09,909 --> 00:04:20,550
So, overall what we are doing is we are doing
0 plus 1, plus 2, plus n minus 2, plus n minus
47
00:04:20,550 --> 00:04:26,250
1 steps. So, this is a familiar summation,
the summation of i is equal to 1 to n minus
48
00:04:26,250 --> 00:04:35,670
1 of i and this you should know is n minus
1 into n, this is a very familiar recurrence
49
00:04:35,670 --> 00:04:40,170
and this we have already seen actually, this
is O n square. So, we just ignore constants
50
00:04:40,170 --> 00:04:44,900
n square by 2 minus n by 2 is O n square actually
we showed that this is theta n square, but
51
00:04:44,900 --> 00:04:47,140
for this moment we are only looking at upper
bounds.
52
00:04:47,140 --> 00:04:52,010
So, let us say this algorithms is O n square.
So, it is not a trivial, O n square in the
53
00:04:52,010 --> 00:04:57,590
sense is not two nested loops of equal size,
it is not i equal to 0 to n, j is equal to
54
00:04:57,590 --> 00:05:02,190
0 to n is i equal to 0 to n minus 1 and j
equal i plus 1 to n minus 1. But, still this
55
00:05:02,190 --> 00:05:06,990
summation 1, 2, 3, 4 up to n is O n square
and this is something we will see often. So,
56
00:05:06,990 --> 00:05:10,000
it is useful to remember this.
57
00:05:10,000 --> 00:05:15,940
So, this is another example of a nested loops
and this is one which has 3 nested loops.
58
00:05:15,940 --> 00:05:20,030
Now, here what we are trying to do is, we
are trying to multiply 2 square matrices A
59
00:05:20,030 --> 00:05:27,220
and B. So, we have two matrices A and B and
we are trying to compute the product C. Now,
60
00:05:27,220 --> 00:05:33,590
in this product C, if I want the i, j’th
entry, then what I do is that I look at row
61
00:05:33,590 --> 00:05:41,670
i in the first matrix, column j in the second
matrix and then I have to pair wise I have
62
00:05:41,670 --> 00:05:45,040
to do the first entry here in this row I do
the first entry in that column I would have
63
00:05:45,040 --> 00:05:49,130
to multiply those two, then I have to multiply
the second entry and so on and then I have
64
00:05:49,130 --> 00:05:51,220
to multiply the last entry and then I have
to add that up.
65
00:05:51,220 --> 00:05:56,940
So, that is what this program is saying. So,
this is for each row for i equal 0 to n minus
66
00:05:56,940 --> 00:06:01,550
1, then for each column j equal to 0 to n
minus 1. So, this is going through all possible
67
00:06:01,550 --> 00:06:07,900
entries C i j. Now, I am saying that for this
new entry I start by assuming C i j is 0 and
68
00:06:07,900 --> 00:06:13,830
then I run through this row k equal to 0 to
n minus 1, I look at A i k that is the kth
69
00:06:13,830 --> 00:06:18,140
element in the row, B k j the kth element
in this column, multiply them and add it to
70
00:06:18,140 --> 00:06:23,770
C i j. So, this is a loop outer loop of size
n, this is another loop, inner loop of size
71
00:06:23,770 --> 00:06:30,280
n and the inner most loop of size n and this
in order n cube. So, this is a natural example
72
00:06:30,280 --> 00:06:32,620
of an n cube algorithm.
73
00:06:32,620 --> 00:06:40,170
So, our final iterative example is one to
find the number of bits in the binary representation
74
00:06:40,170 --> 00:06:48,520
of n. So, this is just the same as dividing
n by 2 until we reach 1 or 0. So, let us assume
75
00:06:48,520 --> 00:06:53,870
that n is a non-negative number. So, n is
0 or 1. So, we assume by saying that the number
76
00:06:53,870 --> 00:06:59,150
of bits is at least 1 and then so long as
we have a number which is bigger than 1, we
77
00:06:59,150 --> 00:07:06,410
will add one more to the count number of digits
and then this is a short form for integer
78
00:07:06,410 --> 00:07:14,690
division. So, we will replace n by n by 2.
So, for instant supposing we start with a
79
00:07:14,690 --> 00:07:23,080
number like 9, then we will start with count
is equal to 1, because... While n is bigger
80
00:07:23,080 --> 00:07:29,450
than 2, I will divide by 2 and add 1 to count.
So, I will replace count make it 2, now make
81
00:07:29,450 --> 00:07:34,790
this 4, then I will say that it is still greater
than 1. So, I will make this 3 and I will
82
00:07:34,790 --> 00:07:40,240
make this 4 by 2, then I will say this is
still bigger than 1. So, I will make this
83
00:07:40,240 --> 00:07:45,220
1, then I make this 4. So, now, I have count
equal to 4 and n equal to 1. So, this loop
84
00:07:45,220 --> 00:07:50,640
exits and I return count. So, it says that
it requires 4 bits to represent number 9 which
85
00:07:50,640 --> 00:07:57,320
is correct, because the number 9 in binary
it is 1001. So, now what is the complexity
86
00:07:57,320 --> 00:08:03,850
of this loop? Well, how many times does this
execute? Well, it will execute as many times
87
00:08:03,850 --> 00:08:11,930
as it takes for n to come down from its value
to 1. So, I want n, n by 2 n by 4 etcetera
88
00:08:11,930 --> 00:08:18,520
to come down to 1.
So, how many times should I divide n by 2
89
00:08:18,520 --> 00:08:27,650
to reach 1 and this is the same as going backwards,
how many times should I multiply 1 by 2 to
90
00:08:27,650 --> 00:08:34,090
reach n. So, dividing n by 2 repeatedly to
reach 1 is the same as multiplying 1 by 2
91
00:08:34,090 --> 00:08:40,419
repeatedly to reach n and this is nothing
but, the definition of the log, what power
92
00:08:40,419 --> 00:08:48,569
of 2 reaches n. So, this iterative loop actually
though does not decrement by 1, decrements
93
00:08:48,569 --> 00:08:54,939
by halving n each time, we can still calculate
it explicitly as requiring log to the base
94
00:08:54,939 --> 00:08:59,670
2 n steps.
95
00:08:59,670 --> 00:09:06,290
So, we have seen iterative examples of linear
time, quadratic time, cubic time, that is
96
00:09:06,290 --> 00:09:11,810
n, n squared, n cube and also a linear example
with log n time. So, now let us look at one
97
00:09:11,810 --> 00:09:16,649
recursive example to see, how we would try
to do this when we have a recursive solution.
98
00:09:16,649 --> 00:09:21,740
So, we would not look at a formal algorithm,
but rather an informal puzzle. So, this is
99
00:09:21,740 --> 00:09:26,460
a well known Towers of Hanoi. So, in the towers
of Hanoi puzzle we have as we see in this
100
00:09:26,460 --> 00:09:32,269
picture here, we have 3 wooden pegs which
we will call for the moment A, B and C.
101
00:09:32,269 --> 00:09:41,370
So, we have pegs A, B and C and our goal is
to move these n disks from A to B. So, the
102
00:09:41,370 --> 00:09:44,881
thing that we are not allowed to do is to
put a larger disk on a smaller disk. So, if
103
00:09:44,881 --> 00:09:50,480
we take the small disk and move it here. So,
we move the first disk here, then we must
104
00:09:50,480 --> 00:09:54,269
take the second disk and move it there, because
we cannot put the second disk on top of the
105
00:09:54,269 --> 00:09:58,540
first disk. So, the goal is to do this in
an effective way.
106
00:09:58,540 --> 00:10:04,879
So, the actual goal is to move everything
from A to B and this is our intermediate thing,
107
00:10:04,879 --> 00:10:08,680
because as we saw, we move the first disk
from A to B we are struck, we cannot move
108
00:10:08,680 --> 00:10:13,570
anything else onto B. So, we must use C as
a kind of transit peg or a temporary auxiliary
109
00:10:13,570 --> 00:10:20,439
peg in order to do this job. So, if you have
not seen this problem before, you might want
110
00:10:20,439 --> 00:10:24,490
to think about it in your spare time, but
this is a very classical puzzle and it has
111
00:10:24,490 --> 00:10:26,920
a very standard recursive solution.
112
00:10:26,920 --> 00:10:31,240
And the standard recursive solution is the
following that you first assume that you know
113
00:10:31,240 --> 00:10:36,290
how to solve the problem for n minus 1 disks.
So, at this moment you want to move n disks
114
00:10:36,290 --> 00:10:42,740
from A to B. So, what you do is you first
move n minus 1 disks. So, you have on A only
115
00:10:42,740 --> 00:10:50,290
the bottom disk left and you have now B empty
and you have moved all the other n minus 1
116
00:10:50,290 --> 00:10:56,610
disks to C. So, there are now n minus 1 disks
here. So, you have assumed that you can do
117
00:10:56,610 --> 00:11:02,730
this using B as my transit peg.
So, now I move things from A to C, now what
118
00:11:02,730 --> 00:11:09,639
I do, I move this disk here. So, I now have
disk here and I no longer have anything there.
119
00:11:09,639 --> 00:11:13,690
So, now, I have one biggest disk on B. So,
I can put anything on it and I have n minus
120
00:11:13,690 --> 00:11:17,680
1 disks on C. So, what I do is I apply the
same algorithm for n minus 1 to move things
121
00:11:17,680 --> 00:11:20,499
from here to here using now A as my transit
pegs.
122
00:11:20,499 --> 00:11:25,329
So, this is the recursive way to solve the
problem, you move n minus 1 disks from A to
123
00:11:25,329 --> 00:11:30,600
C, move the biggest disk from A to B and then
move n minus 1 disks, back from C to B. So,
124
00:11:30,600 --> 00:11:37,130
the question we want to ask is, how many times
do we move disks in this procedure?
125
00:11:37,130 --> 00:11:45,420
So, supposing we write M of n to indicate
the number of moves we need to transfer n
126
00:11:45,420 --> 00:11:51,089
disks from one peg to another peg. So, what
we have seen is that in order to transfer
127
00:11:51,089 --> 00:11:57,319
n disks, we first transfer n minus 1 disks
from A to C, then we transfer one disk from
128
00:11:57,319 --> 00:12:04,490
A to B and then n minus 1 disks back from
C to B. So, it is M of n minus 1, this is
129
00:12:04,490 --> 00:12:08,910
to transfer n minus 1 disks plus 1 for that
1 disk and then M of n minus 1. So, this I
130
00:12:08,910 --> 00:12:14,290
can simplify as 2 times M of n minus 1 plus
1.
131
00:12:14,290 --> 00:12:20,959
So, M of n in general is 2 times M of n minus
1 plus 1 and if we have only one disk to transfer,
132
00:12:20,959 --> 00:12:25,529
then there is no problem we can do it directly
in one step. So, M of 1 where n is equal 1
133
00:12:25,529 --> 00:12:33,360
is 1. So, this kind of expression of describing
M n recursively in terms of smaller values
134
00:12:33,360 --> 00:12:40,970
of capital M, is called a recurrence. So,
we have a recursive expression for M n, now
135
00:12:40,970 --> 00:12:49,959
we have to solve this. So, the way we are
going to solve this is to use a notion of
136
00:12:49,959 --> 00:12:55,459
repeated substitution, we are going to repeatedly
use the same rule to simplify this expression,
137
00:12:55,459 --> 00:12:59,529
until we reach everything in terms of M 1
and then we can plug in the value.
138
00:12:59,529 --> 00:13:04,880
So, we start by the basic expression. So,
M of n is 2 times M n minus 1 plus 1, now
139
00:13:04,880 --> 00:13:10,399
what we do is we substitute for M n minus
1 the same expression in terms that n minus
140
00:13:10,399 --> 00:13:18,810
2. So, M n minus 1 by the same expression,
is 2 times M n minus 2 plus 1, because in
141
00:13:18,810 --> 00:13:25,839
general for any M we have M n is 2 times M
of n minus 1 plus 1. So, this is a general
142
00:13:25,839 --> 00:13:29,779
expression. So, we are taking...
So, we do this and we simplify it, we get
143
00:13:29,779 --> 00:13:36,410
2 times 2. So, we get 2 square coming from
this M n minus 2 and then we take 2 times
144
00:13:36,410 --> 00:13:42,610
1 that gives us this 2 plus 1. So, we have
just rewritten this as 2 square M n minus
145
00:13:42,610 --> 00:13:51,569
2. Now, again if you take this expression
M n minus 2 that becomes 2 times M n minus
146
00:13:51,569 --> 00:13:57,769
3 plus 1 and then this 2 square plus 1 is,
this 2 square remember is 4. So, this I get
147
00:13:57,769 --> 00:14:06,199
4 inside and 2 squared times 2, is 2 cubes.
So, I get 2 cube M n minus 3 plus 2 square
148
00:14:06,199 --> 00:14:13,029
plus 2 plus 1.
Now, you can see that if I do this k times
149
00:14:13,029 --> 00:14:17,509
I will have 2 to the k M of n minus k. Remember,
everywhere I have this and I have this, this
150
00:14:17,509 --> 00:14:22,449
is the same number and this is 1 plus 2 plus
4, next time will be 1 plus 2 plus 4 plus
151
00:14:22,449 --> 00:14:28,949
8. So, this is actually 2 to the k minus 1
to this is nothing but, 2 cube minus 1 it
152
00:14:28,949 --> 00:14:35,459
is nothing but, 2 squared minus 1. So, in
general after k steps I have this, now when
153
00:14:35,459 --> 00:14:44,949
I do this n minus 1 times then n minus n minus
1 is nothing but, 1 n minus n plus 1.
154
00:14:44,949 --> 00:14:52,459
So, if I do this n minus 1 times k is n minus
1 then n minus k becomes 1 and this n minus
155
00:14:52,459 --> 00:14:56,980
1. So, since n minus n minus 1 is 1 I can
just omit it from this thing. So, I have 2
156
00:14:56,980 --> 00:15:02,300
to the n minus 1 plus 2 to the n minus 1 minus
1. But this is nothing but, 2 times 2 to the
157
00:15:02,300 --> 00:15:11,759
n minus 1 which is 2 to the n. So, I can combine
these as 2 to the n. So, therefore, this gives
158
00:15:11,759 --> 00:15:16,350
us by this repeated expansion, substitution
whatever you would like to call it.
159
00:15:16,350 --> 00:15:21,720
We have that M n minus.. M of n is 2 to the
n minus 1 in other words, it takes an exponential
160
00:15:21,720 --> 00:15:28,869
number of steps in order to solve this puzzle.
So, there is a very famous story by someone,
161
00:15:28,869 --> 00:15:35,519
which talks about this some temple word these
pegs are there and it has 64 such disks and
162
00:15:35,519 --> 00:15:40,259
it says that the world will come to an end
when the 64 disks are transfered. So, you
163
00:15:40,259 --> 00:15:45,740
can thing about how much time it will take
to transfer 2 to the 64 disks, in order to
164
00:15:45,740 --> 00:15:50,329
solve the puzzle with 64 disks.
Remember, we said that 2 to the 30 is about
165
00:15:50,329 --> 00:15:54,512
1 billion. So, this is an enormous amount
of time I do not think you really need to
166
00:15:54,512 --> 00:15:57,009
worry about this as a serious problem if that
is indeed the case.
167
00:15:57,009 --> 00:16:05,619
So, to summarize we have looked at some examples
just to illustrate the flavor of how we apply
168
00:16:05,619 --> 00:16:11,250
the concepts we have studied in terms of big
O in an actual algorithm, how do we look at
169
00:16:11,250 --> 00:16:17,029
an algorithm and actually extract its complexity.
So, for an iterative program basically focus
170
00:16:17,029 --> 00:16:21,249
on the loops, because the loops are what take
up the time and you have some times to be
171
00:16:21,249 --> 00:16:26,170
a bit clever about trying to understand, how
many times a loop executes, for recursive
172
00:16:26,170 --> 00:16:29,399
programs we saw only one example, we will
see more as we go along.
173
00:16:29,399 --> 00:16:33,660
But, the main idea is you express the time
complexity of the program as a recurrence,
174
00:16:33,660 --> 00:16:39,339
you write t of n the time taking for n steps,
in terms of a smaller value which is obtained
175
00:16:39,339 --> 00:16:45,100
from the recursive call. So, for the Hanoi
case we had n and n minus 1. So, in order
176
00:16:45,100 --> 00:16:49,660
to solve the problem for n disks we needed
to solve the problem twice for n minus 1 disks.
177
00:16:49,660 --> 00:16:53,790
We will of course, find examples which do
not fit any of these, it will not be a simple
178
00:16:53,790 --> 00:16:59,000
loop that we can calculate and then we will
have to be a little more careful about how
179
00:16:59,000 --> 00:17:04,709
we actually count the operations.
So, in a sense actually estimating the efficiency
180
00:17:04,709 --> 00:17:08,750
of an algorithm, it is really like accounting.
So, you are kind of keeping track of all the
181
00:17:08,750 --> 00:17:14,260
basic operations and you have to do a good
job of making sure that you do that to keep
182
00:17:14,260 --> 00:17:17,440
track of them in the best possible way so
that you get an accurate picture of the answer.