1
00:00:00,799 --> 00:00:05,680
So, this course is called design and analysis
of algorithms. So, design is something that
2
00:00:05,680 --> 00:00:10,180
is easier to understand. We have a problem,
we want to find an algorithm, so we are trying
3
00:00:10,180 --> 00:00:13,990
to design an algorithm. But what exactly does
analysis mean?
4
00:00:13,990 --> 00:00:20,439
So, analysis means trying to estimate how
efficient an algorithm is. Now, there are
5
00:00:20,439 --> 00:00:25,560
two fundamental parameters that are associated
with this. One is time, how long does the
6
00:00:25,560 --> 00:00:29,989
algorithm actually take to execute on a given
piece of hardware. Remember an algorithm will
7
00:00:29,989 --> 00:00:34,370
be executed as a program, so we will have
to write it in some programming language and
8
00:00:34,370 --> 00:00:38,820
then run it on some particular machine. And
when it runs, we have all sorts of intermediate
9
00:00:38,820 --> 00:00:43,780
variables that we need to keep track of in
order to compute the answer. So, how much
10
00:00:43,780 --> 00:00:47,090
space does it take? How much memory does it
require?
11
00:00:47,090 --> 00:00:55,239
Now, we will argue that, in this course at
least, we will focus on time rather than space.
12
00:00:55,239 --> 00:00:59,440
One reason for this is, time is a rather more
limiting parameter in terms of the hardware.
13
00:00:59,440 --> 00:01:04,860
It is not easy to take a computer and change
its speed. So, if we are running algorithms
14
00:01:04,860 --> 00:01:09,300
on a particular platform, then we are more
or less stuck with the performance that that
15
00:01:09,300 --> 00:01:13,980
platform can give us in term of speed.
Memory, on the other hand, is something, which
16
00:01:13,980 --> 00:01:20,370
is relatively more flexible. We can add a
memory card and increase the memory. And so
17
00:01:20,370 --> 00:01:27,210
in a sense, space is a more flexible requirement
and so we can think about it that way. But
18
00:01:27,210 --> 00:01:31,680
essentially, for this course we will be focusing
on time more than space.
19
00:01:31,680 --> 00:01:37,990
So, if you are looking at time, we have to
ask how we measure the running time. So, of
20
00:01:37,990 --> 00:01:41,640
course, we could run a program on a given
computer and report the answer in seconds
21
00:01:41,640 --> 00:01:47,430
or in milliseconds, but the problem with this
is that the running time of an algorithm measured
22
00:01:47,430 --> 00:01:51,200
in terms of a particular piece of hardware
will, of course, not be a robust measure.
23
00:01:51,200 --> 00:01:55,610
We run it on a different computer or we use
a different programming language, we might
24
00:01:55,610 --> 00:02:00,740
find, that the same algorithm takes different
amount of time. More importantly, if we are
25
00:02:00,740 --> 00:02:05,820
trying to compare two different algorithms,
then if we run one algorithm on one computer,
26
00:02:05,820 --> 00:02:11,670
run the other on another computer, we might
get a misleading comparison. So, instead of
27
00:02:11,670 --> 00:02:16,310
looking at the concrete running time in terms
of some units of time, it is better to do
28
00:02:16,310 --> 00:02:22,410
it in terms of the some abstracts units of
how many steps the algorithm takes. So, this
29
00:02:22,410 --> 00:02:28,920
means, that we have to decide what a notion
of a step is. A step is some kind of a basic
30
00:02:28,920 --> 00:02:34,870
operation, a simple one step operation, that
our algorithm performs and the notion of a
31
00:02:34,870 --> 00:02:38,080
step, of course, depends on what kind of language
we are using.
32
00:02:38,080 --> 00:02:41,670
If we are looking at a very low-level, at
an assembly language kind of thing, then the
33
00:02:41,670 --> 00:02:47,670
steps involves moving data from the main memory
to register, moving it back, doing some arithmetic
34
00:02:47,670 --> 00:02:53,480
operation within the CPU and so on. But typically,
we look at algorithms and we design them and
35
00:02:53,480 --> 00:02:58,480
we implement them at a higher level. We use
programming languages such as C, C plus plus
36
00:02:58,480 --> 00:03:03,580
or Java where we have variables, we assign
values to variables, we compute expressions
37
00:03:03,580 --> 00:03:10,170
and so on. So, for the most part we will look
at basic operations as what we would consider
38
00:03:10,170 --> 00:03:15,540
single statement or steps in a high-level
language. So, this could be an operation such
39
00:03:15,540 --> 00:03:21,950
as assigning a value of X equal to Y plus
1 or doing a comparison, if A less than B,
40
00:03:21,950 --> 00:03:26,400
then do. So, checking whether A is less than
B is one step for us.
41
00:03:26,400 --> 00:03:32,430
Now, we could look at slightly more elaborate
notions of steps. For example, we might assume,
42
00:03:32,430 --> 00:03:36,680
that we have a primitive operation to actually
exchange the values in two variables though
43
00:03:36,680 --> 00:03:41,630
we know, that to implement is we actually
have to go via a third temporary variable.
44
00:03:41,630 --> 00:03:45,450
What we will see is, that we will come up
with a notion, which is robust so that the
45
00:03:45,450 --> 00:03:50,010
actual notion of the basic operation is not
so relevant because we will be able to scale
46
00:03:50,010 --> 00:03:56,989
it up and down according to what notion we
choose to focus.
47
00:03:56,989 --> 00:04:01,850
The other important thing to realize, of course
is, that the algorithm will take a different
48
00:04:01,850 --> 00:04:07,190
amount of time depending on the size of the
problem that it is presented with. It is quite
49
00:04:07,190 --> 00:04:11,390
natural and obvious, that if we are trying
to sort an array, it will take longer to sort
50
00:04:11,390 --> 00:04:18,120
a large array than it will take to sort a
short array. So, we would like to represent
51
00:04:18,120 --> 00:04:20,930
the efficiency of an algorithm as a function
of its input size.
52
00:04:20,930 --> 00:04:27,470
So, if the input is of some size n, then it
will take time t of n where t will be function
53
00:04:27,470 --> 00:04:34,229
depending on the input n, where even this
is not immediately an obvious definition because
54
00:04:34,229 --> 00:04:38,770
not all inputs of size n are going to take
the same amount of time. Some inputs will
55
00:04:38,770 --> 00:04:43,860
take less time, some inputs will take more
time. So, what do we take? So, it will turn
56
00:04:43,860 --> 00:04:48,830
out, that the notion, that we will typically
look at is to look at all the inputs of size
57
00:04:48,830 --> 00:04:53,580
n and look at the worst possible one that
takes the longest time, right. So, this is
58
00:04:53,580 --> 00:04:58,009
called a worst case estimate, it is a pessimistic
estimate, but we will justify it as we go
59
00:04:58,009 --> 00:05:02,449
along. But this is what we mean.
Now, when we are looking at the time efficiency
60
00:05:02,449 --> 00:05:08,949
of an algorithm what we mean is, what is the
worst possible time it will take on inputs
61
00:05:08,949 --> 00:05:16,710
of size n and express this as a function of
n? So, before we formalize this, let us just
62
00:05:16,710 --> 00:05:22,789
look at a couple of examples and get a feel
for what efficiency means in terms of practical
63
00:05:22,789 --> 00:05:27,529
running time on the kinds of computers that
we have available to us.
64
00:05:27,529 --> 00:05:32,289
So, let us start with sorting, which is a
very basic step in many algorithms. So, we
65
00:05:32,289 --> 00:05:37,449
have an array with n elements and we would
like to arrange these elements, say in ascending
66
00:05:37,449 --> 00:05:43,839
or descending order, for further processing.
Now, a na•ve sorting algorithm would compare
67
00:05:43,839 --> 00:05:49,610
every pair of elements more or less and try
to determine how to reorder them. This would
68
00:05:49,610 --> 00:05:55,590
take time proportional to n square because
we are comparing all pairs of elements. On
69
00:05:55,590 --> 00:06:01,639
the other hand, we will see soon in this course,
that there are much cleverer sorting algorithms,
70
00:06:01,639 --> 00:06:07,419
which work in time proportional to n log n.
So, we can go from a naive algorithm which
71
00:06:07,419 --> 00:06:12,499
takes time n square to a better algorithm,
which takes time n log n.
72
00:06:12,499 --> 00:06:17,889
So, what we have really done is, we have taken
a factor of n and replaced it by factor of
73
00:06:17,889 --> 00:06:25,529
log n. So, this seems a relatively minor change.
So, can we see what the effect of this change
74
00:06:25,529 --> 00:06:31,889
is? So, one way to look at this is to actually
look at concrete running times. We said, that
75
00:06:31,889 --> 00:06:36,050
we will not look at running times as measuring
the efficiency of the algorithm, but of course,
76
00:06:36,050 --> 00:06:40,749
the efficiency of the algorithm does have
an impact on how long the program will take
77
00:06:40,749 --> 00:06:45,949
to execute, and therefore on how usable the
program is from our practical perspective.
78
00:06:45,949 --> 00:06:51,759
So, if you take a typical CPU that we find
on a desktop or a laptop, which we have these
79
00:06:51,759 --> 00:06:57,469
days, it can process up to about 10 to the
8 operations. These are the basic operations
80
00:06:57,469 --> 00:07:01,949
in a high-level languages like an assignment
or checking whether one value is less than
81
00:07:01,949 --> 00:07:07,300
another value, right. We can say, that it
takes about, it can do about 10 to the 8 operations.
82
00:07:07,300 --> 00:07:12,529
Now, this is an approximate number, but it
is a way, we need to get some handle on numbers
83
00:07:12,529 --> 00:07:17,930
so that we can do some quantitative comparisons
of these algorithms. So, it is a useful number
84
00:07:17,930 --> 00:07:21,879
to have for approximate calculations.
Now, one thing to remember is that this number
85
00:07:21,879 --> 00:07:26,289
is not changing now a days. We used to have
a situation where CPUs were speeding up every
86
00:07:26,289 --> 00:07:31,319
one and a half year, but now we have kind
of reached the physical limits of current
87
00:07:31,319 --> 00:07:36,439
technology. So, CPUs are not speeding up.
We are using different types of technologies
88
00:07:36,439 --> 00:07:41,409
to get around this, parallelizing, multicore
and so on. But essentially, the sequential
89
00:07:41,409 --> 00:07:46,539
speed of a CPU is stuck at about 10 to the
8 operations per second.
90
00:07:46,539 --> 00:07:53,500
So, now when we take a small input, supposing
we are just trying to rearrange our contact
91
00:07:53,500 --> 00:07:57,599
list on our mobile phone. We might have a
few hundred contacts, maybe a thousand contacts,
92
00:07:57,599 --> 00:08:05,879
maybe even a few thousand contacts. If you
try to sort a contact list, say by name, it
93
00:08:05,879 --> 00:08:09,879
really would not make much difference to us
whether we use n square or n log n algorithm.
94
00:08:09,879 --> 00:08:15,389
Both of them will work in a fraction of second
and before we know it, we will have the answer.
95
00:08:15,389 --> 00:08:20,999
But if we go to more non-trivial sizes, then
the difference becomes rather stark.
96
00:08:20,999 --> 00:08:27,259
So, consider the problem of compiling a sorted
list of all the mobile subscribers across
97
00:08:27,259 --> 00:08:33,459
the country. It so turns out that India has
about one billion, that is, 10 to the 9 mobile
98
00:08:33,459 --> 00:08:39,339
subscribers. This includes data cards, phones,
various things, but these are all people who
99
00:08:39,339 --> 00:08:44,660
are all registered with some telecom operator
and own a sim card. So, this is the number
100
00:08:44,660 --> 00:08:50,560
of sim cards, which are in active use in India
today. So, suppose we want to compile a list
101
00:08:50,560 --> 00:08:56,440
of all owners of sim cards in India in some
sorted fashion.
102
00:08:56,440 --> 00:09:01,529
Since we have 10 to the 9 subscribers, if
we run an n square algorithm, this will take
103
00:09:01,529 --> 00:09:07,980
10 to the 18 operations because 10 to the
9 square is 10 to the 18. Now, 10 to the 18
104
00:09:07,980 --> 00:09:12,959
operations per seconds, since we can do only
10 to the 8 operations in 1 second, will take
105
00:09:12,959 --> 00:09:19,670
10 to the 10 seconds. So, how much is 10 to
the 10 seconds? Well, it is about 2.8 million
106
00:09:19,670 --> 00:09:27,460
hours; it is about 115 thousand days, that
is, about 300 years. So, you can imagine,
107
00:09:27,460 --> 00:09:32,579
that if we really want to do this using an
n squared algorithm, it would really not be
108
00:09:32,579 --> 00:09:36,240
practical because it would take more than
our lifetime, more than several generations
109
00:09:36,240 --> 00:09:42,300
in fact, to compute this on the current hardware.
On the other hand, if we were to move to the
110
00:09:42,300 --> 00:09:48,569
smarter n log n algorithm, which we claim
we will find, then it turns out, that sorting
111
00:09:48,569 --> 00:09:54,259
this large number of own users takes only
3 times 10 to the 10 because the log to the
112
00:09:54,259 --> 00:10:01,029
base 2 of 10 to the 9 is 30. It is useful
to remember, that 2 to the 10 is 1000. So,
113
00:10:01,029 --> 00:10:08,259
log to the base 2 of 1000 is 10. Now, since
logs add 1000 times, 1000 is 10 to the 6,
114
00:10:08,259 --> 00:10:16,029
right, so the log of 10 to the 6 is 20, log
of 10 to the 9 is 30. So, 30 into 10 to the
115
00:10:16,029 --> 00:10:21,060
9 n log n is 3 times 10 to the 10. So, this
means, that it will take about 300 seconds.
116
00:10:21,060 --> 00:10:25,279
So, it will take about 5 minutes.
Now, 5 minutes is not a short time, you can
117
00:10:25,279 --> 00:10:29,449
go and have a tea and come back, but still
it will get done in a reasonably short amount
118
00:10:29,449 --> 00:10:33,810
of time, so that we can then work with this
useful information and go on as opposed to
119
00:10:33,810 --> 00:10:36,910
300 years, which is totally impractical.
120
00:10:36,910 --> 00:10:44,471
So, let us look at another example. So, supposing
we are playing a video game, right. So, this
121
00:10:44,471 --> 00:10:48,810
might be one of these action type games where
there are objects moving around the screen
122
00:10:48,810 --> 00:10:55,420
and we have to know, identify certain object,
shoot them down, capture them and whatever.
123
00:10:55,420 --> 00:11:01,790
So, let us assume that as part of the game
in order to compute the score, it has to periodically
124
00:11:01,790 --> 00:11:05,310
find out the closest pair of objects on the
screen.
125
00:11:05,310 --> 00:11:09,670
So, now, how do you find the closest pair
of objects among group of objects? Well, of
126
00:11:09,670 --> 00:11:14,160
course, you can take every pair of them, find
the distance between each pair and then take
127
00:11:14,160 --> 00:11:18,649
the smallest one, right. So, this will been
an n squared algorithm, right. So, you compute
128
00:11:18,649 --> 00:11:23,230
the distance between any two objects and then
after doing this for every pair you take the
129
00:11:23,230 --> 00:11:27,290
smallest value. Now, it turns out, that there
is a clever algorithm, again, which takes
130
00:11:27,290 --> 00:11:33,690
time n log n. So, what is this distinction
between n square and n log n in this context?
131
00:11:33,690 --> 00:11:41,379
Now, on a modern kind of gaming computer with
a large display it is not unreasonable to
132
00:11:41,379 --> 00:11:46,620
assume, that we could have a resolution of
say, 2500 by 1500 pixels. If we have one of
133
00:11:46,620 --> 00:11:51,970
these reasonable size 20 inch monitors, we
could easily get this kind of resolution.
134
00:11:51,970 --> 00:11:57,279
So, we will have about 3.75 millions points
on the screen. Now, we have some objects placed
135
00:11:57,279 --> 00:12:00,579
at some of these points.
So, let us assume that we have five lakh objects,
136
00:12:00,579 --> 00:12:06,319
five hundred thousand objects placed on the
screen. So, if you were to now compute the
137
00:12:06,319 --> 00:12:11,939
pair of objects among these five hundred thousand,
which are closest to each other and we use
138
00:12:11,939 --> 00:12:17,110
the na•ve n squared algorithm, then you
would expect to take 25 times 10 into the
139
00:12:17,110 --> 00:12:23,079
10 steps because that is 5 into 10 to the
5 whole square. 25 into 10 to the 10 is 2500
140
00:12:23,079 --> 00:12:27,279
seconds, which is around 40 minutes.
Now, if you are playing a game, an action
141
00:12:27,279 --> 00:12:33,040
game in which reflexes determine your score,
obviously, each update cannot take 40 minutes.
142
00:12:33,040 --> 00:12:37,230
That would not be an effective game. On the
other hand, you can easily check, that if
143
00:12:37,230 --> 00:12:41,699
you do n log n calculation for 5 into 10 to
the 5, you will take something like 10 to
144
00:12:41,699 --> 00:12:46,310
the 6 or 10 to the 7 seconds. So, this will
be a fraction of second, 1-10th or 1-100th
145
00:12:46,310 --> 00:12:51,300
of a second, which is well below your human
response time. So, it will be, essentially,
146
00:12:51,300 --> 00:12:55,690
instantaneous. So, we move from a game, which
is hopelessly slow to one in which it can
147
00:12:55,690 --> 00:12:59,560
really test your reflexes as a human.
148
00:12:59,560 --> 00:13:06,529
So, we have seen in these two examples, that
there is a real huge practical gulf between
149
00:13:06,529 --> 00:13:12,390
even something as close to each other as n
log n and n squared. The size of the problems,
150
00:13:12,390 --> 00:13:16,440
that we can tackle for n squared are much
smaller than the size of the problems we can
151
00:13:16,440 --> 00:13:25,079
tackle for n log n. So, when we look at these
functions of n, typically we will ignore constants.
152
00:13:25,079 --> 00:13:29,879
Now, this will partly be justified later on
by the fact, that we are not fixing the basic
153
00:13:29,879 --> 00:13:37,079
operation, but essentially by ignoring constants
we are looking at the overall function of...
154
00:13:37,079 --> 00:13:43,189
the efficiency as a function of n, in the,
as it increases, right. So, this is what we
155
00:13:43,189 --> 00:13:47,600
call asymptotic complexity, as n gets large
how does the function behave.
156
00:13:47,600 --> 00:13:53,709
So, for instance, supposing we have some function,
which grows like n squared with a large coefficient
157
00:13:53,709 --> 00:13:58,360
and something, which goes like n cube with
a coefficient of 1, initially it will look
158
00:13:58,360 --> 00:14:04,660
like n squared, say 5000 n squared is much
more than n cube, but very rapidly, say at
159
00:14:04,660 --> 00:14:11,210
5000, right. At 5000 we will have, both will
be 5000 cube and beyond 5000 the function
160
00:14:11,210 --> 00:14:14,660
n cube will grow faster than function 5000
n squared, right.
161
00:14:14,660 --> 00:14:19,570
So, there will be a point beyond which n cube
will overtake n square and after that it will
162
00:14:19,570 --> 00:14:23,629
rapidly pull away. So, this is what we would
typically like to measure. We would like to
163
00:14:23,629 --> 00:14:28,899
look at the functions as functions of n without
looking at the individual constants and we
164
00:14:28,899 --> 00:14:32,339
will look at this little more in detail.
165
00:14:32,339 --> 00:14:38,110
So, since we are only interested in orders
of magnitude, we can broadly classify functions
166
00:14:38,110 --> 00:14:42,649
in terms of what kind of functions they look
like, right. So, we could have functions,
167
00:14:42,649 --> 00:14:47,590
which are proportional to log n; functions,
which are proportional to n, n squared, n
168
00:14:47,590 --> 00:14:54,251
cube, n to the k; which is, so any n to a
fixed k is a polynomial. So, we can call these,
169
00:14:54,251 --> 00:15:03,949
all these functions, we will look at, as polynomial,
right. So, these are polynomial. And then
170
00:15:03,949 --> 00:15:09,519
we could have something, which is 2 to the
n. 2 to the n essentially, comprises of looking
171
00:15:09,519 --> 00:15:13,529
at all possible subsets. So, these are typically
the brute force algorithms where we look at
172
00:15:13,529 --> 00:15:18,430
every possibility and then determine the answer,
right. So, we have logarithmic polynomial
173
00:15:18,430 --> 00:15:23,990
and exponential functions. So, what do these
look like in terms of the numbers that we
174
00:15:23,990 --> 00:15:26,199
were talking about.
175
00:15:26,199 --> 00:15:32,480
So, if you look at this chart, the left column
varies the input size from 10 to the 1 to
176
00:15:32,480 --> 00:15:38,550
10 to the 10 and then each column after that
is a different type of complexity behaviour
177
00:15:38,550 --> 00:15:45,300
as a function, which ignores constants and
only looks at the magnitude. So, we have log
178
00:15:45,300 --> 00:15:50,610
n and then we have something, which is polynomial
n, n square, n cube. In between we have n
179
00:15:50,610 --> 00:15:55,800
log n we saw before and then we have these
two exponential functions, which are 2 to
180
00:15:55,800 --> 00:16:03,639
the n and n factorial, right. So, now, in
this we are trying to determine how efficiency
181
00:16:03,639 --> 00:16:09,420
determines practical useability.
So, now, if you look at this, we said, that
182
00:16:09,420 --> 00:16:14,490
we can do 10 to the 8 operations in 1 second.
So, maybe we are willing to work up to 10
183
00:16:14,490 --> 00:16:18,760
seconds. So, 10 to the 9 operation is about
the limit of what we would consider efficient.
184
00:16:18,760 --> 00:16:24,649
10 to the 10 operation would mean 100 second,
which would mean 2 minutes and may be 2 minutes
185
00:16:24,649 --> 00:16:27,910
is too long.
So, now clearly, if we are looking in the
186
00:16:27,910 --> 00:16:32,600
logarithmic scale, there is no problem. Up
to 10 to the 10, right, we can do everything
187
00:16:32,600 --> 00:16:39,889
in about 33 steps, which will take very, very
little time. Now, in a linear scale given,
188
00:16:39,889 --> 00:16:44,139
that we are expecting 10 to the 9 as our limit,
this becomes our limit, right. So, we can
189
00:16:44,139 --> 00:16:51,629
draw a line here and say, that below this
red line things are impossibly slow. n log
190
00:16:51,629 --> 00:16:58,209
n is only slightly worse than n. So, we can
do inputs of size 10 to the 8 because n log
191
00:16:58,209 --> 00:17:00,949
n of 10 to the 8 is something proportional
to 10 to the 9.
192
00:17:00,949 --> 00:17:06,160
Now, there is this huge gulf that we saw.
If we move from n log n to n squared, then
193
00:17:06,160 --> 00:17:10,810
the feasibility limit is somewhere between
10 to the 4 and 10 to the 5 because, for 10
194
00:17:10,810 --> 00:17:16,850
to the 5, we already reach time.. running
time of 10 to the 10, which is beyond what
195
00:17:16,850 --> 00:17:21,040
we want and now if we move to n cube then
the feasibility limit drops further. So, we
196
00:17:21,040 --> 00:17:26,390
cannot go beyond an input size of 1000 right.
So, we can see that there is a drastic drop
197
00:17:26,390 --> 00:17:30,790
and when we come to exponentials really beyond
some trivial things of the size 10 or 20,
198
00:17:30,790 --> 00:17:36,330
we cannot really do anything, anything, which
is even a 100 will be impossibly slow, right.
199
00:17:36,330 --> 00:17:45,340
So, we have these kind of, this sharp dividing
line of feasibility right. And so we can see
200
00:17:45,340 --> 00:17:50,880
that it is important to get efficient algorithm
because if we have inefficient algorithms,
201
00:17:50,880 --> 00:17:56,260
even if we think that they work correctly,
we will not be able to solve problems of any
202
00:17:56,260 --> 00:18:01,310
reasonable size right. When we are looking
at computational algorithms running on a PC,
203
00:18:01,310 --> 00:18:07,040
you would really expect our input sizes to
be 100s, 1000s and much more right. So, when
204
00:18:07,040 --> 00:18:11,730
we are sorting things, we really do expect
large volumes of data, if you want to you
205
00:18:11,730 --> 00:18:18,320
look at voters list, population data or data
coming out of say a biology experiment about
206
00:18:18,320 --> 00:18:22,660
genetic material, now these things are typically
very large volumes of data, so it is really
207
00:18:22,660 --> 00:18:27,920
important to get the best possible algorithm
to do these things otherwise the problem at
208
00:18:27,920 --> 00:18:29,400
hand will not be effectively solved.