1
00:00:17,510 --> 00:00:24,470
so in last class we were looking at matrix
conditioning how to classify whether a system
2
00:00:24,470 --> 00:00:32,570
of linear algebraic equations is well posed
problem or ill posed problem this boils down
3
00:00:32,570 --> 00:00:39,680
to looking at matrix a and we will come to
that i was just talking about a motivation
4
00:00:39,680 --> 00:00:45,460
to look at this problem i showed a simple
system in which reordering of calculations
5
00:00:45,460 --> 00:00:53,379
can change the results another system in which
inherently bad ill condition
6
00:00:53,379 --> 00:01:01,800
a small error on the right hand side can change
the solution drastically so the idea was to
7
00:01:01,800 --> 00:01:07,820
analyze errors of this type
8
00:01:07,820 --> 00:01:16,119
so we are looking at solutions of ax=b and
as i told you invariably when you solve a
9
00:01:16,119 --> 00:01:22,960
problem using computer you will never solve
this original problem we always solve a problem
10
00:01:22,960 --> 00:01:40,680
which is a+delta a x+delta x=b+delta b we
can never solve the original problem except
11
00:01:40,680 --> 00:01:48,770
some very very simple systems in general for
most of the problems i mean let me just give
12
00:01:48,770 --> 00:01:49,810
you a simple example
13
00:01:49,810 --> 00:02:04,630
you can let say i have this problem pi-e let
us say i want to solve this problem using
14
00:02:04,630 --> 00:02:13,840
computer okay i cannot represent pi exactly
i cannot represent e exactly actually for
15
00:02:13,840 --> 00:02:22,640
this particular problem you can find the exact
solution analytically what is the solution?
16
00:02:22,640 --> 00:02:30,459
you find a determinant and then cofactor inverse
okay so the true solution to this problem
17
00:02:30,459 --> 00:02:33,650
would be x1 x2
18
00:02:33,650 --> 00:02:44,880
can you tell me what is the determinant of
this? 1/pi cube+e square right and what is
19
00:02:44,880 --> 00:03:05,840
the cofactor? pi square e-e pi*root 2 e square
right correct this
20
00:03:05,840 --> 00:03:15,629
is the inverse matrix okay this matrix this
solution which you get here or this original
21
00:03:15,629 --> 00:03:23,329
problem can never be represented in a computer
because pi is not a rational number and when
22
00:03:23,329 --> 00:03:28,319
if you write a rational number approximations
will creep in because of finite procedure
23
00:03:28,319 --> 00:03:29,799
used in computing
24
00:03:29,799 --> 00:03:36,629
so this is the true solution i would say what
you get in the computer is the approximate
25
00:03:36,629 --> 00:03:44,739
solution of this problem okay and now the
real worry is how bad is the approximation
26
00:03:44,739 --> 00:03:51,540
okay when can computing make things wrong
okay? i will give you a simple example now
27
00:03:51,540 --> 00:03:56,319
which is little more involved this looks like
a very simple matrix i have given this in
28
00:03:56,319 --> 00:04:00,529
the notes okay
29
00:04:00,529 --> 00:04:31,590
this matrix is 10 7 8 7 7 5 6 5 8 6 10 9 7
5 9 10 the reason for giving all these examples
30
00:04:31,590 --> 00:04:38,730
is because unless you know motivation is clear
if i just do the raw theory it does not make
31
00:04:38,730 --> 00:04:53,280
sense so this is my a matrix okay i am choosing
x1 x2 x3 x4=1 1 1 1 and then if i choose this
32
00:04:53,280 --> 00:05:13,370
i get 32 23 33 and 31 this is a matrix this
is x this is the right hand side okay 1 1
33
00:05:13,370 --> 00:05:20,100
1 1 i have chosen it is a non-problem to you
if i ask you to solve for x we should exactly
34
00:05:20,100 --> 00:05:21,950
get 1 1 1 1 okay
35
00:05:21,950 --> 00:05:34,830
now what if i perturb this matrix a little
bit okay i will show you what happens if to
36
00:05:34,830 --> 00:06:00,690
this matrix i add +delta a i add delta a okay
now my delta a is going to be 0 0 01 02 008
37
00:06:00,690 --> 00:06:34,640
004 0 0 0 -002 -0110 and 001 0 and -002 instead
of taking this a matrix okay i am going to
38
00:06:34,640 --> 00:06:49,050
solve for a+delta a okay this is my a matrix
this is my original x okay and this is my
39
00:06:49,050 --> 00:06:50,050
right hand side
40
00:06:50,050 --> 00:06:59,910
i am keeping the right hand side same i am
changing the problem now to this problem i
41
00:06:59,910 --> 00:07:09,370
am changing a+delta a i do not know what will
be x and i am keeping the b same okay now
42
00:07:09,370 --> 00:07:16,810
if you see there my matrix delta a contains
very small perturbations compared to you know
43
00:07:16,810 --> 00:07:24,620
the elements 10 7 8 my perturbations are 01
02 001 002 small perturbations in a matrix
44
00:07:24,620 --> 00:07:28,900
okay you might imagine well if these kinds
of small errors will not change the solution
45
00:07:28,900 --> 00:07:30,290
drastically
46
00:07:30,290 --> 00:07:36,270
you would expect the solution what you would
expect? x+delta x to be typically small perturbation
47
00:07:36,270 --> 00:07:43,050
in a you know my x should change a little
bit it does not happen here
48
00:07:43,050 --> 00:08:06,889
if you solve this problem now okay x turns
out to be -81 137 -39 22 with such a small
49
00:08:06,889 --> 00:08:17,780
perturbation in a matrix my x changes from
1 1 1 1 to this vector okay a small error
50
00:08:17,780 --> 00:08:27,490
a tiny error okay in a matrix can cause solution
to change so drastically that you cannot even
51
00:08:27,490 --> 00:08:33,229
recognize okay there is something fundamentally
wrong about that matrix you make a small error
52
00:08:33,229 --> 00:08:36,630
in the representation your solution can be
substantially different
53
00:08:36,630 --> 00:08:45,200
it will not even resemble you know this original
x sorry this is x+delta x should write x+delta
54
00:08:45,200 --> 00:08:58,280
x so x+delta x is this and your x just compare
your x was 1 1 1 1 these 2 are significantly
55
00:08:58,280 --> 00:09:09,310
different vectors okay well i will slightly
make a difference okay instead of solving
56
00:09:09,310 --> 00:09:11,110
this problem i will solve this problem
57
00:09:11,110 --> 00:09:24,430
a+x+delta x b+delta b okay i am going to introduce
a slight perturbation on the right hand side
58
00:09:24,430 --> 00:09:28,670
a matrix nothing is changed you have represented
a matrix correctly okay
59
00:09:28,670 --> 00:10:05,340
my right hand side it changed to my b+delta
b now is 3199 2301 3299 3102 okay look at
60
00:10:05,340 --> 00:10:17,060
the original b vector it is 32 i perturb by
-001 okay this side i have perturb by +001
61
00:10:17,060 --> 00:10:25,170
this is -001 this is 002 very very small perturbations
on the right hand side okay
62
00:10:25,170 --> 00:10:53,601
how does x change? if i introduce this perturbation
my x+delta x becomes 012 246 062 123 look
63
00:10:53,601 --> 00:11:01,790
at the solution where is 1 1 1 1 and where
is this solution or where is 1 1 1 1 and where
64
00:11:01,790 --> 00:11:07,090
is this solution these 2 solutions with slight
perturbation on the right hand side or a slight
65
00:11:07,090 --> 00:11:12,180
perturbation on the left hand side in the
a matrix or b matrix is changing your solution
66
00:11:12,180 --> 00:11:19,050
so drastically that you know there seems to
be something funny about this matrix
67
00:11:19,050 --> 00:11:24,480
you just change something a little bit make
a one small error your calculations are going
68
00:11:24,480 --> 00:11:32,160
haywire okay you would expect a small error
committed in a or b would result in a small
69
00:11:32,160 --> 00:11:39,260
error in a or in the solution x that is not
happening so this particular matrix seems
70
00:11:39,260 --> 00:11:45,670
to blow up even a small or tiny error in the
representation okay this is the background
71
00:11:45,670 --> 00:11:46,670
this is the motivation
72
00:11:46,670 --> 00:11:51,680
and then when you solve partial differential
equations or boundary value problems you have
73
00:11:51,680 --> 00:11:56,770
large matrices you know depending upon how
you discretize how you create grades or number
74
00:11:56,770 --> 00:12:02,940
of collocation points or whatever you know
method of least squares whatever you are using
75
00:12:02,940 --> 00:12:10,460
you have large number of points and you make
small errors there if those matrices are ill
76
00:12:10,460 --> 00:12:15,280
conditioned you can get answers which are
wrong even when you have a very good program
77
00:12:15,280 --> 00:12:22,020
nothing to do with how good your program is
written you know if you have best numerical
78
00:12:22,020 --> 00:12:30,440
package you can end up doing or getting upside
answers so now i want to come up with a method
79
00:12:30,440 --> 00:12:39,040
of you know analytical tool by which i can
say which matrix is good which matrix is bad
80
00:12:39,040 --> 00:12:45,550
and that is what is going to be my theme for
this particular lecture okay so let us look
81
00:12:45,550 --> 00:12:47,850
at this case first
82
00:12:47,850 --> 00:12:58,460
i think with this numerical example at least
you have motivation for why you are looking
83
00:12:58,460 --> 00:13:07,830
at this particular problem now i am going
to do these 2 special cases of this derivation
84
00:13:07,830 --> 00:13:11,370
because they will gives you insight it is
possible to do a more general derivation but
85
00:13:11,370 --> 00:13:18,540
i think this specific derivation gives more
insight so let me look at this case first
86
00:13:18,540 --> 00:13:25,720
okay i wanted to solve ax=b
87
00:13:25,720 --> 00:13:30,750
i was able to represent a perfectly there
was no problem there was some error committed
88
00:13:30,750 --> 00:13:37,040
on the right hand side okay b was represented
wrong that is why the solution became x+delta
89
00:13:37,040 --> 00:13:39,670
x okay
90
00:13:39,670 --> 00:13:47,220
so this is my original let us say x is the
true solution a is the true representation
91
00:13:47,220 --> 00:13:51,830
of a matrix b is the true representation of
b vector and this is what you ended up solving
92
00:13:51,830 --> 00:13:58,770
in the computer okay so this is my ii okay
93
00:13:58,770 --> 00:14:20,650
what will i get ii-i? so this is ax+a delta
x=b+delta b if i subtract i from ii i will
94
00:14:20,650 --> 00:14:34,170
get a delta x=delta b okay let us for the
time being assume that of course when you
95
00:14:34,170 --> 00:14:39,720
are solving for ax=b a is invertible if a
is not invertible you are solving a problem
96
00:14:39,720 --> 00:14:43,320
which does not have a solution or which may
have multiple solutions we does not have a
97
00:14:43,320 --> 00:14:49,420
unique solution let us assume that we have
a problem where a has a unique solution
98
00:14:49,420 --> 00:14:55,100
so i am not talking of a system which is just
mind you i am not talking of a system which
99
00:14:55,100 --> 00:15:05,800
is singular singularity is not a problem okay
it is not a singularity do not confuse singularity
100
00:15:05,800 --> 00:15:11,680
with ill-conditioning a singular system may
not have a solution or it may have multiple
101
00:15:11,680 --> 00:15:16,800
solutions okay singular system may not have
a solution or it may have multiple solutions
102
00:15:16,800 --> 00:15:20,690
depending upon whether b belongs to the column
space of a
103
00:15:20,690 --> 00:15:24,430
and what is the null space of a matrix and
so on so that is the different story this
104
00:15:24,430 --> 00:15:39,930
is not singularity this is something different
okay so now delta x=a inverse delta b okay
105
00:15:39,930 --> 00:15:52,580
now i am going to use properties of matrix
norm so norm delta x=norm a inverse delta
106
00:15:52,580 --> 00:16:05,430
b which is<=norm a inverse using this is basic
definition of matrix
107
00:16:05,430 --> 00:16:16,490
okay i am just using properties of matrix
norms so which means norm delta x change in
108
00:16:16,490 --> 00:16:29,920
solution due to change in the right hand side
okay this ratio is bounded by norm of a inverse
109
00:16:29,920 --> 00:16:40,890
okay well in general when you talk of delta
x and delta b these could be very very small
110
00:16:40,890 --> 00:16:41,890
numbers
111
00:16:41,890 --> 00:16:47,660
and then their ratios sometimes does not help
you to quantify everything we need to talk
112
00:16:47,660 --> 00:16:55,450
about relative change okay i would like to
know about delta x/x delta b/b if i change
113
00:16:55,450 --> 00:17:00,769
you know percentage error with respect to
the original solution so this inequality is
114
00:17:00,769 --> 00:17:13,370
not sufficient i need something more okay
so now i am going back to this first equation
115
00:17:13,370 --> 00:17:16,699
here
116
00:17:16,699 --> 00:17:38,870
ax=b so norm ax=norm b which is<=norm a norm
x norm ax 00:17:47,160
inequality which defines the matrix norm okay
i am just using the definition of induced
118
00:17:47,160 --> 00:18:06,910
matrix now okay so this particular inequality
gives me here we are talking about b norm
119
00:18:06,910 --> 00:18:26,390
b/norm x this quantity is always bounded by
norm a okay now this inequality i am going
120
00:18:26,390 --> 00:18:29,391
to combine with the earlier inequality
121
00:18:29,391 --> 00:18:37,190
see this is the positive number ratio of 2
norms right hand side is the positive number
122
00:18:37,190 --> 00:18:52,470
okay so now combining let us call this result
iv
123
00:18:52,470 --> 00:18:59,530
let us call this inequality as result iii
so if i combine iii and iv okay if i combine
124
00:18:59,530 --> 00:19:07,840
iii and iv i can just multiply the left hand
sides and right hand sides 2 positive numbers
125
00:19:07,840 --> 00:19:18,260
which are<2 other positive numbers so if i
combine those 2 i will get if i combine these
126
00:19:18,260 --> 00:19:28,350
2 inequalities i get this relationship okay
if i do a little bit of rearrangement you
127
00:19:28,350 --> 00:19:33,390
will see why i am doing this okay
128
00:19:33,390 --> 00:19:38,230
is everyone with me on this this inequality
okay? i am multiplying positive numbers on
129
00:19:38,230 --> 00:19:42,170
the left hand side i am multiplying positive
numbers on the right hand side 2 inequalities
130
00:19:42,170 --> 00:19:52,530
are combined to generate this inequality now
how do you derive something out of this?
131
00:19:52,530 --> 00:19:59,980
so this gives me this fundamental inequality
that relative change in the solution i am
132
00:19:59,980 --> 00:20:10,330
looking at sensitivity right relative change
in the solution due to relative change in
133
00:20:10,330 --> 00:20:24,280
the right hand side is bounded
by this number okay what does it mean? it
134
00:20:24,280 --> 00:20:36,170
means that maximum ratio of relative change
in x to relative change in b okay=norm of
135
00:20:36,170 --> 00:20:45,940
a*norm of a inverse what do you see here on
the right hand side neither x appears not
136
00:20:45,940 --> 00:20:49,660
b appears only matrix a appears okay
137
00:20:49,660 --> 00:20:57,340
so this you know if you have a slight error
in representation of b you know what is the
138
00:20:57,340 --> 00:21:03,490
maximum possible fractional error? this is
something like fractional error right norm
139
00:21:03,490 --> 00:21:08,990
delta x/norm x is something like fractional
error right so what is the maximum fractional
140
00:21:08,990 --> 00:21:14,900
error that you get in the solution? this is
bounded fundamentally by multiplication of
141
00:21:14,900 --> 00:21:20,169
2 quantities norm of a and norm of a inverse
okay
142
00:21:20,169 --> 00:21:29,570
this norm of a*norm of a inverse this is called
as condition number of a matrix how do you
143
00:21:29,570 --> 00:21:34,960
evaluate this? you could use 1-norm 2-norm
infinite norm whatever is convenient you can
144
00:21:34,960 --> 00:21:46,100
use that norm and find out this quantity well
finding out this for one or infinite norm
145
00:21:46,100 --> 00:21:52,210
has a problem because 1 or infinite norm would
require computation of a inverse and many
146
00:21:52,210 --> 00:21:54,150
times a inverse is not comfortable
147
00:21:54,150 --> 00:21:59,630
two-norm somehow happens to be convenient
i will give you a way of computing condition
148
00:21:59,630 --> 00:22:06,750
number using 2-norm so condition number using
2-norm is very very you know is used very
149
00:22:06,750 --> 00:22:12,400
often but it does not mean that you cannot
use the other way round you can of course
150
00:22:12,400 --> 00:22:17,890
use the other definition okay
151
00:22:17,890 --> 00:22:27,860
so this in some sense gives a bound on the
amplification factor or amplification of the
152
00:22:27,860 --> 00:22:34,380
error in the solution due to change in the
right hand side or error committed this error
153
00:22:34,380 --> 00:22:38,290
delta b could be committed due to variety
of results it could be because of representation
154
00:22:38,290 --> 00:22:47,700
it could be while doing some computations
so this fundamental quantity which appears
155
00:22:47,700 --> 00:22:51,400
here i will show you that in some other context
also
156
00:22:51,400 --> 00:22:58,660
when you perturb a matrix again same number
will appear okay so there seems to be something
157
00:22:58,660 --> 00:23:04,760
fundamental about this okay now let me analyze
another case
158
00:23:04,760 --> 00:23:18,700
now i want to solve ax=b this is my equation
number i and then i end up solving a+delta
159
00:23:18,700 --> 00:23:58,120
a x+delta x=b okay if i expand this it will
be ax+a delta x+delta ax+delta a delta x=b
160
00:23:58,120 --> 00:24:08,490
okay and then i want to rearrange this and
say that delta x i am returning this i am
161
00:24:08,490 --> 00:24:16,750
returning this delta x delta x on both sides
here okay the idea here is to give you the
162
00:24:16,750 --> 00:24:18,580
spirit of what is happening
163
00:24:18,580 --> 00:24:25,620
as i said it is possible to do a derivation
where a+delta a x+delta x b+delta b i am avoiding
164
00:24:25,620 --> 00:24:31,310
that general derivation i am just looking
at 2 special cases okay now this you get by
165
00:24:31,310 --> 00:24:42,470
subtracting so if i subtract if this is my
equation v and if this is the same as equation
166
00:24:42,470 --> 00:24:50,049
i which we have written earlier so if i subtract
from v from equation i okay b and b on both
167
00:24:50,049 --> 00:24:53,850
sides will disappear okay
168
00:24:53,850 --> 00:25:01,010
b will disappear on both sides ax and ax will
disappear what remains is this term this term
169
00:25:01,010 --> 00:25:07,679
and this term this will go and this will go
if i subtract this equation from this equation
170
00:25:07,679 --> 00:25:28,780
okay then what remains is a delta x=- delta
a times x+delta x okay all that i have done
171
00:25:28,780 --> 00:25:36,570
is i have expanded this and subtracted this
equation okay to get this perturbation equation
172
00:25:36,570 --> 00:25:38,780
okay
173
00:25:38,780 --> 00:25:56,039
so this implies that delta x=-a inverse delta
a or norm delta x=norm a inverse delta a x+delta
174
00:25:56,039 --> 00:26:22,789
x using the fundamental inequality of matrices
i can write this as a inverse norm of delta
175
00:26:22,789 --> 00:26:37,830
x is always less than so i am going to rewrite
this as norm delta x/norm of x+delta x is
176
00:26:37,830 --> 00:26:45,850
everyone with me on this? so this is something
like relative change in the solution except
177
00:26:45,850 --> 00:26:49,270
here we have retained delta x on the denominator
okay
178
00:26:49,270 --> 00:26:54,000
so this is something like relative change
in the solution okay and right hand side is
179
00:26:54,000 --> 00:27:00,320
this okay i am going to play a trick on the
right hand side
180
00:27:00,320 --> 00:27:06,160
i am going to write this as this quantity
on the right hand side i am going to write
181
00:27:06,160 --> 00:27:23,030
as norm of a inverse*norm of a*norm of delta
a
182
00:27:23,030 --> 00:27:36,590
is everyone with me on this? just check this
i am multiplying and dividing by norm a okay
183
00:27:36,590 --> 00:27:53,159
and then i am going to single out this quantity
okay and then using this i am going to write
184
00:27:53,159 --> 00:28:05,289
or i am going to rearrange this inequality
as follows
185
00:28:05,289 --> 00:28:16,610
norm of delta x/norm of x+delta x okay relative
change in the solution to a relative change
186
00:28:16,610 --> 00:28:27,700
in a matrix is again bounded by a*a inverse
that is norm of a*norm of a inverse again
187
00:28:27,700 --> 00:28:35,550
this condition number reappears when you are
trying to analyze a system in which a is represented
188
00:28:35,550 --> 00:28:44,950
slightly erroneously so this is again condition
number this is also condition number of a
189
00:28:44,950 --> 00:28:46,890
okay
190
00:28:46,890 --> 00:28:56,720
so general case where there is error in all
3 becomes little more complex and you can
191
00:28:56,720 --> 00:29:05,260
look at it in some of the text books on numerical
analysis what is important is that this condition
192
00:29:05,260 --> 00:29:14,610
number which seems to play the key role in
how sensitive is your solution to the errors
193
00:29:14,610 --> 00:29:20,289
see what you are looking at here how sensitive
your solution? what is the fractional change
194
00:29:20,289 --> 00:29:27,120
in the solution to fractional change in a
matrix or what are you looking here?
195
00:29:27,120 --> 00:29:32,850
what is the fractional change in the solution
to fractional change in the b matrix? well
196
00:29:32,850 --> 00:29:38,050
we had made while deriving this we had made
that a is perfectly represented there we had
197
00:29:38,050 --> 00:29:45,900
made an assumption that b is perfectly represented
but a is wrong okay so basically you just
198
00:29:45,900 --> 00:29:53,140
get insights into what really seems to be
the key factor this analysis of based on norms
199
00:29:53,140 --> 00:29:58,220
seems to suggest that this quantity is something
fundamental okay
200
00:29:58,220 --> 00:30:06,640
and this determines how well-conditioned how
ill-conditioned a particular system is well
201
00:30:06,640 --> 00:30:15,970
to give you a thumb rule if this number is
large okay then what is large? large is say
202
00:30:15,970 --> 00:30:22,880
about 1000 with this number appears this is
more like reynolds number below 100 if this
203
00:30:22,880 --> 00:30:32,820
quantity is 00:30:35,090
1000 it is a grey area you do not know what
it is
205
00:30:35,090 --> 00:30:45,539
beyond 1000 you can expect trouble okay beyond
1000 you can expect trouble and if it is 10
206
00:30:45,539 --> 00:30:54,850
to the power 5 10 to the power 6 deep trouble
okay nevertheless matlab is a wonderful software
207
00:30:54,850 --> 00:31:01,080
it can give you reasonably accurate solutions
to condition number 10 to the power 5 10 to
208
00:31:01,080 --> 00:31:07,020
the power 6 but matlab starts breaking down
if you give a matrix whose condition number
209
00:31:07,020 --> 00:31:08,600
is very very large
210
00:31:08,600 --> 00:31:16,750
now how do you get more insights into condition
number okay will use here 2-norm to compute
211
00:31:16,750 --> 00:31:26,600
the condition number so i want to get an estimate
of this quantity right
212
00:31:26,600 --> 00:31:37,120
so let us do it using 2-norms for a matrix
if i give you a matrix okay we have done this
213
00:31:37,120 --> 00:31:44,350
earlier 2-norm of the matrix is given by largest
magnitude eigen value of a transpose a okay
214
00:31:44,350 --> 00:32:01,320
so this we have done earlier or max over
i lambda i a transpose a a transpose a is
215
00:32:01,320 --> 00:32:06,600
a positive definite matrix always or positive
semi-definite matrix all the eigen values
216
00:32:06,600 --> 00:32:10,370
are 0 or >0 okay
217
00:32:10,370 --> 00:32:19,169
so non-negative eigen values so you can always
find a maximum so let us call this as lambda
218
00:32:19,169 --> 00:32:29,700
n let us say that we have arranged the eigen
values we have numbered them such that lambda
219
00:32:29,700 --> 00:32:38,020
n is the maximum magnitude eigen value of
a transpose a okay and it is very easy to
220
00:32:38,020 --> 00:32:47,299
show that it is very easy to compute the 2-norm
of a inverse without having to compute a inverse
221
00:32:47,299 --> 00:32:48,850
that is very nice okay
222
00:32:48,850 --> 00:32:55,470
you can compute 2-norm of a inverse you use
a fundamental relationship if you are given
223
00:32:55,470 --> 00:33:06,110
a matrix okay is the relationship between
a non-singular matrix okay its eigen values
224
00:33:06,110 --> 00:33:08,920
and eigen values of a inverse is the relationship
225
00:33:08,920 --> 00:33:26,169
no see if a is a invertible matrix no 0 eigen
value okay then av=or let us take b here because
226
00:33:26,169 --> 00:33:34,309
we are talking about a transpose a bv=lambda
v right okay it is very easy to show if lambda
227
00:33:34,309 --> 00:33:43,029
is a eigen value and if v is the eigen vector
okay i just have to do this pre-multiply both
228
00:33:43,029 --> 00:33:55,090
sides by a inverse so i will get v=lambda
b inverse v okay or 1/lambda v=b inverse v
229
00:33:55,090 --> 00:34:07,539
so if lambda is eigen value of b 1/lambda
is eigen value of b inverse and eigen vectors
230
00:34:07,539 --> 00:34:18,210
are same just see eigen vectors do not change
b inverse v=1/lambda okay non-singular matrix
231
00:34:18,210 --> 00:34:21,980
i am talking about non-singular matrix okay
232
00:34:21,980 --> 00:34:35,930
there is one more thing one more relationship
which i need to use is that a inverse do you
233
00:34:35,930 --> 00:34:50,230
agree with me? a inverse*a is identity transpose=i
okay so
234
00:34:50,230 --> 00:34:57,340
a transpose*a inverse transpose=i is everyone
with me on this? i can interchange inverse
235
00:34:57,340 --> 00:35:07,430
and transpose see what i started with a inverse*a
is identity okay transpose=identity see this
236
00:35:07,430 --> 00:35:10,490
is identity=identity that is what i am writing
237
00:35:10,490 --> 00:35:20,050
then i am expanding this because if you multiply
2 matrices and take their transpose it is
238
00:35:20,050 --> 00:35:36,820
you know this rule cd transpose=d transpose
c transpose so i am just using that here okay
239
00:35:36,820 --> 00:35:45,680
and so
a inverse transpose is same as a transpose
240
00:35:45,680 --> 00:35:46,710
inverse okay
241
00:35:46,710 --> 00:35:57,470
or let us not confuse with that a here let
us put this b notationally i am just talking
242
00:35:57,470 --> 00:36:02,040
about general properties of matrices we were
talking about ax=b i am not talking of that
243
00:36:02,040 --> 00:36:09,760
a in general when a is a invertible matrix
you can write this any square invertible matrix
244
00:36:09,760 --> 00:36:15,810
you can show that b inverse transpose is same
as b transpose inverse you can interchange
245
00:36:15,810 --> 00:36:16,810
transpose and inverse
246
00:36:16,810 --> 00:36:26,470
now these are the 2 properties which i am
going to use i am leaving a few things for
247
00:36:26,470 --> 00:36:46,680
you to derive i am not doing it on the board
but norm a inverse 2 square=max over i lambda
248
00:36:46,680 --> 00:37:04,760
i of a inverse*it is a inverse see what was
here? 2-norm of a is maximum magnitude eigen
249
00:37:04,760 --> 00:37:12,100
value of a transpose a in this case we are
talking about a inverse so in a a inverse
250
00:37:12,100 --> 00:37:18,589
transpose a inverse okay
251
00:37:18,589 --> 00:37:34,140
i am just combining and saying this okay now
eigen values of a transpose a okay a transpose
252
00:37:34,140 --> 00:37:42,500
a is a positive definite matrix all eigen
values are positive okay what is the relationship
253
00:37:42,500 --> 00:37:52,740
between eigen values of a transpose a and
a transpose a inverse? 1 upon right so if
254
00:37:52,740 --> 00:38:13,040
lambda 1 is the smallest eigen value of a
transpose a what will be the largest eigen
255
00:38:13,040 --> 00:38:15,630
value of a transpose a inverse?
256
00:38:15,630 --> 00:38:32,000
1/lambda 1 so 1/lambda 1 is the largest eigen
value
257
00:38:32,000 --> 00:38:46,089
a transpose a inverse okay this is the largest
eigen value of a transpose a inverse is everyone
258
00:38:46,089 --> 00:38:52,930
with me on this? so lambda n what is the largest
eigen value? we have numbered eigen values
259
00:38:52,930 --> 00:38:58,339
the largest one we are calling lambda n the
smallest one we are calling lambda 1 for a
260
00:38:58,339 --> 00:39:02,400
transpose a you have some doubts?
261
00:39:02,400 --> 00:39:07,350
a transpose a we have numbered the eigen values
lambda 1 is the smallest lambda n is the largest
262
00:39:07,350 --> 00:39:12,980
we have chosen the numbering we look at the
eigen values and we number them lambda n is
263
00:39:12,980 --> 00:39:19,060
the largest lambda 1 is the smallest now if
lambda 1 is the smallest eigen value of a
264
00:39:19,060 --> 00:39:24,220
transpose a then 1/lambda 1 is the largest
eigen value of a transpose a inverse it follows
265
00:39:24,220 --> 00:39:30,820
from the first relationship okay
266
00:39:30,820 --> 00:39:49,520
so now what i need to do is okay so i am now
going to combine these 2 things and come here
267
00:39:49,520 --> 00:40:00,970
i am going to combine these 2 things so norm
a2 square*norm a inverse 2 square seems to
268
00:40:00,970 --> 00:40:19,990
be=lambda n/lambda 1 okay or norm a2 norm
a inverse 2 is square root of lambda n/lambda
269
00:40:19,990 --> 00:40:33,790
1 where what is lambda n? largest eigen value
which also called as singular value of a okay
270
00:40:33,790 --> 00:40:45,500
largest eigen value of a transpose a and what
is lambda 1?
271
00:40:45,500 --> 00:40:57,320
smallest eigen value so condition number of
this matrix is simply ratio of largest eigen
272
00:40:57,320 --> 00:41:05,060
value of a transpose a to smallest eigen value
of a transpose a okay computing this largest
273
00:41:05,060 --> 00:41:14,300
or smallest eigen values for a positive definite
matrix not that difficult not very difficult
274
00:41:14,300 --> 00:41:18,000
why positive definite? because a transpose
a is a positive definite matrix okay
275
00:41:18,000 --> 00:41:23,609
if you ask matlab fit your matrix a and say
cond it will give you condition number of
276
00:41:23,609 --> 00:41:31,530
matrix okay actually it will compute this
ratio and tell you what is the condition number
277
00:41:31,530 --> 00:41:32,530
okay
278
00:41:32,530 --> 00:41:51,520
so this way i would call this quantity as
c2 of a okay i will call this quantity c2
279
00:41:51,520 --> 00:42:00,400
of a so condition number based on 2-norm of
the matrix okay likewise i can define condition
280
00:42:00,400 --> 00:42:05,970
number based on infinite norm or 1 norm okay
though in that case i will have to compute
281
00:42:05,970 --> 00:42:15,700
a inverse explicitly and of course we can
do that for a simple matrix to get insights
282
00:42:15,700 --> 00:42:26,200
though for a large matrix that might be not
so suitable
283
00:42:26,200 --> 00:42:38,940
so i can define c infinity of a as
infinite norm okay depending upon way you
284
00:42:38,940 --> 00:42:49,140
choose norms you can define the condition
number okay now let me complete some story
285
00:42:49,140 --> 00:42:55,560
which i began long time back i never told
you exact reason i kept on saying that polynomial
286
00:42:55,560 --> 00:43:04,430
approximations okay polynomial approximations
give rise to difficulties because polynomials
287
00:43:04,430 --> 00:43:12,640
higher and you know fourth order or fifth
order you know you get ill-conditioned matrices
288
00:43:12,640 --> 00:43:13,640
okay
289
00:43:13,640 --> 00:43:18,360
something may appear disconnected now okay
but i want to complete the loop now and go
290
00:43:18,360 --> 00:43:27,760
back and say why polynomial approximations
higher than you know certain order okay so
291
00:43:27,760 --> 00:43:32,970
why does it happen that polynomial approximations
create problems? i am going to analyze this
292
00:43:32,970 --> 00:43:40,900
in condition number okay sometime back i had
put one more quiz in the puzzle
293
00:43:40,900 --> 00:43:46,200
i had showed you that when you try to do polynomial
approximations you get a matrix called as
294
00:43:46,200 --> 00:43:52,970
hilbert matrix h right actually in matlab
there is a command called hilb if you say
295
00:43:52,970 --> 00:43:59,099
hilb 3 it will give you 3 cross 3 matrix which
is hilbert matrix hilb 4 will give you 4 cross
296
00:43:59,099 --> 00:44:04,170
4 matrix and you can just do this once just
go and create hilbert matrices and start looking
297
00:44:04,170 --> 00:44:05,339
at a condition number
298
00:44:05,339 --> 00:44:15,730
condition number is let say this okay condition
number of hilbert matrix are notoriously bad
299
00:44:15,730 --> 00:44:21,140
okay what does it mean? it means that if you
make a slight error in the representation
300
00:44:21,140 --> 00:44:30,040
of numbers okay these ratios are going to
be you know see this condition number tells
301
00:44:30,040 --> 00:44:35,660
you worse case error it is not that for a
particular case it will happen but if it happens
302
00:44:35,660 --> 00:44:39,210
it can be very bad i have shown you some examples
right
303
00:44:39,210 --> 00:44:45,390
you perturb a matrix slightly your solution
just goes you know out of box 1 1 1 1 it goes
304
00:44:45,390 --> 00:44:50,520
somewhere else you know it is not in some
small neighborhood of 1 1 1 1 it goes to 83
305
00:44:50,520 --> 00:44:57,800
and some 52 so solution can be completely
different with a small error okay so let me
306
00:44:57,800 --> 00:45:05,660
take this hilbert matrix h3 is this hilbert
matrix hilbert matrix has a very nice structure
307
00:45:05,660 --> 00:45:20,619
1/2 1/3 1/4 1/5
308
00:45:20,619 --> 00:45:26,650
and i kept on telling you that second order
or cubic polynomial is okay but fourth order
309
00:45:26,650 --> 00:45:32,840
fifth order sixth order seventh order polynomial
okay become bad to solve
310
00:45:32,840 --> 00:45:43,920
why becomes because you get a situation you
get h times theta=some u u is known on the
311
00:45:43,920 --> 00:45:48,359
right hand side h is the hilbert matrix theta
are the parameters to be estimated you got
312
00:45:48,359 --> 00:45:57,030
this kind of an equation okay theta are the
polynomial coefficients okay h is the hilbert
313
00:45:57,030 --> 00:46:01,540
matrix and u is the right hand side whatever
is the right hand side what is important is
314
00:46:01,540 --> 00:46:08,720
how well-conditioned is h matrix because you
are solving ax=b this is another form right
315
00:46:08,720 --> 00:46:21,300
estimating coefficients of a polynomial okay
now for this simple matrix you can show that
316
00:46:21,300 --> 00:46:29,980
i am going to call this as h3 because it is
3 cross 3 you can show that it is 1-norm is
317
00:46:29,980 --> 00:46:39,770
same as x3 infinite norm this is 11/6 and
you can actually for this simple matrix you
318
00:46:39,770 --> 00:46:58,320
can compute the inverse and you can show that
h3 inverse 1-norm=h3 inverse infinite norm=408
319
00:46:58,320 --> 00:47:03,530
you just compute h inverse for 3 cross 3 matrix
you can do it by hand also
320
00:47:03,530 --> 00:47:13,579
you will get exact solution and you can actually
get these numbers and then what you show here
321
00:47:13,579 --> 00:47:33,910
is c1 h3 is same as c infinity h3 which turns
out to be 748 okay the calculations are not
322
00:47:33,910 --> 00:47:44,730
bad condition number is 748 okay the worst
case error that can happen is of the order
323
00:47:44,730 --> 00:47:51,650
of 1000 times okay not so bad not so bad just
let see what happens if you want to fit a
324
00:47:51,650 --> 00:47:54,310
sixth order polynomial okay
325
00:47:54,310 --> 00:48:11,020
in a sixth order polynomial okay but even
in this case for h6 would be 1 1/2 1/3 1/4
326
00:48:11,020 --> 00:48:31,680
1/5 1/6 1/2 1/3 and so on okay up to 1/6 1/7
okay h6 if i want to fit a sixth order polynomial
327
00:48:31,680 --> 00:48:49,550
then i will get h6*theta okay h6*theta=u okay
i will get h6*theta=u and what you can show
328
00:48:49,550 --> 00:48:56,800
is that c1 h6 condition number based on 1-norm
which in this particular case turns out to
329
00:48:56,800 --> 00:49:09,310
be condition number based on infinite norm
also same for a sixth order polynomial okay
330
00:49:09,310 --> 00:49:18,750
just look at this condition number is so bad
okay whatever you try to do you will not going
331
00:49:18,750 --> 00:49:25,940
to get reliable solutions 10 to the power
7 a small error can get amplified in certain
332
00:49:25,940 --> 00:49:31,150
directions in very very bad measures what
are those certain directions? those are related
333
00:49:31,150 --> 00:49:41,300
to the eigen vectors of a transpose a in the
directions in which a transpose a has maximum
334
00:49:41,300 --> 00:49:48,839
magnitude eigen value the eigen vector corresponding
to lambda n will amplify your error worst
335
00:49:48,839 --> 00:49:49,839
okay
336
00:49:49,839 --> 00:49:58,910
so it depends upon how is your b is aligned
you will get so this particular matrix as
337
00:49:58,910 --> 00:50:03,530
you start increasing polynomial order you
will get hilbert matrices of higher and higher
338
00:50:03,530 --> 00:50:12,250
order if you go to tenth you know 10 to the
power 12 so a small error committed can create
339
00:50:12,250 --> 00:50:26,339
a havoc okay i will just illustrate how things
can be different even for this h3 okay then
340
00:50:26,339 --> 00:50:31,770
you can judge what will happen for you know
sixth order polynomial or seventh order polynomial
341
00:50:31,770 --> 00:50:37,560
why we do not get good results why we need
cubics line? why we need polynomial interpolations
342
00:50:37,560 --> 00:50:44,589
which are you know piecewise polynomial interpolations?
why do we really go for that?
343
00:50:44,589 --> 00:51:08,380
i just complete this one example see if h3*
1 1 1 will give you 11/6 13/12 47/60 okay
344
00:51:08,380 --> 00:51:14,240
now what i am going to do is that instead
of solving for this problem i am going to
345
00:51:14,240 --> 00:51:30,190
round off this matrix okay i am going to solve
for h+delta h what is my h+delta h? okay i
346
00:51:30,190 --> 00:51:36,369
have rounded off this right hand side perfectly
okay right this 2 digit approximation very
347
00:51:36,369 --> 00:51:39,130
very often we do in calculations
348
00:51:39,130 --> 00:51:45,940
and this you might see nothing wrong you know
1/3 being represented as 0333 i am truncating
349
00:51:45,940 --> 00:51:54,099
okay nothing wrong my solution here was 1
1 1 okay right hand side was this left hand
350
00:51:54,099 --> 00:52:05,810
side was that i just a+h+delta h and this
is b+delta b how much does the solution change?
351
00:52:05,810 --> 00:52:34,369
okay this x+delta x turns out to be 183 108
0783 just imagine tiny error in every number
352
00:52:34,369 --> 00:52:35,369
okay
353
00:52:35,369 --> 00:52:40,119
trying to fit a third order polynomial okay
you are trying to go from that matrix to this
354
00:52:40,119 --> 00:52:49,700
matrix you might find perfectly reasonable
okay i will get this solution a small perturbation
355
00:52:49,700 --> 00:52:59,000
for a matrix whose condition number is only
700 okay not very bad okay gives me so much
356
00:52:59,000 --> 00:53:04,690
difference in the solution if i decide to
represent this by this and that matrix by
357
00:53:04,690 --> 00:53:08,630
this matrix okay
358
00:53:08,630 --> 00:53:16,790
you see why condition number is so important
when you want to study matrix computations
359
00:53:16,790 --> 00:53:24,650
okay which solution is the correct solution
now? the correct solution is 1 1 1 right what
360
00:53:24,650 --> 00:53:31,520
you are getting here is completely different
just imagine what will happen if condition
361
00:53:31,520 --> 00:53:38,410
number is 10 to the power 6 or 10 to the 4
or 10 to the power 5 okay so the solutions
362
00:53:38,410 --> 00:53:42,530
which matlab gives you not matlab i should
not singular matlab
363
00:53:42,530 --> 00:53:49,171
any software will give you for a matrix with
high condition number is likely to be a garbage
364
00:53:49,171 --> 00:53:56,300
and you should know this okay when the solution
is garbage and when you have committed mistake
365
00:53:56,300 --> 00:53:59,890
if a matrix is well-conditioned and if you
are getting garbage you have made a mistake
366
00:53:59,890 --> 00:54:04,839
in programing if a matrix is ill-conditioned
and if you are getting a garbage not that
367
00:54:04,839 --> 00:54:09,310
the software is wrong not that the program
is wrong it is inherent problem okay
368
00:54:09,310 --> 00:54:14,200
just see here 700 condition number small change
in the right hand side a very small change
369
00:54:14,200 --> 00:54:20,430
in the left hand side a and b matrices you
get drastically different solution even for
370
00:54:20,430 --> 00:54:24,710
third order polynomial you have this situation
that is why we do not try to fit high order
371
00:54:24,710 --> 00:54:35,190
polynomials we will continue this story and
this is end of this series of lectures on
372
00:54:35,190 --> 00:54:38,520
ax=b a little bit of it is remaining will
complete it on the next lecture