1 00:00:17,060 --> 00:00:27,160 so we have been looking at this method of optimization numerical optimization to solve 2 00:00:27,160 --> 00:00:33,070 linear algebraic equations and what we derived in the last class was the gradient method 3 00:00:33,070 --> 00:00:45,010 or the cauchy method the gradient method does a search iterative search in the negative 4 00:00:45,010 --> 00:00:47,370 of the gradient direction 5 00:00:47,370 --> 00:01:02,140 and if i summarize this algorithm this was so we are trying to solve a x=b and we have 6 00:01:02,140 --> 00:01:22,759 this we are assuming that a is symmetric or a=a transpose and a is well in applied mathematics 7 00:01:22,759 --> 00:01:30,399 or in linear algebra you often write that a is positive definite by saying a>0 okay 8 00:01:30,399 --> 00:01:44,680 that means a is positive definite that does not mean all elements of a are >0 when you 9 00:01:44,680 --> 00:01:49,320 write a>0 it means a is positive definite a is asymmetric 10 00:01:49,320 --> 00:01:54,450 and then we formulated an objective function well we know what to do if a is not symmetric 11 00:01:54,450 --> 00:02:00,189 positive definitely we pre-multiply both sides by a transpose and then we get we work with 12 00:02:00,189 --> 00:02:10,899 the modified problem so phi is 1/2 x transpose a x-x transpose b we want to minimize this 13 00:02:10,899 --> 00:02:23,250 with respect to x phi and then we use this gradient method to come up with the search 14 00:02:23,250 --> 00:02:33,740 direction so what was this gradient method? the gradient method was we find the gradient 15 00:02:33,740 --> 00:02:40,010 direction 16 00:02:40,010 --> 00:02:51,060 so dou phi/dou=a x-b this is the gradient and we want to move in the negative of the 17 00:02:51,060 --> 00:03:08,260 gradient so we take g k=-a x k-b this is my negative of the gradient direction and then 18 00:03:08,260 --> 00:03:16,670 we move how much do we move this time we saw a one-dimensional optimization problem and 19 00:03:16,670 --> 00:03:32,489 then we move actually x k+1=x k+ lambda k g k and we found that lambda k is nothing 20 00:03:32,489 --> 00:03:47,250 but b transpose g k okay so we make an optimal move every time in the negative of the gradient 21 00:03:47,250 --> 00:03:48,250 direction 22 00:03:48,250 --> 00:03:52,310 this is negative of the gradient direction lambda k times g k that is the move that we 23 00:03:52,310 --> 00:04:01,480 make every time and this is the classical gradient method or the cauchy method and then 24 00:04:01,480 --> 00:04:14,379 these iterations are stopped when you reach a point or gradient is not changing either 25 00:04:14,379 --> 00:04:19,519 way of them is fine x k is not changing the new iterations are not adding too much information 26 00:04:19,519 --> 00:04:25,500 we specified an epsilon epsilon could be something like 10 to the power-8 10 to the power-10 27 00:04:25,500 --> 00:04:27,890 a very small number okay 28 00:04:27,890 --> 00:04:35,120 and we say that the relative change in x as the new iterates are generated this is insignificant 29 00:04:35,120 --> 00:04:40,890 that is when we stop okay so this is iterative scheme and if you remember if you see here 30 00:04:40,890 --> 00:04:57,420 every time you need to compute a x k-b k you need to compute a x k-b this is the residual 31 00:04:57,420 --> 00:05:06,280 actually when you solve this this g k will be 0 gradient direction is 0 which is the 32 00:05:06,280 --> 00:05:10,889 necessary condition for optimality for this particular objective function okay 33 00:05:10,889 --> 00:05:19,710 so finding the optimum is equivalent to reaching the or meeting the necessary conditions for 34 00:05:19,710 --> 00:05:29,680 optimality and we satisfy or we terminate based on even you 35 00:05:29,680 --> 00:05:35,490 can terminate based on g k if g k is very very small okay the gradient vector is very 36 00:05:35,490 --> 00:05:41,300 very small now this slight modification of this because this the problem with this method 37 00:05:41,300 --> 00:05:46,580 it works very well when you are far away from the optimum when you come close to the optimum 38 00:05:46,580 --> 00:05:49,440 this tends to become very very slow okay 39 00:05:49,440 --> 00:05:53,449 the moves that you make every time when you come close to the optimum are quite small 40 00:05:53,449 --> 00:05:59,319 so that trouble is this we need to do something to accelerate the convergence when you come 41 00:05:59,319 --> 00:06:06,449 to closer to the optimum okay now what i am saying is not just true for this particular 42 00:06:06,449 --> 00:06:11,210 problem it is in general to that gradient method is one of the very basic methods for 43 00:06:11,210 --> 00:06:17,190 doing numerical search or numerical optimization but the problem with gradient method is that 44 00:06:17,190 --> 00:06:18,870 it works very well when you are away from the optimum 45 00:06:18,870 --> 00:06:23,240 and when you come very close to the optimum it becomes very very slow so convergence to 46 00:06:23,240 --> 00:06:28,599 the optimum is slow okay and we need to do something about it okay 47 00:06:28,599 --> 00:06:40,690 so there is a variant which is called as conjugate gradient method okay so what we do is we move 48 00:06:40,690 --> 00:06:55,629 in the conjugate directions a conjugate directions okay so given a matrix a okay a is our matrix 49 00:06:55,629 --> 00:07:13,490 we call see a set of directions s 0 s 1 let us say that s 1 s 2 s 3 are directions in 50 00:07:13,490 --> 00:07:21,910 which we want to move okay these directions are called as a conjugate if this condition 51 00:07:21,910 --> 00:07:37,300 is satisfied if you take 2 directions k and k-1 not any 2 k and k-1 okay say new direction 52 00:07:37,300 --> 00:07:39,659 and previous direction okay 53 00:07:39,659 --> 00:07:48,050 and if this in a product is 0 okay or if this whatever you want to call it dot product is 54 00:07:48,050 --> 00:07:53,860 this 0 which is weighted by a matrix okay then these directions are called as a conjugate 55 00:07:53,860 --> 00:07:59,770 directions okay and what you can show is that instead of moving along the gradient direction 56 00:07:59,770 --> 00:08:06,060 if you move along the a conjugate directions then you reach the optimum very very fast 57 00:08:06,060 --> 00:08:10,919 okay if you have the notes there is a small correction it is on equation 129 58 00:08:10,919 --> 00:08:21,651 so how do you generate this a conjugate directions okay we are going to use this formula s k=beta 59 00:08:21,651 --> 00:08:39,030 k s k-1+ so we are going to use the gradient 60 00:08:39,030 --> 00:08:44,500 direction it is not that i am not going to use gradient direction okay it is the conjugate 61 00:08:44,500 --> 00:08:51,160 gradient algorithm so i am going to use the gradient vector current gradient vector but 62 00:08:51,160 --> 00:08:59,070 to that i am going to add some correction which is based on the previous this is based 63 00:08:59,070 --> 00:09:04,680 on the previous you know previous direction okay 64 00:09:04,680 --> 00:09:17,340 and i kick off this algorithm with s 0= so what means sorry s 0=0 which means s 1 the 65 00:09:17,340 --> 00:09:24,750 first move that i am going to make okay the first move that i am going to make is g k 66 00:09:24,750 --> 00:09:35,880 g 1 okay the first gradient let us say i am starting my iterations at 1 i am taking s 67 00:09:35,880 --> 00:09:42,339 0 to be 0 so the first direction is the gradient direction what will be the subsequent direction 68 00:09:42,339 --> 00:09:49,630 well how do i choose this beta k and all that will become clear soon 69 00:09:49,630 --> 00:10:06,190 what is s 2? s 2 will be beta 2 s 1+ g 2 that means gradient at point 2 negative of the 70 00:10:06,190 --> 00:10:10,980 gradient at the point 2 sorry g here is negative of the gradient we are going to move in the 71 00:10:10,980 --> 00:10:14,800 negative of the gradient we are minimizing not the gradient direction g is the negative 72 00:10:14,800 --> 00:10:36,421 of the gradient direction okay so g 1 is nothing but -of a x 1-b okay and so on okay i cannot 73 00:10:36,421 --> 00:10:47,760 just go on doing this so you can see see what is this? this is beta 2 g 1+g 2 right 74 00:10:47,760 --> 00:10:54,800 so i am taking the linear combination of the 2 gradient directions not just negative of 75 00:10:54,800 --> 00:11:01,920 the current gradient direction but i am taking linear combination and so on so this every 76 00:11:01,920 --> 00:11:13,079 iteration i am going to do this okay now i am going to invoke this a conjugacy so what 77 00:11:13,079 --> 00:11:29,560 i am going to do now is to take s k transpose and pre-multiply it by a s k-1 okay so what 78 00:11:29,560 --> 00:11:57,600 is this beta k s k -1+g k transport a s k-1 right now these are a conjugate directions 79 00:11:57,600 --> 00:12:07,209 what should be this? this should be 0 so since these are a conjugate this left hand side 80 00:12:07,209 --> 00:12:24,540 is 0 okay so i get what do i get from here 0=beta k s k-1 a s k s k -1 transpose a s 81 00:12:24,540 --> 00:12:35,670 k see s k-1 let us put it in the bracket become easier to understand this is s k -1 transpose 82 00:12:35,670 --> 00:13:01,930 a s k-1 okay +g k transpose a s k -1 this see what is g k is the negative of the current 83 00:13:01,930 --> 00:13:09,899 gradient direction g k is known to me okay s k -1 is the previous direction okay so using 84 00:13:09,899 --> 00:13:14,100 this i can find out beta k okay 85 00:13:14,100 --> 00:13:23,160 so that gives me a formula for beta k how much to move what is the linear combination 86 00:13:23,160 --> 00:13:37,670 that should be made okay for a conjugate directions so now and what is this if i write g k to 87 00:13:37,670 --> 00:13:49,360 be -of a x k-b that completes the formula so i have this g k which is given by this 88 00:13:49,360 --> 00:13:57,310 this is how you kick off your iterations and this is how you compute beta k okay this is 89 00:13:57,310 --> 00:14:18,330 how you combine okay so actually i can combine this and okay i can combined the 2 and i can 90 00:14:18,330 --> 00:14:21,350 find out this as my iteration scheme 91 00:14:21,350 --> 00:14:30,829 this is a scalar remember this is a scalar ratio of 2 scalars this is a vector direction 92 00:14:30,829 --> 00:14:37,930 and this is current negative of the gradient direction so this is how you combine and see 93 00:14:37,930 --> 00:14:48,389 how beta k changes with little bit of algebraic jugglery what you can show is that i am skipping 94 00:14:48,389 --> 00:14:55,110 this i have given the derivation here you can just go over it it is that s k with some 95 00:14:55,110 --> 00:15:06,709 algebraic manipulation can be shown to be beta and -beta 1 well if you do a little bit 96 00:15:06,709 --> 00:15:09,089 of index jugglery 97 00:15:09,089 --> 00:15:16,920 if i just come back here i will do a little bit of index jugglery i will call this -1 98 00:15:16,920 --> 00:15:41,519 s -1=0 okay so if s -1=0 then s 0 becomes g 0 which is-same as -a x 0-b okay if you 99 00:15:41,519 --> 00:15:47,550 do a little bit of index jugglery you called 0 as the starting point -1 previous to the 100 00:15:47,550 --> 00:15:54,209 starting point okay because if you put k=0 whether it will start from k=1 or k=0 it is 101 00:15:54,209 --> 00:16:01,690 a matter of just writing the algorithm to be consistent with other notation here i will 102 00:16:01,690 --> 00:16:10,540 just so g 0 is the negative of the initial gradient direction 103 00:16:10,540 --> 00:16:16,860 so what you can show with the little bit of jugglery is that so in this case you find 104 00:16:16,860 --> 00:16:23,300 the current direction in which to move using a linear combination of all the previous gradient 105 00:16:23,300 --> 00:16:27,509 directions you do not use only one current gradient direction you use a linear combination 106 00:16:27,509 --> 00:16:32,630 of all previous gradient directions okay now this is the direction in which you want to 107 00:16:32,630 --> 00:16:37,540 move how much do you want to move? you have to use this lambda business okay 108 00:16:37,540 --> 00:16:48,459 so after we have decided this s k what we do is okay so my new point that is x k+1=x 109 00:16:48,459 --> 00:17:04,310 k+ lambda s k is only the direction just number okay and then i find out so lambda k=min with 110 00:17:04,310 --> 00:17:22,900 respect to lambda phi of x k+ lambda s k this is =min with respect to lambda okay and then 111 00:17:22,900 --> 00:17:36,080 this can be solved very very easily 112 00:17:36,080 --> 00:17:44,650 lambda k turns out to be this scalar how much to move in this direction is given by this 113 00:17:44,650 --> 00:17:51,800 particular scalar okay 114 00:17:51,800 --> 00:18:07,400 now what i can show is that theoretically i can reach the optimum of phi only in n steps 115 00:18:07,400 --> 00:18:13,180 okay i have started with this objective function where a is symmetric positive definite so 116 00:18:13,180 --> 00:18:27,680 theoretically you can show that 1/2 x transpose a x-x transpose b okay solution of this can 117 00:18:27,680 --> 00:18:37,220 be reached starting from arbitrary initial condition only in n steps okay but of course 118 00:18:37,220 --> 00:18:44,870 what happens is there are numerical errors when you compute in a computer and you cannot 119 00:18:44,870 --> 00:18:46,450 practically reach in n steps 120 00:18:46,450 --> 00:18:56,420 on paper you should converge to the solution only in n steps very very powerful method 121 00:18:56,420 --> 00:19:06,441 okay now just look at a matrix which is say 1000 cross 1000 okay your gradient to reach 122 00:19:06,441 --> 00:19:14,430 very close to the solution in 1000 steps okay each step only requires one matrix multiplication 123 00:19:14,430 --> 00:19:23,841 right it requires some matrix multiplications and you know you converge to the solution 124 00:19:23,841 --> 00:19:30,210 very fast okay so this is actually one of the popular methods for solving linear algebraic 125 00:19:30,210 --> 00:19:36,550 equations for large-scale systems and very quickly you can reach the solution so that 126 00:19:36,550 --> 00:19:40,050 is one of the 127 00:19:40,050 --> 00:19:46,150 so you can get very close to the reasonably accurate solution you can get very quickly 128 00:19:46,150 --> 00:19:59,030 yeah you the lambda k can have some problem but 129 00:19:59,030 --> 00:20:09,820 you are multiplying by a right g k transpose a g k no but i do not think that will happen 130 00:20:09,820 --> 00:20:17,510 in conjugate gradient gradient method you can end up in the problem because of that 131 00:20:17,510 --> 00:20:25,710 you can get not 0/0 you get a small number by small number 132 00:20:25,710 --> 00:20:32,820 so that can sometimes cause a problem but more problem with in gradient method is not 133 00:20:32,820 --> 00:20:37,730 because of that the difficulty is because it becomes very slow as you come and this 134 00:20:37,730 --> 00:20:42,700 does not become slow this homes down to the solution in the n steps that least theoretically 135 00:20:42,700 --> 00:20:52,890 so it is a much better method okay so we move on to the next thing now well we have talked 136 00:20:52,890 --> 00:20:58,550 about different methods of solving a x=b let me just take a brief review of what we have 137 00:20:58,550 --> 00:20:59,800 done till now 138 00:20:59,800 --> 00:21:09,370 we started looking at direct methods we had a at least taste of what are called as sparse 139 00:21:09,370 --> 00:21:15,120 matrix methods now we try to exploit number of 0s that are present in the matrix and then 140 00:21:15,120 --> 00:21:24,990 we come up with a way of solving very quickly in less number of multiplication and division 141 00:21:24,990 --> 00:21:29,620 then we moved on to iterative methods for solving a x=b which was completely new to 142 00:21:29,620 --> 00:21:33,930 you because you have been looking at gaussian elimination gauss jordan everything that is 143 00:21:33,930 --> 00:21:34,930 using 144 00:21:34,930 --> 00:21:44,200 but iterative methods can particularly in let us say in newton-raphson when you are 145 00:21:44,200 --> 00:21:49,800 solving multiple a x=b problems it is not that important that you exactly get the true 146 00:21:49,800 --> 00:21:56,840 solution at the iterates you can very quickly get the solution approximate solution very 147 00:21:56,840 --> 00:22:01,480 close to the approximate solution using iterative methods so jacobi method gauss seidel method 148 00:22:01,480 --> 00:22:06,260 we had a look at those methods we also looked at why and how they converge how to make them 149 00:22:06,260 --> 00:22:07,590 converge and so on 150 00:22:07,590 --> 00:22:15,650 so we relate to the eigenvalues we said that inside unit circle know if you have i eigenvalues 151 00:22:15,650 --> 00:22:23,050 of s inverse you can guarantee convergence to the solution necessary and sufficient conditions 152 00:22:23,050 --> 00:22:27,650 we also derive some sufficient conditions some necessary and sufficient conditions or 153 00:22:27,650 --> 00:22:33,300 how do you transform the problem to guarantee solution and so on and we looked at very quickly 154 00:22:33,300 --> 00:22:39,420 gradient based methods optimization based methods okay gradient method and conjugate 155 00:22:39,420 --> 00:22:42,350 gradient method 156 00:22:42,350 --> 00:22:45,951 if you do a course on optimization you will see that gradient method and conjugate gradient 157 00:22:45,951 --> 00:22:52,030 method they form you know basis of many many variants and there are more powerful variants 158 00:22:52,030 --> 00:23:02,590 of but these are 2 basic methods which are used in numerical optimization the next thing 159 00:23:02,590 --> 00:23:12,620 that i want to move is what is called as a condition number well what i want to talk 160 00:23:12,620 --> 00:23:22,150 about now is very very important aspect of numerical computing this is how well-conditioned 161 00:23:22,150 --> 00:23:27,360 or ill-conditioned system of linear algebraic equation is okay 162 00:23:27,360 --> 00:23:42,610 so now the problems that occur in numerical computing one of the reasons for these problems 163 00:23:42,610 --> 00:23:49,170 is fixed point arithmetic okay what is fixed point arithmetic? fixed point arithmetic is 164 00:23:49,170 --> 00:23:56,370 that we have finite number of bits to represent a number okay and then we truncate okay so 165 00:23:56,370 --> 00:24:01,720 suppose a number goes beyond this representation or if there is a number which cannot be represented 166 00:24:01,720 --> 00:24:07,800 exactly using finite number of bits we just take the nearest approximation and work with 167 00:24:07,800 --> 00:24:08,910 it okay 168 00:24:08,910 --> 00:24:17,100 just to give you an idea that if you have a computer which has 3 bit precision okay 169 00:24:17,100 --> 00:24:39,780 and if we are to add point you know suppose i want to do this addition okay a 3 bit precision 170 00:24:39,780 --> 00:24:45,460 computer or 3 digit precision let us not say 3 bit 3 digit precision computer cannot actually 171 00:24:45,460 --> 00:24:51,350 look at this part of the number it will ignore this part of the number it will only take 172 00:24:51,350 --> 00:25:04,770 this+0002 okay and that 3 digit computer will commit an error of 31 10 to the power-4 in 173 00:25:04,770 --> 00:25:08,030 this addition okay 174 00:25:08,030 --> 00:25:15,250 and just imagine this is now of course we have numbers floating point which is 16 digit 175 00:25:15,250 --> 00:25:22,150 precision and so on but there is something that is being neglected okay and even though 176 00:25:22,150 --> 00:25:28,970 every time it is very very small okay each time when you do it it is very very small 177 00:25:28,970 --> 00:25:32,710 number that is neglected but you know you are doing millions of operations right you 178 00:25:32,710 --> 00:25:38,910 are doing millions of operations large matrix you are doing so many 1000s of multiplications 179 00:25:38,910 --> 00:25:39,910 division 180 00:25:39,910 --> 00:25:45,390 each time you are neglecting a small number okay each time you neglect a small number 181 00:25:45,390 --> 00:25:51,830 that can cascade and grow and build up and create a problem now this happens only for 182 00:25:51,830 --> 00:26:01,100 those cases where the matrix is problematic when matrix is not problematic okay the errors 183 00:26:01,100 --> 00:26:07,420 have natural tendency to die down and things work out you get reasonable solutions so actually 184 00:26:07,420 --> 00:26:15,960 what you have to remember is that when you are solving a x=b using a computer 185 00:26:15,960 --> 00:26:20,880 you actually never end of solving this exact equation you end up solving something like 186 00:26:20,880 --> 00:26:41,050 this you end up solving a+ delta a x+ delta x=b+ delta b i want to solve this a x=b i 187 00:26:41,050 --> 00:26:53,760 end up solving this problem that is because i cannot represent a accurately inside my 188 00:26:53,760 --> 00:27:00,110 computer i cannot represent b accurately inside my computer if a and b are not accurately 189 00:27:00,110 --> 00:27:05,430 represented i cannot get exact solution to my problem okay 190 00:27:05,430 --> 00:27:19,520 now just to give you so what i want to do is i want to have some idea when this solution 191 00:27:19,520 --> 00:27:33,000 will be far away from what i really want to get okay what are the factors that influence 192 00:27:33,000 --> 00:27:39,490 how this solution behaves first of all the right hand side b is something which is specified 193 00:27:39,490 --> 00:27:46,330 by the person who is using this set of equations by the user let us say so b is not in our 194 00:27:46,330 --> 00:27:55,870 hands okay what is fixed about this system of equations is matrix a okay 195 00:27:55,870 --> 00:28:04,930 so how the solutions behave it actually depends upon how you know a matrix is or how well-conditioned 196 00:28:04,930 --> 00:28:15,390 or how ill-conditioned a matrix is so our main you know culprit or when things go wrong 197 00:28:15,390 --> 00:28:21,800 is a matrix how to recognize whether a matrix is good or a matrix is bad if a matrix is 198 00:28:21,800 --> 00:28:28,740 bad whatever you do okay to some extent well good programs like mat lab can recover and 199 00:28:28,740 --> 00:28:33,550 give you a reasonable solution but beyond the point even mat lab fails 200 00:28:33,550 --> 00:28:39,870 and it is not because there is something wrong with mat lab because there is a problem with 201 00:28:39,870 --> 00:28:46,340 the matrix the matrix is ill-conditioned in iterations sometimes you may have received 202 00:28:46,340 --> 00:28:52,620 a message like this matrix is ill-conditioned the results will not be reliable so mat lab 203 00:28:52,620 --> 00:28:57,340 does its job it tells you that there is something going on wrong here you should know what it 204 00:28:57,340 --> 00:29:04,570 means you should know what the warning means you should understand and analyze okay 205 00:29:04,570 --> 00:29:14,300 so let me give you a motivation for what is going to happen next that is condition numbering 206 00:29:14,300 --> 00:29:21,770 i will need one more lecture to cover this business of condition numbers but i am going 207 00:29:21,770 --> 00:29:26,620 to give a motivation for this today why do i want to look at condition number then we 208 00:29:26,620 --> 00:29:39,290 will look at the mathematical details okay so this example i have taken from strang it 209 00:29:39,290 --> 00:29:53,030 is a very nice simple example which illustrates he using a very simple example 2 cross 2 example 210 00:29:53,030 --> 00:30:00,080 strang has illustrated the point that sometimes the calculations go wrong because you are 211 00:30:00,080 --> 00:30:04,890 not done proper ordering of the calculations if you reorder the calculations things can 212 00:30:04,890 --> 00:30:15,550 be okay but sometimes the matrix is bad whatever you do there will be a problem okay so this 213 00:30:15,550 --> 00:30:29,370 system 1 he considers is the simple system see this is the very very simple system 00001 214 00:30:29,370 --> 00:30:43,510 1 1 and 1 okay so now how will you do gaussian elimination? 215 00:30:43,510 --> 00:30:48,880 if you do it without thinking about the problem you will just take this is a non 0 number 216 00:30:48,880 --> 00:30:57,580 okay you will take this as a pivot and then proceed with making you know the matrix to 217 00:30:57,580 --> 00:30:59,330 be lower triangular 218 00:30:59,330 --> 00:31:26,480 so your next step would be 00001 and 1 0 -9999 x1 x2 so this will 219 00:31:26,480 --> 00:31:39,680 give you x2=this and then you will back substitute and get x1 right so we round this off to 1 220 00:31:39,680 --> 00:31:45,580 let us say you have a computer which has finite digit position so you round this off to 1 221 00:31:45,580 --> 00:31:55,410 okay 0999 will get rounded off to 1 if this is rounded off to 1 what is the solution for 222 00:31:55,410 --> 00:32:09,230 x1 if this is rounded off to 1 what will happen x1 will be 0 okay this is because of the rounding 223 00:32:09,230 --> 00:32:10,230 off error 224 00:32:10,230 --> 00:32:18,970 now what is the exact what is the correct way of doing calculations here well i will 225 00:32:18,970 --> 00:32:28,410 do maximal pivoting you know what is maximal pivoting? in gaussian elimination what you 226 00:32:28,410 --> 00:32:39,460 do you rearrange the rows such that the diagonal elements are maximum so if i do maximal pivoting 227 00:32:39,460 --> 00:32:59,400 this will be 1 1 00001 1 x1 x2=2 1 my modified problem same system okay i have done maximal 228 00:32:59,400 --> 00:33:11,620 pivoting i have moved this pivot here 1 now do gaussian elimination okay so now if i do 229 00:33:11,620 --> 00:33:29,090 gaussian elimination here i will get 1 1 0 09999 okay 230 00:33:29,090 --> 00:33:47,750 what is x2 right what is x1 now? if x2=1 x1=1 you see the problem here same system of equations 231 00:33:47,750 --> 00:33:59,700 i re-order the calculation once i got solution 0 and 1 here i got solution 1 and 1 rounding 232 00:33:59,700 --> 00:34:06,110 off okay what mattered here this is the correct solution 1 1 is the correct solution what 233 00:34:06,110 --> 00:34:12,490 mattered was way i ordered my calculations if i do wrong ordering my calculations this 234 00:34:12,490 --> 00:34:19,320 particular matrix is not ill-conditioned matrix it is well-conditioned matrix okay 235 00:34:19,320 --> 00:34:23,010 why this well-conditioned matrix will come to that little later but right now just take 236 00:34:23,010 --> 00:34:27,540 it for granted this is a well-conditioned matrix what went wrong was we did ordering 237 00:34:27,540 --> 00:34:36,500 of calculations and okay we got a solution which is not acceptable but we did not recognized 238 00:34:36,500 --> 00:34:41,240 it of course that type we thought everything is logical we did rounding off there and then 239 00:34:41,240 --> 00:34:47,520 we got a solution which is but here when you correct the order of calculations you recover 240 00:34:47,520 --> 00:34:55,370 the solution 1 1 and then okay 241 00:34:55,370 --> 00:35:10,160 now let me take another system now second system my system 2 is this 1 1 1 10001 x1 242 00:35:10,160 --> 00:35:25,200 x2=2 2 okay if i apply you know first step in the gaussian 243 00:35:25,200 --> 00:35:37,310 elimination this will be 0 this will be 00001 right i just did gaussian elimination 2 times 244 00:35:37,310 --> 00:35:47,480 right so i just subtracted this row from this row this from this i got 0 on this side right 245 00:35:47,480 --> 00:36:08,410 and then what is the solution x2=0 and x1=2 okay is this okay? okay 246 00:36:08,410 --> 00:36:21,720 what i am going to do now i am going to say that well this was a slight error on representing 247 00:36:21,720 --> 00:36:27,110 the right hand side okay now you would say well how does it matter if this slight 10 248 00:36:27,110 --> 00:36:33,480 to the power-4 okay i make an error of 10 to the power-4 on the right hand side so instead 249 00:36:33,480 --> 00:36:52,310 of presenting 2 and 2 you know i ended up representing 2 and 20001 okay instead of typing 250 00:36:52,310 --> 00:36:59,070 2 and 2 let us say i typed 2 and 00001 you would expect what is there if these are these 251 00:36:59,070 --> 00:37:06,220 are 2 very close vectors error is of the order of just see what i will happen 252 00:37:06,220 --> 00:37:25,600 if this was instead by mistake you know represented as 20001 this will become what next if this 253 00:37:25,600 --> 00:37:42,820 happens my x1 will be=1 because 00001/00001 okay x1 will become 1 what will be x2? or 254 00:37:42,820 --> 00:37:57,450 the other way around suppose this was you know 19999 it will be x1 will be-1 x 2 will 255 00:37:57,450 --> 00:38:07,110 be 3 so a slight error on the right hand side slight 10 to the power-4 which you think you 256 00:38:07,110 --> 00:38:12,040 know looking at these numbers 10 to the power-4 we are working with 1 1 1 2 and all that 10 257 00:38:12,040 --> 00:38:13,560 to the power-4 error 258 00:38:13,560 --> 00:38:22,960 but such a small error is causing a drastic change in the solution such a small error 259 00:38:22,960 --> 00:38:27,200 this will not happen for system 1 system once you reorder the calculations if you perturb 260 00:38:27,200 --> 00:38:34,830 the right hand side slightly it is not going to change the solution too much okay here 261 00:38:34,830 --> 00:38:45,350 if you have a solution which is if you have the right hand side which is slightly perturbed 262 00:38:45,350 --> 00:38:57,400 okay you have a problem okay the solution changes drastically okay so from 0 to 2 it 263 00:38:57,400 --> 00:39:02,190 changes to 1 to 1 it is not a small change right not small change 264 00:39:02,190 --> 00:39:08,220 so this problem what we will show a little later is that this is because whatever you 265 00:39:08,220 --> 00:39:14,250 try to do here you rearrange the calculations you do by pivoting well here there is no scope 266 00:39:14,250 --> 00:39:19,510 for pivoting both pivots are 1 1 so there is no question of maximal pivoting but here 267 00:39:19,510 --> 00:39:23,062 nothing is going to change this problem there is a slight error in the representation on 268 00:39:23,062 --> 00:39:29,220 the right hand side you will get drastically different solution okay so this problem is 269 00:39:29,220 --> 00:39:32,390 ill-conditioned no reordering is going to help you 270 00:39:32,390 --> 00:39:37,530 the first problem is not ill-conditioned it is a problem of you know reordering the calculations 271 00:39:37,530 --> 00:39:45,480 to get the correct solution okay now i want measure that will separate these 2 situations 272 00:39:45,480 --> 00:39:50,560 where i can say well this a matrix is ill-conditioned whatever i do i will get into trouble this 273 00:39:50,560 --> 00:39:56,430 a matrix is well-conditioned if my solutions are absurd i have made an error okay which 274 00:39:56,430 --> 00:40:00,060 is not the case for the second case second case solutions are absurd it is because there 275 00:40:00,060 --> 00:40:03,400 is a limitation we will continue in the next lecture