From the series: Differential Equations and Linear Algebra
Gilbert Strang, Massachusetts Institute of Technology (MIT)
dy/dt = Ay contains solutions y = eλtx where λ and x are an eigenvalue / eigenvector pair for A.
So this is the key video about solving a system of n linear constant coefficient equations. So how do I write those equations? Y is now a vector, a vector with n components. Instead of one scalar, just a single number y-- do you want me to put an arrow on y? No, I won't repeat it again. But that's to emphasize that y is a vector. Its first derivative, it's a first order system. System meaning that there can be and will be more than one unknown, y1, y2, to yn.
So how do we solve such a system? Then the matrix is multiplying that y and they equate. The y's are coupled together by that matrix. They're coupled together, and how do we uncouple them? That is the magic of eigenvalues and eigenvectors.
Eigenvectors are vectors that go in their own way. So when you have an eigenvector, it's like you have a one by one problem and the a becomes just a number, lambda. So for a general vector, everything is a mixed together. But for an eigenvector, everything stays one dimensional. The a changes just to a lambda for that special direction.
And of course, as always, we need n of those eigenvectors because we want to take the starting value. Just as we did for powers, we're doing it now for differential equations. I take my starting vector, which is probably not an eigenvector. I'd make it a combination of eigenvectors. And I'm OK because I'm assuming that I have n independent eigenvectors.
And now I follow each starting value c1 x1-- what does that have? What happens if I'm in the direction of x1, then all the messiness of A disappears. It acts just like lambda 1 on that vector x1. Here's what you get. You get c1, that's just a number, times e to the lambda 1t x1. You see there, instead of powers, which we had-- that we had lambda 1 to the kth power when we were doing powers of a matrix, now we're solving differential equations. So we get an e to the lambda 1t.
And of course, next by superposition, I can add on the solution for that one, which is e to the lambda 2t x2 plus so on, plus cne to the lambda nt xn. You can see when, I could ask, when is this stable? When do the solutions go to 0? Well, as t gets large, this number will go to 0, provided lambda 1 is negative. Or provided its real part is negative. We can understand everything from this piece by piece formula.
Let me just do an example. Take a matrix A. In the powers of a matrix-- in that video I took a Markov matrix-- let me take here the equivalent for differential equations. So this will give us a Markov differential equation. So let me take A now.
The columns of a Markov matrix add to 1 but in the differential equation situation, they'll add to 0. Like minus 1 and 1, or like minus 2 and 2. So there is the eigenvalue of 1 for our powers is like the eigenvalue 0 for differential equations. Because e to the 0t is 1.
So anyway, let's find the eigenvalues of that. The first eigenvalue is 0. That's what I'm interested in. That column adds to 0, that column adds to 0. That tells me there's an eigenvalue of 0.
And what's its eigenvector? Probably 2, 1 because if I multiply that matrix by that vector, I get 0. So lambda 1 is 0. And my second eigenvalue, well the trace is minus 3 and the lambda 1 plus lambda 2 must give minus 3. And its eigenvector is-- it's probably 1 minus 1 again.
So I've done the preliminary work. Given this matrix, we've got the eigenvalues and eigenvectors. Now I take u0-- what should we say for u0? U0-- y0, say y of 0 to start. Y of 0 as some number c1 times x1 plus c2 times x2. Yes, no problem, no problem. Whatever we have, we take this-- some combination of that vector and that eigenvector will give us y of 0.
And now the y of t is c1 e to the 0t-- e to the lambda 1t times x1, right? You see, we started with c1x1 but after a time t, it's either the lambda t and here's c2. E to the lambda 2 is minus 3t x2. That's the evolution of a Markov process, a continuous Markov process. Compared to the powers of a matrix, this is a continuous evolving evolution of this vector.
Now, I'm interested in steady state. Steady state is what happens as t gets large. As t gets large, that number goes quickly to 0. In our Markov matrix example, we had 1/2 to a power, and that went quickly to 0. Now we have the exponential with a minus 3, that goes to zero. E to the 0t is the 1. This e to the 0t equals 1. So that 1 is the signal of a steady state. Nothing changing, nothing really depending on time, just sits there. So I have c1x1 is the steady state.
And x1 was this. So what am I thinking? I'm thinking that no matter how you start, no matter what y of 0 is, as time goes on, the x2 part is going to disappear. And if you only have the x1 part in that ratio 2:1. So again, if we had movement between Y1 Y2 or we have things evolving in time, the steady state is-- this is the steady state.
There is an example of a differential equation, happen to have a Markov matrix. And with a Markov matrix, then we know that we'll have an eigenvalue of - in the continuous case and a negative eigenvalue that will disappear as time goes forward. E to the minus 3t goes to 0. Good.
I guess I might just add a little bit to this video, which is to explain why is 0 an eigenvalue when whenever-- if the columns added to 0, minus 1 plus 1 is 0. 2 minus 2 is zero. That tells me 0 is an eigenvalue. For a Markov matrix empowers the columns added to 1 and 1 was an eigenvalue.
So I guess I have now two examples of the following fact. That if all columns add to some-- what shall I say for the sum, s for the sum-- then lambda equal s is an eigenvalue. That was the point from Markov matrices, s was 1. Every column added to 1 and then lambda equal 1 was an eigenvalue. And for this video, every column added to 0 and then lambda equal 0 was an eigenvalue.
And also, this is another point about eigenvalues, good to make. The eigenvalues of a transpose are the same as the eigenvalues of A. So I could also say if all rows of A add to s, then lambda equal s is an eigenvalue. I'm saying that the eigenvalues of a matrix and the eigenvalues of the transpose are the same. And maybe you would like to just see why that's true.
If I want the eigenvalues of a matrix, I look at the determinant of lambda I minus A. That gives me eigenvalues of A. If I want the eigenvalues of a transpose, I would look at this equals 0, right? This equaling 0. That equation would give me the eigenvalues of a transpose just the way this one gives me the eigenvalues of A.
But why are they the same? Because the determinant of a matrix and the determinant of its transpose are equal. A matrix and its transpose have the same determinant. Let me just check that with A, B, C, D. And the transpose would be A, C, B, D. And the determinant in both cases is AD minus BC, AD minus BC. Transposing doesn't affect.
So this, that is the same as that. And the lambdas are the same. And therefore we can look at the columns adding to s or the rows added to s.
So this explains why those two statements are both true together because I could look at the rows or the columns and reach this conclusion. That if all columns add to s-- now why is that, or all rows add to s? Let me just-- I'll just show you the eigenvector.
In this case, A times the eigenvector would be all ones. Suppose the matrix is 4 by 4. If I multiply A by all ones, when you multiply a matrix by a vector of ones, then the dot product of this row with that is the sum, is that plus that plus that plus that, would be we s. This would be s because this first row-- here is A-- first row of A adds to s.
So these numbers add to s, I get s. These numbers add to s, I get s again. These numbers add to s. And these, finally those numbers add to s. And I have s times 1, 1, 1, 1.
Are you OK with this? When all the rows add to s, I can tell you what the eigenvector is, 1, 1, 1, 1. And then the eigenvalue, I can see that that's the sum s. So again, for special matrices, in this case named after Markov, we are able to identify important fact about their eigenvalue, which is that it's that common row sum s equal 1 in the case of powers and s equal 0 in this video's case with-- let me bring down A again.
So here, every column added to 0. It didn't happen that the rows added to 0. I'm not requiring that. I'm just saying either way, A or A transpose has the same eigenvalues and one of them is 0 and the other is whatever the trace tells us, that one.
These collection of useful fact about eigenvalues show up when you have a particular matrix and you need to know something about its eigenvalues. Good, thank you.
1.1 Overview of Differential Equations Linear equations include dy/dt = y, dy/dt = –y, dy/dt = 2ty . The equation dy/dt = y *y is nonlinear.
1.2 The Calculus You Need The sum rule, product rule, and chain rule produce new derivatives from the derivatives of xn , sin(x ) and ex . The Fundamental Theorem of Calculus says that the integral inverts the derivative.
1.4b Response to Exponential Input exp(s*t) With exponential input, est , from outside and exponential growth, eat , from inside, the solution, y(t), is a combination of two exponentials.
1.4c Response to Oscillating Input cos(w*t) An oscillating input cos(ωt ) produces an oscillating output with the same frequency ω (and a phase shift).
1.4d Solution for Any Input q(t) To solve a linear first order equation, multiply each input q(s) by its growth factor and integrate those outputs.
1.4e Step Function and Delta Function A unit step function jumps from 0 to 1. Its slope is a delta function: zero everywhere except infinite at the jump.
1.5 Response to Complex Exponential exp(i*w*t) = cos(w*t)+i*sin(w*t) For linear equations, the solution for f = cos(ωt ) is the real part of the solution for f = eiωt . That complex solution has magnitude G (the gain).
1.6 Integrating Factor for a Constant Rate a The integrating factor e-at multiplies the differential equation, y’=ay+q, to give the derivative of e-at y: ready for integration.
1.6b Integrating Factor for a Varying Rate a(t) The integral of a varying interest rate provides the exponent in the growing solution (the bank balance).
1.7 The Logistic Equation When –by2 slows down growth and makes the equation nonlinear, the solution approaches a steady state y( ∞) = a/b.
1.7c The Stability and Instability of Steady States Steady state solutions can be stable or unstable – a simple test decides.
2.1 Second Order Equations For the oscillation equation with no damping and no forcing, all solutions share the same natural frequency.
2.1b Forced Harmonic Motion With forcing f = cos(ωt ), the particular solution is Y *cos(ωt ). But if the forcing frequency equals the natural frequency there is resonance.
2.3 Unforced Damped Motion With constant coefficients in a differential equation, the basic solutions are exponentials est . The exponent s solves a simple equation such as As2 + Bs + C = 0 .
2.3c Impulse Response and Step Response The impulse response g is the solution when the force is an impulse (a delta function). This also solves a null equation (no force) with a nonzero initial condition.
2.4 Exponential Response - Possible Resonance Resonance occurs when the natural frequency matches the forcing frequency — equal exponents from inside and outside.
2.4b Second Order Equations With Damping A damped forced equation has a particular solution y = G cos(ωt – α). The damping ratio provides insight into the null solutions.
2.5 Electrical Networks Voltages and Currents Current flowing around an RLC loop solves a linear equation with coefficients L (inductance), R (resistance), and 1/C (C = capacitance).
2.6 Methods of Undetermined Coefficients With constant coefficients and special forcing terms (powers of t , cosines/sines, exponentials), a particular solution has this same form.
2.6b An Example of Method of Undetermined Coefficients This method is also successful for forces and solutions such as (at2 + bt +c) est : substitute into the equation to find a, b, c .
2.6c Variations of Parameters Combine null solutions y1 and y2 with coefficients c1(t) and c2(t) to find a particular solution for any f(t).
2.7 Laplace Transform First Order Equation Transform each term in the linear differential equation to create an algebra problem. You can then transform the algebra solution back to the ODE solution, y(t) .
2.7b Laplace Transform Second Order Equation The second derivative transforms to s2Y and the algebra problem involves the transfer function 1/ (As2 + Bs +C).
3.1 Pictures of the Solutions The direction field for dy/dt = f(t,y) has an arrow with slope f at each point t, y . Arrows with the same slope lie along an isocline.
3.2 Phase Plane Pictures Source Sink Saddle Solutions to second order equations can approach infinity or zero. Saddle points contain a positive and also a negative exponent or eigenvalue.
3.2b Phase Plane Pictures Spirals and Centers Imaginary exponents with pure oscillation provide a “center” in the phase plane. The point (y, dy/dt) travels forever around an ellipse.
3.2c Two First Order Equations Stability A second order equation gives two first order equations for y and dy/dt . The matrix becomes a companion matrix.
3.3 Linearization at Critical Points A critical point is a constant solution Y to the differential equation y’ = f(y) . Near that Y , the sign of df/dy decides stability or instability.
3.3b Linearization of y'=f(yz) and z'=g(yz) With two equations, a critical point has f(Y,Z) = 0 and g(Y,Z) = 0. Near those constant solutions, the two linearized equations use the 2 by 2 matrix of partial derivatives of f and g .
3.3c Eigenvalues and Stability 2 by 2 Matrix A Two equations y’ = Ay are stable (solutions approach zero) when the trace of A is negative and the determinant is positive.
5.1 The Column Space of a Matrix A An m by n matrix A has n columns each in R m . Capturing all combinations Av of these columns gives the column space – a subspace of R m .
5.4 Independence Basis and Dimension Vectors v 1 to v d are a basis for a subspace if their combinations span the whole subspace and are independent: no basis vector is a combination of the others. Dimension d = number of basis vectors.
5.5 The Big Picture of Linear Algebra A matrix produces four subspaces – column space, row space (same dimension), the space of vectors perpendicular to all rows (the nullspace), and the space of vectors perpendicular to all columns.
5.6 Graphs A graph has n nodes connected by m edges (other edges can be missing). This is a useful model for the Internet, the brain, pipeline systems, and much more.
6.1 Eigenvalues and Eigenvectors The eigenvectors x remain in the same direction when multiplied by the matrix (A x = λx ). An n x n matrix has n eigenvalues.
6.2 Diagonalizing a Matrix A matrix can be diagonalized if it has n independent eigenvectors. The diagonal matrix Λis the eigenvalue matrix.
6.3 Solving Linear Systems d y /dt = A y contains solutions y = eλt x where λ and x are an eigenvalue / eigenvector pair for A .
6.4 The Matrix Exponential exp(A*t) The shortest form of the solution uses the matrix exponential y = eAt y (0) . The matrix eAt has eigenvalues eλt and the eigenvectors of A.
6.4b Similar Matrices A and B=M^(-1)*A*M A and B are “similar” if B = M-1AM for some matrix M . B then has the same eigenvalues as A .
6.5 Symmetric Matrices Real Eigenvalues Orthogonal Eigenvectors Symmetric matrices have n perpendicular eigenvectors and n real eigenvalues.
7.2 Positive Definite Matrices S=A'*A A positive definite matrix S has positive eigenvalues, positive pivots, positive determinants, and positive energy vT Sv for every vector v. S = AT A is always positive definite if A has independent columns.
7.2b Singular Value Decomposition SVD The SVD factors each matrix A into an orthogonal matrix U times a diagonal matrix Σ (the singular value) times another orthogonal matrix VT : rotation times stretch times rotation.
7.3 Boundary Conditions Replace Initial Conditions A second order equation can change its initial conditions on y(0) and dy/dt(0) to boundary conditions on y(0) and y(1) .
8.1 Fourier Series A Fourier series separates a periodic function F(x) into a combination (infinite) of all basis functions cos(nx) and sin(nx) .
8.1b Examples of Fourier Series Even functions use only cosines (F(–x) = F(x) ) and odd functions use only sines. The coefficients an and bn come from integrals of F(x) cos(nx ) and F(x) sin(nx ).
8.1c Fourier Series Solution of Laplace's Equation Inside a circle, the solution u (r , θ) combines rn cos(n θ) and rn sin(n θ). The boundary solution combines all entries in a Fourier series to match the boundary conditions.
8.3 Heat Equation The heat equation ∂u /∂t = ∂2u /∂x2 starts from a temperature distribution u at t = 0 and follows it for t > 0 as it quickly becomes smooth.