1

Convergence Behavior of Left Preconditioning Techniques for GMRES

ECS 231: Large Scale Scientific Computing University of California, Davis Winter Quarter 2013 March 20, 2013 Joshua Zorn jezorn@ucdavis.edu

2 Abstract: Jacobi, successive over relaxation (SOR), and incomplete LU factorization (ILU) preconditioner matrices are used to alter the convergence behavior of the generalized minimum residual (GMRES) algorithm. The use of these preconditioners significantly reduced the number of iterations needed to solve linear systems. The most consistent preconditioner observed is the ILU preconditioner; in the most extreme case. The ILU preconditioner reduced the number of iterations needed to solve a linear system by 1000. When useable, the SOR preconditioner is inconsistent, but performs just as well as the ILU on the SAYLR1 matrix. However, the Jacobi preconditioner performs the poorest, failing to reduce the number iterations needed to solve any system significantly. Introduction: The GMRES algorithm is a basic method for solving linear systems. The basic algorithm takes advantage of subspace projection theory to implicitly solve linear systems. Unfortunately for some linear systems, the convergence can be slow, and the benefit of using an implicit solver is lost. The issue is addressed through preconditioning. Often the convergence behavior is dependent on condition number. A relatively large condition number is indicative of poor convergence behavior. A small condition number is indicative of good convergence behavior. Theory: The GMRES algorithm is developed to solve simple linear systems shown below in which A is a matrix, b is a vector of known quantities, and x is an unknown vector: Equation 1 A leftward preconditioner modifies this linear system in the manner seen in equation 2. Equation 2 (Bai, 2013) The ideally, the product of M and A is the Identity matrix. Unfortunately, if M where equal to A, the computational cost would be too expensive, and the linear system would be solved before the implicit algorithm is implemented. Therefore, approximate values of A are chosen for M. There are three types of preconditioners that can be applied to any linear system. The first and simplest is the Jacobi preconditioner. A Jacobi preconditioner is simply the diagonal of the matrix in the linear system, as seen in equation 3. This preconditioner fails in the event that any of the diagonal elements of the system matrix is zero. Equation 3 (Ferronato, 2012) The SOR preconditioner assumes that the system that the system matrix decomposes in the manner below in which D is a diagonal matrix, L is a lower triangular matrix, and U is an upper triangular matrix. Equation 4 (Ferronato, 2012) The SOR method uses the decomposition above to form the preconditioner below. This method also only works if all the diagonal elements are non-zero.

Equation 5 (Ferronato, 2012) The ILU preconditioner is based on the idea that each matrix has an LU decomposition; however, to form an LU decomposition, flops are needed. Therefore, another similar decomposition is used, shown in equation 6. In this decomposition L is a lower triangular matrix, U is an upper triangular matrix, and E is an error matrix off the true LU decomposition. This is a significantly faster algorithm that scales by . Equation 6 (Bai, 2013) The ILU method uses the decomposition above to form the decomposition below. This method works on any non-singular linear system. Equation 7 (Bai, 2013) Algorithms: The algorithm to build the SOR preconditioner is fairly simple show below: 1: 2: 3: 4: 5: 6: 7: 8: 9: Take in parameter for i=1 to n for j=1, i-1 end for j end for i output and matrix A

The algorithm above is implemented in the MATLAB code shown in appendix C. A MATLAB function, luinc, is used to perform the incomplete LU factorization. The preconditioned GMRES algorithm is shown below. 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: Take in A, b form initial guess and Loop for convergence or maximum iteration if

, then exit convergence loop

for j=1 to m for i=1 to j

end for i if

, then exit convergence loop

4 14: 15: 16: 17: end for j solve

end convergence loop

One may want to note that in order to implement a GMRES algorithm without preconditioning, one can use the identity matrix for . The algorithm above is implemented in MATLAB codes shown in appendices A and B. The code in Appendix A is the core of the algorithm. Appendix B is a template which shows where the preconditioner matrix is built, and holds the outer loop. The program is written like this so it can be modified and readapted for different purposes. In its current form, the program can easily be altered to include restarts. Implementation: Please refer to appendix D for all figures. Figure 1 shows the sparsity pattern of the four matrices observed, WEST0479, MAHINDAS, BFW398 and SAYLR1. WEST0479 and MAHINDAS were chosen due to their use in previous tests; however, the have zeros in their diagonals, which makes SOR and Jacobi preconditioners impossible to use. Due to their full diagonals, BFW398 and SAYLR1 are used to observe the behavior all three preconditioners. Despite their symmetric looking sparsity patterns, BFW398 and SAYLR1 are non-symmetric (National Institute of Standards and Technology, 2013). This asymmetry is verified by the expression below, equation 8, of the LU decomposition, which yields nonzero values for both test matrices. If the matrices were symmetric, equation 8 would yield a zero. Equation 8

Furthermore, the SAYRL1 and BFW398 are chosen for their condition numbers. SAYLR1 has a fairly large condition number of ; meanwhile BFW398 is chosen for its significantly smaller condition number of 7.5755E3. Due to its small condition number, BFW398 should have a better convergence behavior than SAYLR1. Finally, the values for the vector, b, are chosen such the solution of the vector xm, is a vector full of ones. This is done to easily verify that the algorithm converged on the correct solution. Second, a guess vector is generated using MATLABs rand function. Because this function only yield values between zero and one, output of the rand function is multiplied by 100 to yield bad guesses. This can be seen in appendix B. Results: Figures 2, 3, and 4 show the convergence behaviors of the GMRES algorithm with ILU preconditioners with different drop tolerances for the WEST0497, MAHINDAS, and SAYLR1. One important feature of these graphs is the different curves for different drop tolerances. Typically the greater the drop tolerance, the slower the convergence, but the quicker the preconditioner is built. One issue that arises is that if the drop tolerance is too large, the preconditioner can become singular. This results in instantaneous divergence. Figure 2 shows a case in which a large drop tolerance results in slower convergence behavior. Figure 5 compares convergence behaviors of the SOR method for different values of on the SAYLR1 matrix. It is important that different values of where tested on BFW398; however little difference in convergence behavior is observed with changing values of . A divergence behavior is observed with a value greater than 10. For SAYLR1, the smaller the value of used, the faster the rate of convergence. is deceased by factors of 10 until convergence is obtained in one iteration, which happened to be =10-4. The benefit of the SOR method is that computational cost of building a

5 preconditioner is not changed as is changed, but the ideal is system dependent. Figures 6 and 7 depict comparisons between the convergence behaviors of systems using different preconditioners. Finally, although BFW398 is a larger matrix with more non-zero values than SAYLR1, BFW398 converges in fewer iterations than SAYLR1 in the basic GMRES algorithm. BFW398 converges in approximately 40 iterations; meanwhile SAYRL1 converges in about 100 iterations. This is consistent with the condition numbers of the matrices. Conclusion: The most effective preconditioner is the ILU preconditioner. Unfortunately the ILU is the most expensive computationally. The cheapest preconditioner in flops and locations in memory is the Jacobi preconditioner. In MATLAB, the Jacobi Preconditioner is a matrix, but with effective programing, diagonal Jacobi can be reduced down to a vector. However, the Jacobi preconditioner does very little to improve performance. In linear systems that take a couple hundred of iterations to solve, the Jacobi preconditioner only reduced the number of necessary iterations to reach a solution by at most 30. A nice medium between the two is the SOR method, but the SOR method has the parameter, , which has problem dependent effects. If one needs to solve a system fast without any knowledge about the system, the ILU preconditioner is the best. Finally, based on the convergence behavior of the basic GMRES algorithm, condition number is a significant indicator of convergence behavior. The matrices with the largest condition numbers, WEST0479 and MAHINDAS, converged the slowest and needed preconditioning for the basic GMRES algorithm to converge in a reasonable number of iterations. Meanwhile, SAYLR1 and BFW398, matrices with smaller condition numbers, converged with the basic GMRES algorithm without preconditioning. The fastest convergence behavior is observed with BFW398. Overall, the preconditioning can vastly improve the convergence behavior of GMRES when using matrices with very large condition numbers. This is observed with MAHINDAS and WEST0479. In the unconditioned algorithm, convergence is not obtained until the subspace size is approximately the same size as the subspace; however, with preconditioning, these systems converge on the second iteration. Bibliography Bai, Z. (2013, March 12). Preconditioning Techniques. Retrieved from ECS 231, Winter 2013: http://www.cs.ucdavis.edu/~bai/ECS231/ Ferronato, M. (2012). Preconditioning for Sparse Linear Systems at the Dawn of the 21st Century: History, Current Developments, and Future Perspectives. ISRN Applied Mathematics. National Institute of Standards and Technology. (2013, 3 20). Matrix Market. Retrieved from Math, Statistics, and Computational Science: http://math.nist.gov/MatrixMarket/ Appendix A: Preconditioned GMRES MATLAB codefunction
[x_m,r,i]=myLPgmres(A,x_m,b,M_prime,m,eta,tol) %A=system matrix %b=system input %x_o=first guess %M_prime=Preconditioner: written as M^-1 in many algorithms %m=subspace size %eta is maximum sub iteration %tol=convergence tolerance %overall: M'Ax=M'b

6
%r=residual vector: used to plot convergence behavoir %x_s=final x from process %i=iteration of completion %allocation phase n=size(A); h=zeros(n(1)+1,n(1)); v=zeros(n(1),n(1)); v_mplus=zeros(n(1),n(1)+1); r=zeros(eta,1); %preliminary phase r_m=M_prime*(b-A*x_m); beta=norm(r_m,2); e1=zeros(n(1)+1,1); e1(1)=1; %convergence loop for i=1:eta %arnoldi procedure %not written as subroutine to %facilitate lucky breakdown v_mplus(:,1)=r_m/beta; for j=1:m v(:,j)=v_mplus(:,j); omega=M_prime*(A*v(:,j)); for k=1:j h(k,j)=v(:,k)'*omega; omega=omega-h(k,j)*v(:,k); end h(j+1,j)=norm(omega,2); %lucky breakdown check if (h(j+1,j)==0) return; end v_mplus(:,j+1)=omega/h(j+1,j); end y=h\(beta*e1); x_m=x_m+v*y; r_m=M_prime*(b-A*x_m); beta=norm(r_m,2); r(1)=beta/norm(x_m,2); if (r(i)<tol) return; end end

Appendix B: Outer Loop and Preconditioner Build for Jacobi MATLAB CODE
function [x_m,r,i]=myLPJgmres(A,b,tol) %my Jacobi GMRES: other programs have their inputs (omega, droptolerance) %input %A=system matrix %b=system input %m=subspace size %tol is solution tolerance

7
%output %x_m=final x from process %i=iteration of completion %M_prime=inverse(jacobi preconditioner) n=size(A); %this step is different for each %preconditioner type M_prime=(diag(diag(A)))^-1; r=zeros(n(1)-9,1); i=zeros(n(1)-10,1); %automatically generates first guess: attempt to be bad x_m=rand(n(1),1)*100; r(1)=norm(M_prime*(b-A*x_m),2)/norm(x_m,2); %calls gerneral left preconditioned GMRES %repeat loop until convergence reached %or subspace is system size for m=1:(n(1)-10) [x_m,r(m+1),i(m)]=myLPgmres(A,x_m,b,M_prime,10+m,1,tol); if(r(m+1)<tol) return; end end end

Appendix C: SOR Preconditioner Build MATLAB code

function [M_prime]=mySOR(A,omega) %builds preconditioner matrix using SOR n=size(A); M_prime=zeros(n(1),n(2)); D=zeros(n(1),n(2)); L=zeros(n(1),n(2)); for l=1:n(1) D(l,l)=A(l,l); for m=1:l-1 L(l,m)=-A(l,m); end end M_prime=((1/omega)*D-L)^-1; end

Appendix D: Figures

Figure 1: Sparsity pattern of matrices used. SAYLR1 has a condition number of . WEST0479 has a condition number of 1.4244E12. MAHINDAS has a condition number of 1.0343E13. BFW398 has a condition number of 7.5755E3 (National Institute of Standards and Technology, 2013)

9 Figure 2: ILU Convergence behavior of WEST0479 in comparison to basic GMRES performance. Blue is unconditioned GMRES baseline. Red is with a drop-tolerance of 0, magenta has drop tolerance of 10-7, green has drop-tolerance of 10-6, black shows behavior with a drop-tolerance of 10-5.

Figure 3: ILU convergence behavior of MAHINDAS in comparison to basic GMRES. Blue is unconditioned GMRES baseline. Red is with a drop-tolerance of 0, magenta has drop tolerance of 10-10, green has droptolerance of 10-9, black shows behavior with a drop-tolerance of 10-8.

Figure 4: ILU convergence behavior of SAYLR1 in comparison to basic GMRES. Blue is the unconditioned baseline GMRES behavior. Black is preconditioned with a drop tolerance of 10-1, magenta is

10 preconditioned with a drop tolerance to 10-2, and blue is preconditioned with a drop tolerance of 10-3. Red is conditioned with a drop tolerance of zero.

Figure 5: SOR convergence behavior of SAYLR1 in comparison to basic GMRES. Blue is unconditioned GMRES baseline. The rest are SOR convergence behaviors. =1 for red, =.1 for black, =.01 for magenta, =.001 for green, and =.0001 for cyan.

Figure 6: Comparison of different preconditioner convergence behaviors with SAYLR1. Blue is unconditioned baseline GMRES. Black is convergence behavior with Jacobi Preconditioning. Green is convergence with SOR in which =.0001, Red is convergence with ILU preconditioning with no drop tolerance.

11

Figure 7: Comparison of different preconditioner convergence behaviors with BFW398. Blue is unconditioned GMRES baseline. Red is SOR with =1, magenta is the convergence behavior with a Jacobi preconditioned; green show ILU convergence behavior with drop-tolerance of 10-1, cyan shows behavior with a drop-tolerance of 10-2, Black shows an ILU preconditioned with drop tolerance of 0.