\chapter{Sparse Linear Equations} \label{c-spsolvers}
In this section we describe routines for solving sparse sets of linear 
equations.

A real symmetric or complex Hermitian sparse matrix is stored as
an \spmtrx\ object \var{X}  of size (\tm{n}, \tm{n}) and an 
additional character argument \code{uplo} with possible values 
\code{'L'} and \code{'U'}.  
If \code{uplo} is \code{'L'}, the lower triangular part
of \var{X} contains the lower triangular part of the
symmetric or Hermitian matrix, and the upper triangular matrix
of \var{X} is ignored.
If \code{uplo} is \code{'U'}, the upper triangular part
of \var{X} contains the upper triangular part of the
matrix, and the lower triangular matrix of \var{X} is ignored.

A general sparse square matrix of order \tm{n} is represented by an
\spmtrx\ object of size (\tm{n}, \tm{n}).

Dense matrices, which appear as righthand sides of equations, are 
stored using the same conventions as in the BLAS and LAPACK modules.

\section{Matrix Orderings (\module{cvxopt.amd})} \label{s-orderings}

CVXOPT includes an interface to the AMD library for computing 
approximate minimum degree orderings of sparse matrices.

\textbf{See also:}
\BIT
\item \seelink{http://www.cise.ufl.edu/research/sparse/amd}{AMD code, 
documentation, copyright and license.}{}
\item \seetext{P.\ R.\ Amestoy, T.\ A.\ Davis, I.\ S.\ Duff,
Algorithm 837: AMD, An Approximate Minimum Degree Ordering Algorithm,
ACM Transactions on Mathematical Software, 30(3), 381-388, 2004.}
\EIT

\begin{funcdesc}{order}{A\optional{, uplo='L'}}
Computes the approximate mimimum degree ordering of a symmetric  sparse
matrix \tm{A}.  
The ordering is returned as an integer dense matrix with length equal 
to the order of \var{A}.  Its entries specify a permutation that 
reduces fill-in during the Cholesky factorization.
More precisely, if \code{\var{p} = order(\var{A})}, then  
\code{\var{A}[\var{p},\var{p}]} has sparser Cholesky factors 
than \code{\var{A}}.   
\end{funcdesc}

As an example we consider the matrix 
\[
\left[ \begin{array}{rrrr}
 10 &  0 & 3 &  0 \\
  0 &  5 & 0 & -2 \\
  3 &  0 & 5 &  0 \\
  0 & -2 & 0 &  2 
\end{array}\right].
\]
\begin{verbatim}
>>> from cvxopt.base import spmatrix
>>> from cvxopt import amd 
>>> A = spmatrix([10,3,5,-2,5,2], [0,2,1,2,2,3], [0,0,1,1,2,3])
>>> P = amd.order(A)
>>> print P
[ 1]
[ 0]
[ 2]
[ 3]
\end{verbatim}

\section{General Linear Equations (\module{cvxopt.umfpack})} 
\label{s-umfpack}
The module \module{cvxopt.umfpack} includes four functions for solving 
sparse non-symmetric sets of linear equations.  
They call routines from the UMFPACK library, with all control options 
set to the default values described in the UMFPACK user guide.  

\textbf{See also:}
\BIT
\item \seelink{http://www.cise.ufl.edu/research/sparse/umfpack}{UMFPACK code, documentation, copyright and license.}{}
\item \seetext{T.\ A.\ Davis,
Algorithm 832: UMFPACK -- an unsymmetric-pattern multifrontal method 
with a column pre-ordering strategy,
ACM Transactions on Mathematical Software, 30(2), 196-199, 2004.}
\EIT

\begin{funcdesc}{linsolve}{A, B\optional{, trans='N'}}
Solves a sparse set of linear equations 
\[
 AX = B \quad (\mathrm{trans} = \mathrm{'N'}), \qquad
 A^TX = B \quad (\mathrm{trans} = \mathrm{'T'}), \qquad
 A^HX = B \quad (\mathrm{trans} = \mathrm{'C'}),
\]
where {\var A} is a sparse matrix and \var{B} is a dense matrix of
the same type (\dtc\ or \ztc) as \var{A}.  On exit \var{B} contains 
the solution.
Raises an \code{ArithmeticError} exception if the coefficient matrix 
is singular.
\end{funcdesc}
In the following example we solve an equation with 
coefficient matrix 
\BEQ \label{e-sp-Adef}
A = \left[\begin{array}{rrrrr}
 2 & 3 & 0 & 0 & 0 \\
 3 & 0 & 4 & 0 & 6 \\
 0 &-1 &-3 & 2 & 0 \\
 0 & 0 & 1 & 0 & 0 \\
 0 & 4 & 2 & 0 & 1 
 \end{array}\right].
\EEQ
\begin{verbatim}
>>> from cvxopt.base import spmatrix, matrix
>>> from cvxopt import umfpack 
>>> V = [2,3, 3,-1,4, 4,-3,1,2, 2, 6,1]
>>> I = [0,1, 0, 2,4, 1, 2,3,4, 2, 1,4]
>>> J = [0,0, 1, 1,1, 2, 2,2,2, 3, 4,4]
>>> A = spmatrix(V,I,J)
>>> B = matrix(1.0, (5,1))
>>> umfpack.linsolve(A,B)
>>> print B
[ 5.79e-01]
[-5.26e-02]
[ 1.00e+00]
[ 1.97e+00]
[-7.89e-01]
\end{verbatim}
The function \function{umfpack.linsolve()} is equivalent to the 
following three functions called in sequence.  
\begin{funcdesc}{symbolic}{A}
Reorders the columns of \var{A} to reduce fill-in and performs a 
symbolic LU factorization.  \var{A} is a sparse, possibly rectangular,
matrix.
Returns the symbolic factorization as an opaque C object that can be 
passed on to \function{umfpack.numeric()}.
\end{funcdesc}

\begin{funcdesc}{numeric}{A, F}
Performs a numeric LU factorization of a sparse, possibly rectangular,
matrix \var{A}.   The argument \var{F} is the symbolic factorization
computed by \function{umfpack.symbolic()} applied to the matrix \var{A},
or another sparse matrix with the same sparsity pattern, dimensions,
and type.  The numeric factorization is returned as an opaque C object 
that that can be passed on to \function{umfpack.solve()}.  Raises an
\code{ArithmeticError} if the matrix is singular.
\end{funcdesc}

\begin{funcdesc}{solve}{A, F, B\optional{, trans='N'}}
Solves a set of linear equations
\[
 AX = B \quad (\mathrm{trans} = \mathrm{'N'}), \qquad
 A^TX = B \quad (\mathrm{trans} = \mathrm{'T'}), \qquad
 A^HX = B \quad (\mathrm{trans} = \mathrm{'C'}),
\]
where {\var A} is a sparse matrix and \var{B} is a dense matrix of 
the same type as \var{A}.
The argument \var{F} is a numeric factorization computed by 
\function{umfpack.numeric()}.
On exit \var{B} is overwritten by the solution.
\end{funcdesc}
These separate functions are useful for solving several sets
of linear equations with the same coefficient matrix and different 
righthand sides, or with coefficient matrices that share the same 
sparsity pattern.
The symbolic factorization depends only on the sparsity pattern of
the matrix, and not on the numerical values of the nonzero 
coefficients. 
The numerical factorization on the other hand depends on the sparsity 
pattern of the matrix and on its the numerical values.

As an example, suppose \tm{A} is the matrix~(\ref{e-sp-Adef}) and 
\[
B = \left[\begin{array}{rrrrr}
 4 & 3 & 0 & 0 & 0 \\
 3 & 0 & 4 & 0 & 6 \\
 0 &-1 &-3 & 2 & 0 \\
 0 & 0 & 1 & 0 & 0 \\
 0 & 4 & 2 & 0 & 2 
 \end{array}\right],
\]
which differs from \tm{A} in its first and last entries.
The following code computes
\[
 x = A^{-T}B^{-1}A^{-1}\ones.
\]
\begin{verbatim}
>>> from cvxopt.base import spmatrix, matrix
>>> from cvxopt import umfpack
>>> VA = [2,3, 3,-1,4, 4,-3,1,2, 2, 6,1]
>>> VB = [4,3, 3,-1,4, 4,-3,1,2, 2, 6,2]
>>> I =  [0,1, 0, 2,4, 1, 2,3,4, 2, 1,4]
>>> J =  [0,0, 1, 1,1, 2, 2,2,2, 3, 4,4]
>>> A = spmatrix(VA, I, J)
>>> B = spmatrix(VB, I, J)
>>> x = matrix(1.0, (5,1))
>>> Fs = umfpack.symbolic(A)
>>> FA = umfpack.numeric(A, Fs)
>>> FB = umfpack.numeric(B, Fs)
>>> umfpack.solve(A, FA, x)
>>> umfpack.solve(B, FB, x)
>>> umfpack.solve(A, FA, x, trans='T')
>>> print x
[ 5.81e-01]
[-2.37e-01]
[ 1.63e+00]
[ 8.07e+00]
[-1.31e-01]
\end{verbatim}

\section{Positive Definite Linear Equations (\module{cvxopt.cholmod})} 
\label{s-cholmod}

\module{cvxopt.cholmod} is an interface to the Cholesky factorization 
routines of the CHOLMOD package.
It includes functions for Cholesky factorization of sparse positive
definite matrices, and for solving sparse sets of linear equations with 
positive definite matrices. 
The routines can also be used for computing $\mathrm{LDL^T}$ 
(or $\mathrm{LDL^H}$) factorizations of symmetric indefinite matrices 
(with L unit lower-triangular and D diagonal and nonsingular) if 
such a factorization exists.  

\textbf{See also:}
\seelink{http://www.cise.ufl.edu/research/sparse/cholmod}{CHOLMOD code, documentation, copyright and license.}{}

\begin{funcdesc}{linsolve}{A, B\optional{, p=\None\optional{, 
uplo='L'}}}
Solves
\[
 AX = B 
\]
with \tm{A} sparse and real symmetric or complex Hermitian.
\var{B} is a dense matrix of the same type as \var{A}.  On exit it 
is overwritten with the solution.
The argument \var{p} is an integer matrix with length equal to the 
order of \var{A}, and specifies an optional reordering of \var{A}.
If \var{p} is not specified, CHOLMOD used a reordering from the
AMD library.
Raises an \code{ArithmeticError} if the factorization does not exist.
\end{funcdesc}

As an  example, we solve 
\BEQ \label{e-A-pd}
\left[ \begin{array}{rrrr}
 10 &  0 & 3 &  0 \\
  0 &  5 & 0 & -2 \\
  3 &  0 & 5 &  0 \\
  0 & -2 & 0 &  2 
  \end{array}\right] X = \left[ \begin{array}{cc}
   0 & 4 \\ 1 & 5 \\ 2 & 6 \\ 3 & 7\end{array} \right].
\EEQ
\begin{verbatim}
>>> from cvxopt.base import matrix, spmatrix
>>> from cvxopt import cholmod
>>> A = spmatrix([10,3, 5,-2, 5, 2], [0,2, 1,3, 2, 3], [0,0, 1,1, 2, 3])
>>> X = matrix(range(8), (4,2), 'd')
>>> cholmod.linsolve(A,X)
>>> print X
[-1.46e-01  4.88e-02]
[ 1.33e+00  4.00e+00]
[ 4.88e-01  1.17e+00]
[ 2.83e+00  7.50e+00]
\end{verbatim}

\begin{funcdesc}{splinsolve}{A, B\optional{, p=\None\optional{, 
uplo='L'}}}
Similar to \function{linsolve()} except that \var{B} is a sparse 
matrix and that the solution is returned as an output 
argument (as a new sparse matrix).  \var{B} is not modified.
\end{funcdesc}

The following code computes the inverse of the coefficient matrix 
in~(\ref{e-A-pd}) as a sparse matrix.
\begin{verbatim}
>>> X = cholmod.splinsolve(A, spmatrix(1.0,range(4),range(4)))
>>> print X
[ 1.22e-01     0     -7.32e-02     0    ]
[    0      3.33e-01     0      3.33e-01]
[-7.32e-02     0      2.44e-01     0    ]
[    0      3.33e-01     0      8.33e-01]
\end{verbatim}

The functions \function{linsolve()} and \function{splinsolve()} are
equivalent to \function{symbolic()} and \function{numeric()} called in
sequence, followed by \function{solve()}, respectively, 
\function{spsolve()}.

\begin{funcdesc}{symbolic}{A\optional{, p=\None\optional{, uplo='L'}}}
Performs a symbolic analysis of a sparse real symmetric or
complex Hermitian matrix \var{A} for one of the two factorizations:
\BEQ \label{e-chol-ll}
   PAP^T = LL^T, \qquad PAP^T = LL^H, 
\EEQ
and 
\BEQ \label{e-chol-ldl}
   PAP^T = LDL^T, \qquad PAP^T = LDL^H,
\EEQ
where \tm{P} is a permutation matrix, \tm{L} is lower triangular 
(unit lower triangular in the second factorization), and 
\tm{D} is nonsingular diagonal.  The type of factorization depends 
on the value of \code{options['supernodal']} (see below).

If \var{uplo} is \code{'L'}, only the lower triangular part of \var{A} 
is accessed and the upper triangular part is ignored.
If \var{uplo} is \code{'U'}, only the upper triangular part of \var{A} 
is accessed and the lower triangular part is ignored.

The symbolic factorization is returned as an opaque C object that 
can be passed to \function{cholmod.numeric()}.
\end{funcdesc}

\begin{funcdesc}{numeric}{A, F}
Performs a numeric factorization of a sparse symmetric matrix 
as~(\ref{e-chol-ll}) or~(\ref{e-chol-ldl}).  
The argument \var{F} is the symbolic factorization
computed by \function{cholmod.symbolic()} applied to the matrix \var{A},
or to another sparse  matrix with the same sparsity pattern and 
typecode, or by
\function{cholmod.numeric()} applied to a matrix with the same
sparsity pattern and typecode as \var{A}.

If \var{F} was created by a \function{cholmod.symbolic} with 
\var{uplo} equal to \code{'L'}, then only the lower triangular part 
of \var{A} is accessed and the upper triangular part is ignored.
If it was created with \var{uplo} is \code{'U'}, then only the upper 
triangular part of \var{A} is accessed and the lower triangular part 
is ignored.

On successful exit, the factorization is stored in \var{F}.
Raises an \code{ArithmeticError} if the factorization does not
exist.
\end{funcdesc}

\begin{funcdesc}{solve}{F, B\optional{, sys=0}}
Solves one of the following linear equations where \var{B} is a dense 
matrix and \var{F} is the numeric 
factorization~(\ref{e-chol-ll}) or~(\ref{e-chol-ldl}) computed by 
\function{cholmod\_numeric()}.  
\var{sys} is an integer with values between 0 and 8. 

\begin{center}
\begin{tabular}{c|c|} \hline
 \var{sys} & equation \\ \hline
 0 & $AX=B$     \\ \hline 
 1 & $LDL^TX=B$ \\ \hline 
 2 & $LDLX=B$   \\ \hline
 3 & $DL^TX=B$  \\ \hline
 4 & $LX=B$     \\ \hline 
 5 & $L^TX=B$   \\ \hline
 6 & $DX=B$     \\ \hline 
 7 & $P^TX=B$   \\ \hline 
 8 & $PX=B$  \\ \hline  
\end{tabular}
\end{center}

(If \var{F} is a Cholesky factorization of the form~(\ref{e-chol-ll}), 
\tm{D} is an identity matrix in this table. 
If \var{A} is complex, $L^T$ should be replaced by $L^H$.)

The matrix \var{B} is a dense \dtc\ or \ztc\ matrix, with the same type
as \var{A}.  On exit it is overwritten by the solution.
\end{funcdesc}

\begin{funcdesc}{spsolve}{F, B\optional{, sys=0}}
Similar to \function{solve()}, except that \var{B} is a 
sparse matrix, and the solution is returned as an output argument
(as a sparse matrix).  \var{B} must have the same typecode as \var{A}.
\end{funcdesc}

For the same example as above:
\begin{verbatim}
>>> X = matrix(range(8), (4,2), 'd')
>>> F = cholmod.symbolic(A)
>>> cholmod.numeric(A,F)
>>> cholmod.solve(F,X)
>>> print X
[-1.46e-01  4.88e-02]
[ 1.33e+00  4.00e+00]
[ 4.88e-01  1.17e+00]
[ 2.83e+00  7.50e+00]
\end{verbatim}

\begin{funcdesc}{diag}{F}
Returns the diagonal elements of the Cholesky factor \tm{L} 
in~(\ref{e-chol-ll}), as a dense matrix of the same type as \var{A}.
Note that this only applies to Cholesky factorizations.
The matrix \tm{D} in an $\mathrm{LDL^T}$ factorization can be 
retrieved via \function{cholmod.solve()} with \var{sys} equal to 6.
\end{funcdesc}

In the functions listed above, the default values of the control 
parameters described in the CHOLMOD user guide are used, except for 
\ctype{Common->print} which is set to 0 instead of 3
and 
\ctype{Common->supernodal} which is set to 2 instead of 1.
These parameters (and a few others) can be modified by making an 
entry in the dictionary \member{cholmod.options}. 
The meaning of these parameters is as follows.
\begin{description}
\item[\code{options['supernodal']}] If equal to 0, a 
factorization~(\ref{e-chol-ldl}) is computed using a simplicial 
algorithm.
If equal to 2, a factorization~(\ref{e-chol-ll}) is
computed using a supernodal algorithm.
If equal to 1, the most efficient of the two factorizations is
selected, based on the sparsity pattern.  Default: 2.

\item[\code{options['print']}] A nonnegative integer that controls the 
amount of output printed to the screen.
Default: 0 (no output).
\end{description}

As an example that illustrates \function{diag()} and the use of 
\member{cholmod.options}, we compute the logarithm of the determinant 
of the coefficient matrix in~(\ref{e-A-pd}) by two methods.
\begin{verbatim}
>>> import math
>>> from cvxopt.cholmod import options
>>> from cvxopt.base import log
>>> F = cholmod.symbolic(A)
>>> cholmod.numeric(A,F)
>>> print 2.0 * sum(log(cholmod.diag(F)))
5.50533153593
\end{verbatim}
\begin{verbatim}
>>> options['supernodal'] = 0
>>> F = cholmod.symbolic(A)
>>> cholmod.numeric(A,F)
>>> Di = matrix(1.0, (4,1))
>>> cholmod.solve(F,Di,sys=6)
>>> print -sum(log(Di))
5.50533153593
\end{verbatim}

\section{Example: Covariance Selection}
This example illustrates the use of the routines for sparse Cholesky 
factorization.  We consider the problem 
\BEQ \label{e-covsel}
 \begin{array}{ll}
 \mbox{minimize} & -\log\det K + \Tr(KY)\\
 \mbox{subject to} & K_{ij}=0,\quad (i,j) \not \in S.
 \end{array}
\EEQ
The optimization variable is a symmetric matrix \tm{K} of order \tm{n}
and the domain of the problem is the set of positive definite matrices.
The matrix \tm{Y} and the index set  \tm{S} are given.  We assume that all 
the diagonal positions are included in \tm{S}.
This problem arises in maximum likelihood estimation of the covariance
matrix of a zero-mean normal distribution, with constraints 
that specify that pairs of variables are conditionally independent.

We can express \tm{K} as
\[
   K(x) = E_1\diag(x)E_2^T+E_2\diag(x)E_1^T
\]
where \tm{x} are the nonzero elements in the lower triangular part
of \tm{K}, with the diagonal elements scaled by 1/2,
and
\[
 E_1 = \left[ \begin{array}{cccc}
  e_{i_1} & e_{i_2} & \cdots & e_{i_q} \end{array}\right], \qquad
 E_2 = \left[ \begin{array}{cccc}
  e_{j_1} & e_{j_2} & \cdots & e_{j_q} \end{array}\right], 
\]
where (\tm{\s{i}{k}}, \tm{\s{j}{k}}) are the positions of the nonzero 
entries in the lower-triangular part of \tm{K}.
With this notation, we can solve problem~(\ref{e-covsel}) by solving
the unconstrained problem
\[
 \begin{array}{ll}
   \mbox{minimize} & f(x) = -\log\det K(x) + \Tr(K(x)Y).
 \end{array}
\]
The code below implements Newton's method with a backtracking line
search.  The gradient and Hessian of the objective function are given 
by
\begin{eqnarray*}
\nabla f(x) 
&=& 2 \diag( E_1^T (Y - K(x)^{-1}) E_2)) \\
&=& 2\diag(Y_{IJ} - \left(K(x)^{-1}\right)_{IJ})\\
\nabla^2 f(x) & = & 
 2 (E_1^T K(x)^{-1} E_1) \circ (E_2^T K(x)^{-1} E_2) 
   + 2 (E_1^T K(x)^{-1} E_2) \circ (E_2^T K(x)^{-1} E_1) \\
&=& 2 \left(K(x)^{-1}\right)_{II} \circ 
      \left(K(x)^{-1}\right)_{JJ}
   +2 \left(K(x)^{-1}\right)_{IJ} \circ 
      \left(K(x)^{-1}\right)_{JI},
\end{eqnarray*} 
where \code{o} denotes Hadamard product.

\begin{verbatim}
from cvxopt.base import matrix, spmatrix, log, mul
from cvxopt import blas, lapack, amd, cholmod

def covsel(Y):
    """
    Returns the solution of

         minimize    -logdet K + Tr(KY)
         subject to  K_{ij}=0,  (i,j) not in indices listed in I,J.

    Y is a symmetric sparse matrix with nonzero diagonal elements.
    I = Y.I,  J = Y.J.
    """

    I, J = Y.I, Y.J
    n, m = Y.size[0], len(I) 
    N = I + J*n         # non-zero positions for one-argument indexing 
    D = [k for k in xrange(m) if I[k]==J[k]]  # position of diagonal elements

    # starting point: symmetric identity with nonzero pattern I,J
    K = spmatrix(0, I, J) 
    K[::n+1] = 1

    # Kn is used in the line search
    Kn = spmatrix(0, I, J)

    # symbolic factorization of K 
    F = cholmod.symbolic(K)

    # Kinv will be the inverse of K
    Kinv = matrix(0.0, (n,n))
    
    for iters in xrange(100):

        # numeric factorization of K
        cholmod.numeric(K, F)
        d = cholmod.diag(F)

        # compute Kinv by solving K*X = I 
        Kinv[:] = 0
        Kinv[::n+1] = 1
        cholmod.solve(F, Kinv)

        # solve Newton system
        grad = 2*(Y.V - Kinv[N])
        hess = 2*(mul(Kinv[I,J],Kinv[J,I]) + mul(Kinv[I,I],Kinv[J,J]))
        v = -grad
        lapack.posv(hess,v) 
        
        # stopping criterion
        sqntdecr = -blas.dot(grad,v) 
        print "Newton decrement squared:%- 7.5e" %sqntdecr
        if (sqntdecr < 1e-12):
            print "number of iterations: ", iters+1 
            break

        # line search
        dx = +v
        dx[D] *= 2      # scale the diagonal elems        
        f = -2.0 * sum(log(d))    # f = -log det K
        s = 1
        for lsiter in xrange(50):
            Kn.V = K.V + s*dx
            try: 
                cholmod.numeric(Kn, F)
            except ArithmeticError:
                s *= 0.5
            else:
                d = cholmod.diag(F)
                fn = -2.0 * sum(log(d)) + 2*s*blas.dot(v,Y.V)
                if (fn < f - 0.01*s*sqntdecr): 
                     break
                s *= 0.5
            
        K.V = Kn.V

    return K
\end{verbatim}
