org.apache.spark.mllib.linalg.distributed

Class IndexedRowMatrix

Object

org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix

All Implemented Interfaces:

java.io.Serializable, DistributedMatrix

public class IndexedRowMatrix
extends Object
implements DistributedMatrix

:: Experimental :: Represents a row-oriented DistributedMatrix with indexed rows.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
IndexedRowMatrix(RDD<IndexedRow> rows) Alternative constructor leaving matrix dimensions to be determined automatically.
IndexedRowMatrix(RDD<IndexedRow> rows, long nRows, int nCols)

Method Summary

Methods

Modifier and Type	Method and Description
Matrix	computeGramianMatrix() Computes the Gramian matrix A^T A.
SingularValueDecomposition<IndexedRowMatrix,Matrix>	computeSVD(int k, boolean computeU, double rCond) Computes the singular value decomposition of this matrix.
IndexedRowMatrix	multiply(Matrix B) Multiply this matrix by a local matrix on the right.
long	numCols() Gets or computes the number of columns.
long	numRows() Gets or computes the number of rows.
RDD<IndexedRow>	rows()
RowMatrix	toRowMatrix() Drops row indices and converts this matrix to a RowMatrix.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IndexedRowMatrix

public IndexedRowMatrix(RDD<IndexedRow> rows,
                long nRows,
                int nCols)

IndexedRowMatrix

public IndexedRowMatrix(RDD<IndexedRow> rows)

Alternative constructor leaving matrix dimensions to be determined automatically.

Method Detail

rows

public RDD<IndexedRow> rows()

numCols

public long numCols()

Description copied from interface: DistributedMatrix

Gets or computes the number of columns.

Specified by:

numCols in interface DistributedMatrix

numRows

public long numRows()

Description copied from interface: DistributedMatrix

Gets or computes the number of rows.

Specified by:

numRows in interface DistributedMatrix

toRowMatrix

public RowMatrix toRowMatrix()

Drops row indices and converts this matrix to a RowMatrix.

computeSVD

public SingularValueDecomposition<IndexedRowMatrix,Matrix> computeSVD(int k,
                                                             boolean computeU,
                                                             double rCond)

Computes the singular value decomposition of this matrix. Denote this matrix by A (m x n), this will compute matrices U, S, V such that A = U * S * V'.

There is no restriction on m, but we require n^2 doubles to fit in memory. Further, n should be less than m.

The decomposition is computed by first computing A'A = V S^2 V', computing svd locally on that (since n x n is small), from which we recover S and V. Then we compute U via easy matrix multiplication as U = A * (V * S^-1). Note that this approach requires O(n^3) time on the master node.

At most k largest non-zero singular values and associated vectors are returned. If there are k such values, then the dimensions of the return will be:

U is an IndexedRowMatrix of size m x k that satisfies U'U = eye(k), s is a Vector of size k, holding the singular values in descending order, and V is a local Matrix of size n x k that satisfies V'V = eye(k).

Parameters:

k - number of singular values to keep. We might return less than k if there are numerically zero singular values. See rCond.

computeU - whether to compute U

rCond - the reciprocal condition number. All singular values smaller than rCond * sigma(0) are treated as zero, where sigma(0) is the largest singular value.

Returns:

SingularValueDecomposition(U, s, V)

multiply

public IndexedRowMatrix multiply(Matrix B)

Multiply this matrix by a local matrix on the right.

Parameters:

B - a local matrix whose number of rows must match the number of columns of this matrix

Returns:

an IndexedRowMatrix representing the product, which preserves partitioning

computeGramianMatrix

public Matrix computeGramianMatrix()

Computes the Gramian matrix A^T A.