org.apache.spark.ml.source.libsvm

Class DefaultSource

java.lang.Object

org.apache.spark.ml.source.libsvm.DefaultSource

All Implemented Interfaces:

DataSourceRegister, RelationProvider

public class DefaultSource
extends java.lang.Object
implements RelationProvider, DataSourceRegister

libsvm package implements Spark SQL data source API for loading LIBSVM data as DataFrame. The loaded DataFrame has two columns: label containing labels stored as doubles and features containing feature vectors stored as Vectors.

To use LIBSVM data source, you need to set "libsvm" as the format in DataFrameReader and optionally specify options, for example:

   // Scala
   val df = sqlContext.read.format("libsvm")
     .option("numFeatures", "780")
     .load("data/mllib/sample_libsvm_data.txt")

   // Java
   DataFrame df = sqlContext.read().format("libsvm")
     .option("numFeatures, "780")
     .load("data/mllib/sample_libsvm_data.txt");
 

LIBSVM data source supports the following options: - "numFeatures": number of features. If unspecified or nonpositive, the number of features will be determined automatically at the cost of one additional pass. This is also useful when the dataset is already split into multiple files and you want to load them separately, because some features may not present in certain files, which leads to inconsistent feature dimensions. - "vectorType": feature vector type, "sparse" (default) or "dense".

See Also:

Constructor Summary

Constructors

Constructor and Description
DefaultSource()

Method Summary

Methods

Modifier and Type	Method and Description
BaseRelation	createRelation(SQLContext sqlContext, scala.collection.immutable.Map<java.lang.String,java.lang.String> parameters) Returns a new base relation with the given parameters.
java.lang.String	shortName() The string that represents the format that this data source provider uses.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DefaultSource

public DefaultSource()

Method Detail

shortName

public java.lang.String shortName()

Description copied from interface: DataSourceRegister

The string that represents the format that this data source provider uses. This is overridden by children to provide a nice alias for the data source. For example:

   override def format(): String = "parquet"
 

Specified by:

shortName in interface DataSourceRegister

Returns:

(undocumented)

createRelation

public BaseRelation createRelation(SQLContext sqlContext,
                          scala.collection.immutable.Map<java.lang.String,java.lang.String> parameters)

Description copied from interface: RelationProvider

Returns a new base relation with the given parameters. Note: the parameters' keywords are case insensitive and this insensitivity is enforced by the Map that is passed to the function.

Specified by:

createRelation in interface RelationProvider

Parameters:

sqlContext - (undocumented)

parameters - (undocumented)

Returns:

(undocumented)