org.apache.spark.ml.feature

Class CountVectorizerModel

Object

org.apache.spark.ml.PipelineStage

org.apache.spark.ml.Transformer

org.apache.spark.ml.Model<CountVectorizerModel>

org.apache.spark.ml.feature.CountVectorizerModel

All Implemented Interfaces:

java.io.Serializable, Logging, Params, Identifiable, MLWritable

public class CountVectorizerModel
extends Model<CountVectorizerModel>
implements MLWritable

:: Experimental :: Converts a text document to a sparse vector of token counts. param: vocabulary An Array over terms. Only the terms in the vocabulary will be counted.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
CountVectorizerModel(String[] vocabulary)
CountVectorizerModel(String uid, String[] vocabulary)

Method Summary

Methods

Modifier and Type	Method and Description
CountVectorizerModel	copy(ParamMap extra) Creates a copy of this instance with the same UID and some extra params.
double	getMinDF()
double	getMinTF()
int	getVocabSize()
static CountVectorizerModel	load(String path)
DoubleParam	minDF() Specifies the minimum number of different documents a term must appear in to be included in the vocabulary.
DoubleParam	minTF() Filter to ignore rare words in a document.
static MLReader<CountVectorizerModel>	read()
CountVectorizerModel	setInputCol(String value)
CountVectorizerModel	setMinTF(double value)
CountVectorizerModel	setOutputCol(String value)
DataFrame	transform(DataFrame dataset) Transforms the input dataset.
StructType	transformSchema(StructType schema) :: DeveloperApi ::
String	uid() An immutable unique ID for the object and its derivatives.
StructType	validateAndTransformSchema(StructType schema) Validates and transforms the input schema.
IntParam	vocabSize() Max size of the vocabulary.
String[]	vocabulary()
MLWriter	write() Returns an MLWriter instance for this ML instance.

Methods inherited from class org.apache.spark.ml.Model

hasParent, parent, setParent

Methods inherited from class org.apache.spark.ml.Transformer

transform, transform, transform

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.param.Params

clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn, validateParams

Methods inherited from interface org.apache.spark.ml.util.Identifiable

toString

Methods inherited from interface org.apache.spark.ml.util.MLWritable

save

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

CountVectorizerModel

public CountVectorizerModel(String uid,
                    String[] vocabulary)

CountVectorizerModel

public CountVectorizerModel(String[] vocabulary)

Method Detail

read

public static MLReader<CountVectorizerModel> read()

load

public static CountVectorizerModel load(String path)

uid

public String uid()

Description copied from interface: Identifiable

An immutable unique ID for the object and its derivatives.

Specified by:

uid in interface Identifiable

Returns:

(undocumented)

vocabulary

public String[] vocabulary()

setInputCol

public CountVectorizerModel setInputCol(String value)

setOutputCol

public CountVectorizerModel setOutputCol(String value)

setMinTF

public CountVectorizerModel setMinTF(double value)

transform

public DataFrame transform(DataFrame dataset)

Description copied from class: Transformer

Transforms the input dataset.

Specified by:

transform in class Transformer

Parameters:

dataset - (undocumented)

Returns:

(undocumented)

transformSchema

public StructType transformSchema(StructType schema)

Description copied from class: PipelineStage

:: DeveloperApi ::

Derives the output schema from the input schema.

Specified by:

transformSchema in class PipelineStage

Parameters:

schema - (undocumented)

Returns:

(undocumented)

copy

public CountVectorizerModel copy(ParamMap extra)

Description copied from interface: Params

Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly.

Specified by:

copy in interface Params

Specified by:

copy in class Model<CountVectorizerModel>

Parameters:

extra - (undocumented)

Returns:

(undocumented)

See Also:

defaultCopy()

write

public MLWriter write()

Description copied from interface: MLWritable

Returns an MLWriter instance for this ML instance.

Specified by:

write in interface MLWritable

Returns:

(undocumented)

vocabSize

public IntParam vocabSize()

Max size of the vocabulary. CountVectorizer will build a vocabulary that only considers the top vocabSize terms ordered by term frequency across the corpus.

Default: 2^18^

Returns:

(undocumented)

getVocabSize

public int getVocabSize()

minDF

public DoubleParam minDF()

Specifies the minimum number of different documents a term must appear in to be included in the vocabulary. If this is an integer >= 1, this specifies the number of documents the term must appear in; if this is a double in [0,1), then this specifies the fraction of documents.

Default: 1

Returns:

(undocumented)

getMinDF

public double getMinDF()

validateAndTransformSchema

public StructType validateAndTransformSchema(StructType schema)

Validates and transforms the input schema.

minTF

public DoubleParam minTF()

Filter to ignore rare words in a document. For each document, terms with frequency/count less than the given threshold are ignored. If this is an integer >= 1, then this specifies a count (of times the term must appear in the document); if this is a double in [0,1), then this specifies a fraction (out of the document's token count).

Note that the parameter is only used in transform of CountVectorizerModel and does not affect fitting.

Default: 1

Returns:

(undocumented)

getMinTF

public double getMinTF()