org.apache.spark.rdd

Class DoubleRDDFunctions

Object

org.apache.spark.rdd.DoubleRDDFunctions

All Implemented Interfaces:

java.io.Serializable, Logging

public class DoubleRDDFunctions
extends Object
implements Logging, scala.Serializable

Extra functions available on RDDs of Doubles through an implicit conversion. Import org.apache.spark.SparkContext._ at the top of your program to use these functions.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
DoubleRDDFunctions(RDD<Object> self)

Method Summary

Methods

Modifier and Type	Method and Description
long[]	histogram(double[] buckets, boolean evenBuckets) Compute a histogram using the provided buckets.
scala.Tuple2<double[],long[]>	histogram(int bucketCount) Compute a histogram of the data using bucketCount number of buckets evenly spaced between the minimum and maximum of the RDD.
double	mean() Compute the mean of this RDD's elements.
PartialResult<BoundedDouble>	meanApprox(long timeout, double confidence) :: Experimental :: Approximate operation to return the mean within a timeout.
double	sampleStdev() Compute the sample standard deviation of this RDD's elements (which corrects for bias in estimating the standard deviation by dividing by N-1 instead of N).
double	sampleVariance() Compute the sample variance of this RDD's elements (which corrects for bias in estimating the variance by dividing by N-1 instead of N).
StatCounter	stats() Return a StatCounter object that captures the mean, variance and count of the RDD's elements in one operation.
double	stdev() Compute the standard deviation of this RDD's elements.
double	sum() Add up the elements in this RDD.
PartialResult<BoundedDouble>	sumApprox(long timeout, double confidence) :: Experimental :: Approximate operation to return the sum within a timeout.
double	variance() Compute the variance of this RDD's elements.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

DoubleRDDFunctions

public DoubleRDDFunctions(RDD<Object> self)

Method Detail

sum

public double sum()

Add up the elements in this RDD.

stats

public StatCounter stats()

Return a StatCounter object that captures the mean, variance and count of the RDD's elements in one operation.

mean

public double mean()

Compute the mean of this RDD's elements.

variance

public double variance()

Compute the variance of this RDD's elements.

stdev

public double stdev()

Compute the standard deviation of this RDD's elements.

sampleStdev

public double sampleStdev()

Compute the sample standard deviation of this RDD's elements (which corrects for bias in estimating the standard deviation by dividing by N-1 instead of N).

sampleVariance

public double sampleVariance()

Compute the sample variance of this RDD's elements (which corrects for bias in estimating the variance by dividing by N-1 instead of N).

meanApprox

public PartialResult<BoundedDouble> meanApprox(long timeout,
                                      double confidence)

:: Experimental :: Approximate operation to return the mean within a timeout.

sumApprox

public PartialResult<BoundedDouble> sumApprox(long timeout,
                                     double confidence)

:: Experimental :: Approximate operation to return the sum within a timeout.

histogram

public scala.Tuple2<double[],long[]> histogram(int bucketCount)

Compute a histogram of the data using bucketCount number of buckets evenly spaced between the minimum and maximum of the RDD. For example if the min value is 0 and the max is 100 and there are two buckets the resulting buckets will be [0, 50) [50, 100]. bucketCount must be at least 1 If the RDD contains infinity, NaN throws an exception If the elements in RDD do not vary (max == min) always returns a single bucket.

histogram

public long[] histogram(double[] buckets,
               boolean evenBuckets)

Compute a histogram using the provided buckets. The buckets are all open to the right except for the last which is closed e.g. for the array [1, 10, 20, 50] the buckets are [1, 10) [10, 20) [20, 50] e.g 1<=x<10 , 10<=x<20, 20<=x<=50 And on the input of 1 and 50 we would have a histogram of 1, 0, 1

Note: if your histogram is evenly spaced (e.g. [0, 10, 20, 30]) this can be switched from an O(log n) inseration to O(1) per element. (where n = # buckets) if you set evenBuckets to true. buckets must be sorted and not contain any duplicates. buckets array must be at least two elements All NaN entries are treated the same. If you have a NaN bucket it must be the maximum value of the last position and all NaN entries will be counted in that bucket.