org.apache.spark.streaming.rdd

Class WriteAheadLogBackedBlockRDD<T>

Object

org.apache.spark.rdd.RDD<T>

org.apache.spark.rdd.BlockRDD<T>

org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD<T>

All Implemented Interfaces:

java.io.Serializable, Logging

public class WriteAheadLogBackedBlockRDD<T>
extends BlockRDD<T>

This class represents a special case of the BlockRDD where the data blocks in the block manager are also backed by segments in write ahead logs. For reading the data, this RDD first looks up the blocks by their ids in the block manager. If it does not find them, it looks up the corresponding file segment.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
WriteAheadLogBackedBlockRDD(SparkContext sc, BlockId[] blockIds, WriteAheadLogFileSegment[] segments, boolean storeInBlockManager, StorageLevel storageLevel, scala.reflect.ClassTag<T> evidence$1)

Method Summary

Methods

Modifier and Type	Method and Description
scala.collection.Iterator<T>	compute(Partition split, TaskContext context) Gets the partition data by getting the corresponding block from the block manager.
Partition[]	getPartitions() Implemented by subclasses to return the set of partitions in this RDD.
scala.collection.Seq<String>	getPreferredLocations(Partition split) Get the preferred location of the partition.

Methods inherited from class org.apache.spark.rdd.BlockRDD

assertValid, blockIds, isValid, locations_, removeBlocks

Methods inherited from class org.apache.spark.rdd.RDD

aggregate, cache, cartesian, checkpoint, checkpointData, coalesce, collect, collect, collectPartitions, computeOrReadCheckpoint, conf, context, count, countApprox, countApproxDistinct, countApproxDistinct, countByValue, countByValueApprox, creationSite, dependencies, distinct, distinct, doCheckpoint, elementClassTag, filter, filterWith, first, flatMap, flatMapWith, fold, foreach, foreachPartition, foreachWith, getCheckpointFile, getCreationSite, getNarrowAncestors, getStorageLevel, glom, groupBy, groupBy, groupBy, id, intersection, intersection, intersection, isCheckpointed, iterator, keyBy, map, mapPartitions, mapPartitionsWithContext, mapPartitionsWithIndex, mapPartitionsWithSplit, mapWith, markCheckpointed, max, min, name, partitioner, partitions, persist, persist, pipe, pipe, pipe, preferredLocations, randomSplit, reduce, repartition, retag, retag, sample, saveAsObjectFile, saveAsTextFile, saveAsTextFile, setName, sortBy, sparkContext, subtract, subtract, subtract, take, takeOrdered, takeSample, toArray, toDebugString, toJavaRDD, toLocalIterator, top, toString, union, unpersist, zip, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipWithIndex, zipWithUniqueId

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

WriteAheadLogBackedBlockRDD

public WriteAheadLogBackedBlockRDD(SparkContext sc,
                           BlockId[] blockIds,
                           WriteAheadLogFileSegment[] segments,
                           boolean storeInBlockManager,
                           StorageLevel storageLevel,
                           scala.reflect.ClassTag<T> evidence$1)

Method Detail

getPartitions

public Partition[] getPartitions()

Description copied from class: RDD

Implemented by subclasses to return the set of partitions in this RDD. This method will only be called once, so it is safe to implement a time-consuming computation in it.

Overrides:

getPartitions in class BlockRDD<T>

compute

public scala.collection.Iterator<T> compute(Partition split,
                                   TaskContext context)

Gets the partition data by getting the corresponding block from the block manager. If the block does not exist, then the data is read from the corresponding segment in write ahead log files.

Overrides:

compute in class BlockRDD<T>

getPreferredLocations

public scala.collection.Seq<String> getPreferredLocations(Partition split)

Get the preferred location of the partition. This returns the locations of the block if it is present in the block manager, else it returns the location of the corresponding segment in HDFS.

Overrides:

getPreferredLocations in class BlockRDD<T>