org.apache.spark.storage

Class StorageUtils

Object

org.apache.spark.storage.StorageUtils

public class StorageUtils
extends Object

Helper methods for storage-related objects.

Constructor Summary

Constructors

Constructor and Description
StorageUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static void	dispose(java.nio.ByteBuffer buffer) Attempt to clean up a ByteBuffer if it is direct or memory-mapped.
static scala.collection.Map<BlockId,scala.collection.Seq<String>>	getRddBlockLocations(int rddId, scala.collection.Seq<StorageStatus> statuses) Return a mapping from block ID to its locations for each block that belongs to the given RDD.
static void	unmap(java.nio.ByteBuffer buffer) Attempt to unmap a ByteBuffer if it is memory-mapped.
static void	updateRddInfo(scala.collection.Seq<RDDInfo> rddInfos, scala.collection.Seq<StorageStatus> statuses) Update the given list of RDDInfo with the given list of storage statuses.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StorageUtils

public StorageUtils()

Method Detail

dispose

public static void dispose(java.nio.ByteBuffer buffer)

Attempt to clean up a ByteBuffer if it is direct or memory-mapped. This uses an *unsafe* Sun API that will cause errors if one attempts to read from the disposed buffer. However, neither the bytes allocated to direct buffers nor file descriptors opened for memory-mapped buffers put pressure on the garbage collector. Waiting for garbage collection may lead to the depletion of off-heap memory or huge numbers of open files. There's unfortunately no standard API to manually dispose of these kinds of buffers.

See also unmap

Parameters:

buffer - (undocumented)

unmap

public static void unmap(java.nio.ByteBuffer buffer)

Attempt to unmap a ByteBuffer if it is memory-mapped. This uses an *unsafe* Sun API that will cause errors if one attempts to read from the unmapped buffer. However, the file descriptors of memory-mapped buffers do not put pressure on the garbage collector. Waiting for garbage collection may lead to huge numbers of open files. There's unfortunately no standard API to manually unmap memory-mapped buffers.

See also dispose

Parameters:

buffer - (undocumented)

updateRddInfo

public static void updateRddInfo(scala.collection.Seq<RDDInfo> rddInfos,
                 scala.collection.Seq<StorageStatus> statuses)

Update the given list of RDDInfo with the given list of storage statuses. This method overwrites the old values stored in the RDDInfo's.

Parameters:

rddInfos - (undocumented)

statuses - (undocumented)

getRddBlockLocations

public static scala.collection.Map<BlockId,scala.collection.Seq<String>> getRddBlockLocations(int rddId,
                                                                              scala.collection.Seq<StorageStatus> statuses)

Return a mapping from block ID to its locations for each block that belongs to the given RDD.

Parameters:

rddId - (undocumented)

statuses - (undocumented)

Returns:

(undocumented)