org.apache.spark.scheduler.cluster

Class CoarseGrainedSchedulerBackend

Object

org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend

All Implemented Interfaces:

ExecutorAllocationClient, Logging, SchedulerBackend

Direct Known Subclasses:

CoarseMesosSchedulerBackend, SimrSchedulerBackend, SparkDeploySchedulerBackend, YarnSchedulerBackend

public class CoarseGrainedSchedulerBackend
extends Object
implements ExecutorAllocationClient, SchedulerBackend, Logging

A scheduler backend that waits for coarse grained executors to connect to it through Akka. This backend holds onto each executor for the duration of the Spark job rather than relinquishing executors whenever a task is done and asking the scheduler to launch a new executor for each new task. Executors may be launched in a variety of ways, such as Mesos tasks for the coarse-grained Mesos mode or standalone processes for Spark's standalone deploy mode (spark.deploy.*).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	CoarseGrainedSchedulerBackend.DriverActor

Constructor Summary

Constructors

Constructor and Description
CoarseGrainedSchedulerBackend(TaskSchedulerImpl scheduler, akka.actor.ActorSystem actorSystem)

Method Summary

Methods

Modifier and Type	Method and Description
static String	ACTOR_NAME()
akka.actor.ActorSystem	actorSystem()
SparkConf	conf()
long	createTime()
int	defaultParallelism()
akka.actor.ActorRef	driverActor()
boolean	isReady()
boolean	killExecutors(scala.collection.Seq<String> executorIds) Request that the cluster manager kill the specified executors.
void	killTask(long taskId, String executorId, boolean interruptThread)
int	maxRegisteredWaitingTime()
double	minRegisteredRatio()
int	numExistingExecutors() Return the number of executors currently registered with this backend.
void	removeExecutor(String executorId, String reason)
boolean	requestExecutors(int numAdditionalExecutors) Request an additional number of executors from the cluster manager.
boolean	requestTotalExecutors(int numExecutors) Express a preference to the cluster manager for a given total number of executors.
void	reviveOffers()
void	start()
void	stop()
void	stopExecutors()
boolean	sufficientResourcesRegistered()
scala.collection.mutable.HashMap<String,scala.collection.mutable.HashSet<String>>	taskIdsOnSlave()
java.util.concurrent.atomic.AtomicInteger	totalCoreCount()
java.util.concurrent.atomic.AtomicInteger	totalRegisteredExecutors()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ExecutorAllocationClient

killExecutor

Methods inherited from interface org.apache.spark.scheduler.SchedulerBackend

appId, applicationId

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

CoarseGrainedSchedulerBackend

public CoarseGrainedSchedulerBackend(TaskSchedulerImpl scheduler,
                             akka.actor.ActorSystem actorSystem)

Method Detail

ACTOR_NAME

public static String ACTOR_NAME()

actorSystem

public akka.actor.ActorSystem actorSystem()

totalCoreCount

public java.util.concurrent.atomic.AtomicInteger totalCoreCount()

totalRegisteredExecutors

public java.util.concurrent.atomic.AtomicInteger totalRegisteredExecutors()

conf

public SparkConf conf()

minRegisteredRatio

public double minRegisteredRatio()

maxRegisteredWaitingTime

public int maxRegisteredWaitingTime()

createTime

public long createTime()

driverActor

public akka.actor.ActorRef driverActor()

taskIdsOnSlave

public scala.collection.mutable.HashMap<String,scala.collection.mutable.HashSet<String>> taskIdsOnSlave()

start

public void start()

Specified by:

start in interface SchedulerBackend

stopExecutors

public void stopExecutors()

stop

public void stop()

Specified by:

stop in interface SchedulerBackend

reviveOffers

public void reviveOffers()

Specified by:

reviveOffers in interface SchedulerBackend

killTask

public void killTask(long taskId,
            String executorId,
            boolean interruptThread)

Specified by:

killTask in interface SchedulerBackend

defaultParallelism

public int defaultParallelism()

Specified by:

defaultParallelism in interface SchedulerBackend

removeExecutor

public void removeExecutor(String executorId,
                  String reason)

sufficientResourcesRegistered

public boolean sufficientResourcesRegistered()

isReady

public boolean isReady()

Specified by:

isReady in interface SchedulerBackend

numExistingExecutors

public int numExistingExecutors()

Return the number of executors currently registered with this backend.

requestExecutors

public final boolean requestExecutors(int numAdditionalExecutors)

Request an additional number of executors from the cluster manager.

Specified by:

requestExecutors in interface ExecutorAllocationClient

Returns:

whether the request is acknowledged.

requestTotalExecutors

public final boolean requestTotalExecutors(int numExecutors)

Express a preference to the cluster manager for a given total number of executors. This can result in canceling pending requests or filing additional requests.

Specified by:

requestTotalExecutors in interface ExecutorAllocationClient

Returns:

whether the request is acknowledged.

killExecutors

public final boolean killExecutors(scala.collection.Seq<String> executorIds)

Request that the cluster manager kill the specified executors. Return whether the kill request is acknowledged.

Specified by:

killExecutors in interface ExecutorAllocationClient

Returns:

whether the request is acknowledged by the cluster manager.