org.apache.spark

Interface ExecutorPlugin

@DeveloperApi
public interface ExecutorPlugin

A plugin which can be automatically instantiated within each Spark executor. Users can specify plugins which should be created with the "spark.executor.plugins" configuration. An instance of each plugin will be created for every executor, including those created by dynamic allocation, before the executor starts running any tasks. The specific api exposed to the end users still considered to be very unstable. We will hopefully be able to keep compatibility by providing default implementations for any methods added, but make no guarantees this will always be possible across all Spark releases. Spark does nothing to verify the plugin is doing legitimate things, or to manage the resources it uses. A plugin acquires the same privileges as the user running the task. A bad plugin could also interfere with task execution and make the executor fail in unexpected ways.

Method Summary

Modifier and Type	Method and Description
default void	init(ExecutorPluginContext pluginContext) Initialize the executor plugin.
default void	shutdown() Clean up and terminate this plugin.

Method Detail

init

default void init(ExecutorPluginContext pluginContext)

Initialize the executor plugin.

Each executor will, during its initialization, invoke this method on each plugin provided in the spark.executor.plugins configuration. The Spark executor will wait on the completion of the execution of the init method.

Plugins should create threads in their implementation of this method for any polling, blocking, or intensive computation.

Parameters:

pluginContext - Context information for the executor where the plugin is running.

shutdown

default void shutdown()

Clean up and terminate this plugin.

This function is called during the executor shutdown phase. The executor will wait for the plugin to terminate before continuing its own shutdown.